Network configuration on Linux — practice with a workstation from scratch, a misconfigured one and a script that reads the ifcfg (and a ping that only got halfway)

The day I configured my first workstation in the lab, the ping to the gateway worked and the ping to the router didn’t. I spent ten minutes checking cables, the gateway, the router… and the problem was mine: I hadn’t set the default gateway. Since then I check layer by layer: first my card, then my network, then whatever lies beyond. The lesson: one failing ping tells you very little; three well-chosen pings tell you exactly where the problem is.

This practice applies the network configuration theory in the same isolated lab (workstation, gateway and lab router, simulated with network namespaces). There are three intermediate programs with lab situations: bringing up a workstation by hand checking each step, diagnosing and fixing a workstation with the wrong IP, and applying an ifcfg with a script that does the same as CentOS’s network service.

Before you start

Make sure you’re clear on:

• What the IP, the mask (/24 = 255.255.255.0) and the default gateway are.
• The three configuration commands: ip link set … up, ip addr add … dev … and ip route add default via ….
• How to read ip -br addr, ip route and the output of ping.
• What an ifcfg contains (DEVICE, BOOTPROTO, ONBOOT, IPADDR, NETMASK, GATEWAY).

If any of them sounds shaky, go over the theory before carrying on.

Program 1: bring the workstation up by hand

The desk 7 workstation has just been installed: the enp2s0 card is down and has no IP. It has to end up with 10.110.7.4/24 and the gateway 10.110.7.1 as its default gateway, checking after each step what can and can’t be reached.

The idea

Three steps with a control ping between them:

1. Before touching anything, a ping to the gateway to see the starting point.
2. Bring the card up and give it an IP with a mask. Control: the gateway (my network) must reply; the router (another network), not yet.
3. Add the default gateway. Control: now the router replies.

Testing both destinations at each step is what tells you which part is missing.

The code

# @lab base
ip -br addr                              # starting point
ping -c 1 10.110.7.1                     # first of all: can I reach the gateway?
ip link set enp2s0 up                    # step 1: bring it up
ip addr add 10.110.7.4/24 dev enp2s0     # step 2: IP with its mask
ping -c 2 10.110.7.1                     # the gateway replies now
ping -c 1 10.110.1.1                     # the router not yet
ip route add default via 10.110.7.1      # step 3: default gateway
ping -c 2 10.110.1.1                     # now it works
ip -br addr
ip route

Each command has a comment with its reason. The key thing is the order of the checks: the ping to the router (the “router not yet” line) is placed before the default gateway on purpose, so we see the failure with our own eyes.

Output

# ip -br addr
lo               UNKNOWN        127.0.0.1/8 
enp2s0@if41      DOWN           
# ping -c 1 10.110.7.1
ping: connect: Network is unreachable
# ip link set enp2s0 up
# ip addr add 10.110.7.4/24 dev enp2s0
# ping -c 2 10.110.7.1
PING 10.110.7.1 (10.110.7.1) 56(84) bytes of data.
64 bytes from 10.110.7.1: icmp_seq=1 ttl=64 time=0.306 ms
64 bytes from 10.110.7.1: icmp_seq=2 ttl=64 time=0.055 ms

--- 10.110.7.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1006ms
rtt min/avg/max/mdev = 0.055/0.180/0.306/0.125 ms
# ping -c 1 10.110.1.1
ping: connect: Network is unreachable
# ip route add default via 10.110.7.1
# ping -c 2 10.110.1.1
PING 10.110.1.1 (10.110.1.1) 56(84) bytes of data.
64 bytes from 10.110.1.1: icmp_seq=1 ttl=63 time=0.839 ms
64 bytes from 10.110.1.1: icmp_seq=2 ttl=63 time=0.068 ms

--- 10.110.1.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1014ms
rtt min/avg/max/mdev = 0.068/0.453/0.839/0.385 ms
# ip -br addr
lo               UNKNOWN        127.0.0.1/8 
enp2s0@if41      UP             10.110.7.4/24 
# ip route
default via 10.110.7.1 dev enp2s0 
10.110.7.0/24 dev enp2s0 proto kernel scope link src 10.110.7.4 

Step-by-step check:

Moment	enp2s0	Routes	Ping gateway	Ping router
start	DOWN, no IP	none	Network is unreachable	(not tested)
after up and addr add	UP, 10.110.7.4/24	10.110.7.0/24	2 of 2, ttl=64	Network is unreachable
after route add default	same	+ default via 10.110.7.1	(already worked)	2 of 2, ttl=63

The ttl adds up: 64 when talking to the gateway (same segment) and 63 when reaching the router, because the gateway subtracts 1 when it forwards the packet. In the drawing:

Questions you might ask

Why does the first ping fail with Network is unreachable and not with a timeout? Because the workstation has no route at all: the kernel knows before sending anything that there’s no path, and ping doesn’t even send the packet. A long timeout would mean the packet goes out but nobody replies.

Does the order of link set up and addr add matter? Not for the IP: you can add it with the card down. But the network route only appears once the card is UP, so until then nothing works (that’s snippet 2 in the exercises).

Why isn’t the 10.110.7.0/24 route added by hand? Because the kernel creates it when you set the IP with its mask. That’s the proto kernel on the last line of ip route.

What exactly does the gateway do so the router replies? It has the net.ipv4.ip_forward=1 option, which lets it forward packets between its two cards, and the router knows how to get back to 10.110.7.0/24 through it. In the lab, a classmate sets that up (or you, at the gateway desk) with sysctl.

Try it yourself

• Remove the default gateway with ip route del default and check that the ping to the router goes back to Network is unreachable while the gateway one keeps working.
• Repeat the program with the IP 10.110.7.5/24. Does anything change in the outputs, apart from the IP?

Program 2: fix a misconfigured workstation

A classmate says their workstation “has no network”. Someone typed 10.110.70.4 instead of 10.110.7.4, and the default gateway with the 70 too. It has to be diagnosed with commands that only look, and then fixed without rebooting.

The idea

Diagnosis from the inside out:

1. ip -br addr: is the card UP? What IP and prefix does it have?
2. ip route: is there a default route? Is the gateway on my network?
3. ping the gateway: what error does it give?
4. ip route get: which path does the kernel choose for that destination?

Fix: remove the wrong IP, set the right one and add the default gateway again.

The code

# @lab mal
ip -br addr                              # 1. what IP do I have?
ip route                                 # 2. which way do I go out?
ping -c 2 10.110.7.1                     # 3. can I reach the gateway?
ip route get 10.110.7.1                  # 4. which path does the kernel choose to get there?
ip addr del 10.110.70.4/24 dev enp2s0    # remove the wrong IP
ip route                                 # the default route has gone with it
ip addr add 10.110.7.4/24 dev enp2s0
ip route add default via 10.110.7.1
ip route get 10.110.7.1
ping -c 2 10.110.1.1

The first four commands change nothing: they’re the diagnosis. The fix is the next three, and the last two check it worked.

Output

# ip -br addr
lo               UNKNOWN        127.0.0.1/8 
enp2s0@if37      UP             10.110.70.4/24 
# ip route
default via 10.110.70.1 dev enp2s0 
10.110.70.0/24 dev enp2s0 proto kernel scope link src 10.110.70.4 
# ping -c 2 10.110.7.1
PING 10.110.7.1 (10.110.7.1) 56(84) bytes of data.
From 10.110.70.4 icmp_seq=1 Destination Host Unreachable
From 10.110.70.4 icmp_seq=2 Destination Host Unreachable

--- 10.110.7.1 ping statistics ---
2 packets transmitted, 0 received, +2 errors, 100% packet loss, time 1020ms
pipe 2
# ip route get 10.110.7.1
10.110.7.1 via 10.110.70.1 dev enp2s0 src 10.110.70.4 uid 0 
    cache 
# ip addr del 10.110.70.4/24 dev enp2s0
# ip route
# ip addr add 10.110.7.4/24 dev enp2s0
# ip route add default via 10.110.7.1
# ip route get 10.110.7.1
10.110.7.1 dev enp2s0 src 10.110.7.4 uid 0 
    cache 
# ping -c 2 10.110.1.1
PING 10.110.1.1 (10.110.1.1) 56(84) bytes of data.
64 bytes from 10.110.1.1: icmp_seq=1 ttl=63 time=0.318 ms
64 bytes from 10.110.1.1: icmp_seq=2 ttl=63 time=0.057 ms

--- 10.110.1.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1007ms
rtt min/avg/max/mdev = 0.057/0.187/0.318/0.130 ms

What each part says:

• ip -br addr: the card is UP with 10.110.70.4/24. Its network runs from 10.110.70.0 to 10.110.70.255.
• ip route: the default gateway is 10.110.70.1, which is on that network (that’s why ip accepted it), but it doesn’t exist in the lab.
• The ping to 10.110.7.1 gives Destination Host Unreachable from the workstation itself (From 10.110.70.4): the gateway is outside the workstation’s network, so the packet is sent to the default gateway 10.110.70.1; the workstation asks on the wire who has that IP (ARP), nobody answers and delivery is given up. +2 errors, 100% packet loss: 2 sent, 0 received, 2 errors.
• ip route get 10.110.7.1 confirms it: via 10.110.70.1. It goes through the default gateway instead of directly.
• After ip addr del, ip route is empty: the default route left with the IP (its gateway was no longer reachable). That’s why it has to be added again.
• After the fix, ip route get 10.110.7.1 no longer has via: straight out through enp2s0. And the router replies.

The drawing shows the decision the kernel makes in each case: “is the destination on my network?”.

Questions you might ask

Why doesn’t it give Network is unreachable like in program 1? Because this workstation does have routes, including a default one. The kernel finds a path (the default gateway) and sends; the failure comes later, when nobody answers for the gateway. Different error, different cause: that’s why you have to read them.

Wasn’t it enough to change the default gateway? No: ip route add default via 10.110.7.1 would have given Nexthop has invalid gateway, because with the IP 10.110.70.4/24 the gateway isn’t on my network. The IP has to be fixed first.

Why does ip addr del need the /24? Because a card can have several IPs, and the one to remove is identified by address and prefix. If you write a different prefix, ip doesn’t find it and gives an error.

What if I don’t know which IP my desk gets? In the lab it follows a pattern (10.110.R.x, with R the desk number); elsewhere the administrator gives it to you, or a DHCP server does (BOOTPROTO=dhcp).

Try it yourself

• Before fixing anything, run ip route get 10.110.70.9. Is there a via? Why?
• Change the IP with ip addr flush dev enp2s0 instead of ip addr del. What difference would it make if the card had two IPs?

Program 3: from the ifcfg to the network, with a script

We’ve mastered the temporary side. Now, the permanent one: at boot, CentOS’s network service reads each ifcfg and configures the card. Since our lab isn’t CentOS, we write a script, apply-ifcfg.sh, that does the same with an ifcfg, and we test it with two files: the enp2s0 one (with ONBOOT=yes) and one for a second card with ONBOOT=no.

The idea

The ifcfg uses the KEY=value format, which is exactly the syntax of a bash assignment. So source loads it and every key becomes a variable ($IPADDR, $GATEWAY…). Then:

1. If ONBOOT isn’t yes, nothing is done (as at boot).
2. If BOOTPROTO is dhcp, the IP isn’t in the file: warn and exit.
3. Turn the mask into a prefix (255.255.255.0 → 24) and issue the three commands from program 1.

The code

The script:

#!/bin/bash
# apply-ifcfg.sh: does by hand what CentOS's network service does at boot with each ifcfg
# Usage: ./apply-ifcfg.sh ifcfg-enp2s0
source "$1"                          # each KEY=value line of the ifcfg becomes a variable

if [ "$ONBOOT" != "yes" ]; then      # ONBOOT=no: at boot, nobody touches this card
    echo "$DEVICE: ONBOOT=$ONBOOT, not brought up"
    exit 0
fi
if [ "$BOOTPROTO" = "dhcp" ]; then   # with DHCP the IP comes from a server (dhclient), not the file
    echo "$DEVICE: BOOTPROTO=dhcp, dhclient asks for the IP"
    exit 0
fi

prefix() {                           # 255.255.255.0 -> 24: adds up the bits set in each byte of the mask
    local n=0 b
    for b in ${1//./ }; do           # ${1//./ } turns the dots into spaces: four numbers
        case $b in
            255) n=$((n + 8)) ;; 254) n=$((n + 7)) ;; 252) n=$((n + 6)) ;; 248) n=$((n + 5)) ;;
            240) n=$((n + 4)) ;; 224) n=$((n + 3)) ;; 192) n=$((n + 2)) ;; 128) n=$((n + 1)) ;;
        esac                         # 0 adds nothing
    done
    echo $n
}

p=$(prefix "$NETMASK")
ip link set "$DEVICE" up                       # 1. bring the card up
ip addr add "$IPADDR/$p" dev "$DEVICE"         # 2. the IP WITH its prefix (without it, /32)
echo "$DEVICE: $IPADDR/$p"
if [ -n "$GATEWAY" ]; then                     # 3. the default route, only if there is a GATEWAY
    ip route add default via "$GATEWAY"
    echo "$DEVICE: default gateway $GATEWAY"
fi

And the session that uses it:

# @lab base
cat -n ifcfg-enp2s0
cat -n ifcfg-enp3s0
./apply-ifcfg.sh ifcfg-enp3s0          # ONBOOT=no
./apply-ifcfg.sh ifcfg-enp2s0          # ONBOOT=yes
ip -br addr
ip route
ping -c 2 router

The hardest piece is the prefix function. Zooming in:

prefix() {                           # 255.255.255.0 -> 24: adds up the bits set in each byte of the mask
    local n=0 b
    for b in ${1//./ }; do           # ${1//./ } turns the dots into spaces: four numbers
        case $b in
            255) n=$((n + 8)) ;; 254) n=$((n + 7)) ;; 252) n=$((n + 6)) ;; 248) n=$((n + 5)) ;;
            240) n=$((n + 4)) ;; 224) n=$((n + 3)) ;; 192) n=$((n + 2)) ;; 128) n=$((n + 1)) ;;
        esac                         # 0 adds nothing
    done
    echo $n
}

${1//./ } replaces every dot in the first argument with a space, so for walks through the four numbers. case adds up each one’s bits. Trace with 255.255.255.0:

Loop	b	Adds	n afterwards
1	255	8	8
2	255	8	16
3	255	8	24
4	0	0 (no case)	24

echo $n returns it and p=$(prefix "$NETMASK") picks it up: 24.

Output

# cat -n ifcfg-enp2s0
     1	DEVICE=enp2s0
     2	BOOTPROTO=none
     3	ONBOOT=yes
     4	IPADDR=10.110.7.4
     5	NETMASK=255.255.255.0
     6	GATEWAY=10.110.7.1
# cat -n ifcfg-enp3s0
     1	DEVICE=enp3s0
     2	BOOTPROTO=none
     3	ONBOOT=no
     4	IPADDR=10.110.9.4
     5	NETMASK=255.255.255.0
# ./apply-ifcfg.sh ifcfg-enp3s0
enp3s0: ONBOOT=no, not brought up
# ./apply-ifcfg.sh ifcfg-enp2s0
enp2s0: 10.110.7.4/24
enp2s0: default gateway 10.110.7.1
# ip -br addr
lo               UNKNOWN        127.0.0.1/8 
enp2s0@if45      UP             10.110.7.4/24 
# ip route
default via 10.110.7.1 dev enp2s0 
10.110.7.0/24 dev enp2s0 proto kernel scope link src 10.110.7.4 
# ping -c 2 router
PING router (10.110.1.1) 56(84) bytes of data.
64 bytes from router (10.110.1.1): icmp_seq=1 ttl=63 time=0.416 ms
64 bytes from router (10.110.1.1): icmp_seq=2 ttl=63 time=0.093 ms

--- router ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 0.093/0.254/0.416/0.161 ms

• With ifcfg-enp3s0, the script stops at the first check: ONBOOT=no, not brought up. It touches nothing, even though the file has an IP.
• With ifcfg-enp2s0 it sets up 10.110.7.4/24 (the mask converted to 24) and the default gateway 10.110.7.1.
• ip -br addr and ip route end up identical to the end of program 1: the file produced the same configuration as the commands by hand.
• ping -c 2 router uses the name from /etc/hosts and replies with ttl=63, as before.

In the drawing, each ifcfg line with the command it turns into.

Questions you might ask

Is it safe to source a file? Only if you trust it: source runs whatever is inside. An ifcfg is a root-owned file in /etc, so it’s fine; with any random downloaded file, never.

Why does the function use local? So that n and b don’t clobber variables in the rest of the script. Without local, every variable in bash is global.

What happens if the ifcfg has no GATEWAY? [ -n "$GATEWAY" ] is false and no default route is added: the card will only see its own network. That’s normal for a second card facing only an internal network.

Does the real service only do this? It does a lot more (DHCP, several IPs, extra routes, DNS in resolv.conf…), but for an ifcfg like the lab’s the result is this: link up, addr add and route add.

Try it yourself

• Change NETMASK to 255.255.255.192 in a copy of the ifcfg and check which prefix the script works out (count it yourself first: 8 + 8 + 8 + 2).
• Add the real BOOTPROTO=dhcp case to the script: make it run dhclient "$DEVICE" instead of exiting.

Common mistakes in this practice

• Testing only one ping: the gateway tests your IP and mask; something outside your network tests the default gateway. Do both.
• Mixing up the ping errors: Network is unreachable = your machine is missing a route; Destination Host Unreachable = there’s a route but whoever should receive it doesn’t answer.
• Fixing the default gateway before the IP: with the wrong IP, the right gateway gives Nexthop has invalid gateway.
• Forgetting that ip addr del takes the routes with it: you have to add the default route again.
• Spaces in the ifcfg: KEY=value, all together.
• Testing on your own laptop: use a practice machine or a virtual one.

Try it in your terminal

Every time you ping a machine on your network, yours needs its MAC, and asks for it with ARP. ip neigh shows that “address book” (the ARP table). As a normal user on the configured workstation:

# @lab conf
# @sergio
ip neigh                                 # the ARP table: empty
ping -c 1 gateway
ip neigh                                 # now it knows the gateway's MAC

The real session:

$ ip neigh
$ ping -c 1 gateway
PING gateway (10.110.7.1) 56(84) bytes of data.
64 bytes from gateway (10.110.7.1): icmp_seq=1 ttl=64 time=0.208 ms

--- gateway ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.208/0.208/0.208/0.000 ms
$ ip neigh
10.110.7.1 dev enp2s0 lladdr 08:00:27:a1:07:01 REACHABLE 

What to watch for:

1. At first ip neigh prints nothing: the workstation hasn’t talked to anyone yet.
2. ping -c 1 gateway resolves the name through /etc/hosts and replies in a single round trip.
3. Afterwards, ip neigh has an entry: the IP 10.110.7.1, which card it’s reached through (dev enp2s0) and its MAC (lladdr 08:00:27:a1:07:01).
4. REACHABLE means the entry is recent and reliable. After a while with no traffic it becomes STALE.
5. On your computer, ip neigh will show at least your home or university router: your default gateway.

Summary

• To get a network: card up, IP with a mask and a default gateway on your network; check each step with one ping inside and one outside your network.
• Network is unreachable points at your configuration (missing route); Destination Host Unreachable, at nobody answering on the local network.
• ip route get IP tells you which path the kernel picks: with via it goes through the default gateway, without via it goes direct.
• Removing an IP also deletes the routes that depended on it.
• An ifcfg is bash variables: at boot it becomes ip link set up, ip addr add and ip route add default, if ONBOOT=yes.
• ip neigh shows the MACs your machine has learnt.

Checklist: have you practised it?

• Have you run the three programs and checked the outputs?
• Have you tried the “Try it yourself” tasks without looking at the theory?
• Do you know which ping error to expect if the default gateway is missing and which if the IP is wrong?
• Can you explain why 10.110.7.1 is outside the 10.110.70.0/24 network?
• Can you work out the prefix of a mask like 255.255.255.192?
• Do you understand why ONBOOT=no makes the script leave the card alone?

If you answered yes to almost everything, you’re ready for the network configuration exercises, where I no longer give you the commands. See you there!

Next step: exercisesNetwork configuration on Linux — exercises with the workstation’s ID card, ifconfig commands translated and a generated ifcfg (and the /32 that left the card all alone)

Did this article help you? Here is how you can support SergioLearns →