Configure a second IP

Table of Contents

1. Introduction
2. Assign IP to the main network interface
3. Assign IP to incus container

1. Introduction

Sometimes we may need to use a second IP on the server. This is the case for example if we need to install two mail servers, since the mail ports (POP, IMAP, etc.) cannot be forwarded to two different applications, and cannot be proxied (like HTTP/HTTPS ports, for example).

These hetzner doc pages have some information about some possible network configurations:

This stack-exchange discussion has some more details.

We will see several ways of doing it.

2. Assign IP to the main network interface

Let’s assume that the network interface is called eth0, the public IP of the server is 101.102.103.104/24 (IP1), the gateway is 101.102.103.1 (GW1), the second IP is 201.202.203.204/24 (IP2), and the gateway of the second IP is 201.202.203.1 (GW2).

IF=eth0

IP1=101.102.103.104/24
GW1=101.102.103.1
NW1=101.102.103.0/24

IP2=201.202.203.204/24
GW2=201.202.203.1
NW2=201.202.203.0/24

2.1. Manually

While playing with the network configuration, there is a chance that we can do something wrong and lock ourselves outside the server. A reboot of the server will usually reset the correct configuration because the modifications that we are doing manually are not persistent and will be lost on reboot.

For this reason, in another terminal (in a tmux tab) I usually give a command like this: sleep 600 && reboot. It will reboot the server after 10 min, unless I stop it with "Ctrl+C".

In case I break the network configuration and I am not able to access the server anymore, the server will be rebooted after the timeout, and this will reset the original configuration. Otherwise, if everything goes well, I go and stop the timer with "Ctrl+C".

We can check the current network configuration with the commands:

ip address show
ip route show

Now, let’s add the second IP to the main network interface:

ip address add $IP2 dev $IF

ip addr
ip ro

curl icanhazip.com
curl --interface $IP2 icanhazip.com

All the packets that have IP2 as a destination now should arrive at our server. The problem is that when a replay is sent, it is going to be routed through GW1 (the gateway of the first IP). This is an asymmetric flow, which can fail for various reasons.

We can solve this problem by using different routing tables, based on the source of the outgoing packets. It is usually done like this:

ip route add $NW2 dev $IF table 1000
ip route add default via $GW2 dev $IF table 1000
ip route show table 1000

ip rule add from $IP2 lookup 1000

We are using another routing table, with number 1000.

The number 1000 is arbitrary and can be almost any number, except for the reserved values.

cat /etc/iproute2/rt_tables

#
# reserved values
#
255     local
254     main
253     default
0       unspec
#
# local
#
#1      inr.ruhep

In the table 1000 we are adding a route for the second network, and also a default route to GW2 (the second gateway).

The last command (ip rule add from $IP2 lookup 1000) is adding a rule that says: "For packets that have a source IP equal to IP2 (the second IP), use the routing table number 1000". This means that these packets will be routed via GW2 (the second gateway). The other packets (that have source IP1) will be handled by the default routing table, which normally sends them to GW1.

ip rule

0:      from all lookup local
32765:  from 201.202.203.204 lookup 1000
32766:  from all lookup main
32767:  from all lookup default

Let’s try to check that routing is done correctly:

ip route get from $IP1 to 8.8.8.8
ip route get from $IP2 to 8.8.8.8

2.2. Permanently

As mentioned in the previous section, the configurations made with the ip command will be lost on a reboot. To make them persistent, we can edit the file /etc/network/interfaces so that it looks like this:

auto eth0
iface eth0 inet static
    address 101.102.103.104
    netmask 255.255.255.0
    gateway 101.102.103.1
    # add the second ip
    up ip addr add 201.202.203.204/24 dev eth0
    down ip addr del 201.202.203.204/24 dev eth0
    # add routes to table 1000
    post-up ip route add 201.202.203.0/24 dev eth0 table 1000
    post-up ip route add default via 201.202.203.1 dev eth0 table 1000
    post-down ip route del 201.202.203.0/24 dev eth0 table 1000
    post-down ip route del default via 201.202.203.1 dev eth0 table 1000
    # for source 201.202.203.204, lookup on routing table 1000
    post-up ip rule add from 201.202.203.204 lookup 1000
    post-down ip rule del from 201.202.203.204 lookup 1000

cat <<EOF
auto $IF
iface $IF inet static
    address $IP1
    gateway $GW1
    # add the second ip
    up ip addr add $IP2 dev $IF
    down ip addr del $IP2 dev $IF
    # add routes to table 1000
    post-up ip route add $NW2 dev $IF table 1000
    post-up ip route add default via $GW2 dev $IF table 1000
    post-down ip route del $NW2 dev $IF table 1000
    post-down ip route del default via $GW2 dev $IF table 1000
    # for source $IP2, lookup on routing table 1000
    post-up ip rule add from $IP2 lookup 1000
    post-down ip rule del from $IP2 lookup 1000
EOF

Activate the new configuration:

systemctl restart networking

Check:

ip address show
ip route show
ip route show table 1000

curl icanhazip.com
curl --interface $IP2 icanhazip.com

ip route get from $IP1 to 8.8.8.8
ip route get from $IP2 to 8.8.8.8

After a reboot, the server should still be configured with two IPs.

2.3. Test with netcat

We can also use netcat for testing:

On the server, open the port 12345/tcp on the firewall:

firewall-cmd --zone=public --add-port=12345/tcp
firewall-cmd --zone=public --list-all

On the server, start nc to listen on the $IP2 and the port 12345:
```
nc -l -s $IP2 -p 12345
```
Outside the server, start nc to connect to $IP2 and the port 12345:
```
nc $IP2 12345
```
Make sure that both nc commands can communicate with each-other (if you type anything on one side and press Enter, it should appear on the other side).

Stop the nc commands (with "Ctrl+C") and close the port on the firewall:

firewall-cmd --zone=public --remove-port=12345/tcp
firewall-cmd --zone=public --list-all

2.4. Port forwarding

Let’s say that we want to forward the TCP ports "25, 465, 587, 110, 143, 993, 995, 5222, 6071" to the incus container "carbonio". We can do it like this:

IP2=201.202.203.204
CONTAINER_IP=10.210.64.201

incus network forward --help

incus network forward create incusbr0 $IP2
incus network forward list incusbr0
incus network forward show incusbr0 $IP2

incus network forward port add incusbr0 \
    $IP2 tcp 25,465,587,110,143,993,995,5222,6071 \
    $CONTAINER_IP
incus network forward show incusbr0 $IP2

Test port forwarding

We can use netcat to test that ports are forwarded correctly. On the server run:

incus exec carbonio -- nc -l -p xyz

Outside the server run:

nc $IP2 xyz

3. Assign IP to incus container

It is also possible to assign the second IP directly to an incus container, using an ipvlan network interface, instead of a bridged one. More details here.

First of all, make sure that $IP2 is removed from the main network interface of the server ($IF). For this, edit /etc/network/interfaces and remove any related configuration lines. Then restart networking: systemctl restart networking

Next, lets create a container:

incus init images:debian/12 test1 \
    -c security.nesting=true \
    -c security.syscalls.intercept.mknod=true \
    -c security.syscalls.intercept.setxattr=true

incus ls
incus config show test1 --expanded
incus config device show test1

We notice that it has a bridged network interface, which is the default one. Let’s replace (override) it with an ipvlan one:

incus config device add test1 eth0 nic \
    nictype=ipvlan \
    mode=l2 \
    parent=$IF \              (1)
    name=eth0 \               (2)
    ipv4.address=$IP2 \
    ipv4.gateway=$GW2

incus config show test1 --expanded
incus config device show test1

1	Name of the interface on the host.
2	Name of the interface on the container.

Now, let’s start the container and make sure that it has a proper network configuration:

incus start test1
incus shell test1

cat <<EOF > /etc/systemd/network/eth0.network
[Match]
Name=eth0

[Address]
Address=$IP2

[Route]
Gateway=$GW2

[Network]
DHCP=no
DNS=8.8.8.8
EOF

cat /etc/systemd/network/eth0.network
systemctl restart systemd-networkd

ip addr
ip ro
ping 8.8.8.8

exit

We can test with ping and with nc that we are able to access the container from outside, using the second public IP.