Homelab setup v6.5

This isn’t a significant update, but given its technical aspects, I’ve created a post that might be useful to someone.

What’s new:

I upgraded the nanoPi R4S router to the subsequent model nanoPi R5S that ha 2.5 Gb/s ports and I also used the SFP+ ports to connect it to the switch.
I configured all the network components using LAGs
I removed two Raspberry PIs 4 (one used for Homebridge and the other for Pi-Hole) and replaced both with one device (the previously router nanoPi R4S) with USB3 boot and one IP for port
I added the dockcheck script to auto-update the Docker containers and receive a push notification via Pushover

The hardware involved here, and now connected via LAG, is:

router (nanoPi R5S)
Access Point (Netgear WAX206)
server Grafana (Beelinq EQR6 miniPC)
server Archive (Beelinq EQR6 miniPC)

💡

I wanted to write this post also because I found very little documentation on configuring LAGs on OpenWrt, the USB3 boot on the nanoPi R4S and the routing through two interfaces with the same subnet, one acting as the DNS of the other, on the same machine.

To understand how my network is configured and where I have implemented the LAGs, and which were the two Raspberry PI I replace with the nanoPi R4S, I suggest to give a glance at the “v6” post:

Now > the details of this last update.
Aug 01, 2025 > My home setup - v6
Jul 8, 2024 > My network home setup - v5.5
- Sep 13, 2024 - cPiArtFrame - The PiArtFrame in C++ with Julia fractal
Aug 20, 2024 > My network home setup - v5.0
Sep 14, 2023 > My network home setup - v4.5
Jan 5, 2023 > My network home setup - v4.0
Aug 14, 2022 > My network home setup - v3.0
- Dec 12, 2022 > Some custom push notifications that improved my life
- Dec 19, 2022 > Retrieve a scheduled report of your Grafana dashboards via mail
Jun 02, 2022 > My network home setup - v2.0
Jan 14, 2022 > My hardware and software setup for a new house

The photos of the current setup

setup

setup1

lag

Here the 2 ethernet cables per device

What’s a LAG?

LAG, aka Link Aggregation Group (or bonding) means joining two network interfaces (ports) and making them appear as one to the switch. The switch communicates with the client using the 802.1AX standard across both ports, treating them as a single logical link. Then, a hash algorithm is used to decide on which physical port the traffic of each flow will be forwarded.

This effectively doubles the available bandwidth (1 Gb/s + 1 Gb/s = 2 Gb/s), but the maximum speed of a single flow remains limited to 1 Gb/s. It’s like having two highway lanes, both with a 130 km/h speed limit: no car can go faster, but more cars can travel in parallel.

In practice, if one client communicates over port 1 and saturates its 1 Gb/s bandwidth, another client can communicate over port 2, where the bandwidth is still completely free.

It is particularly useful when there are many clients or services generating high traffic, as the load can be distributed intelligently across the aggregated links.

Regarding this, there are two main ways to configure bond hashing: layer2+3 and layer3+4.

With layer2+3 traffic is distributed based on the source/destination MAC addresses and IPs. This means that all flows to or from the same host will usually go through the same link.
With layer3+4 the hashing also includes the TCP/UDP ports. This allows balancing per connection (flow), so different services or applications communicating with the same host can be spread across multiple links.

I prefer to use layer3+4, as I have few different clients but many opened connections.

To make a simple example: if a Prometheus server (IP 192.168.1.6) scrapes metrics from different exporters on the router, all with the same IP but on different ports (192.168.1.2:2113, 192.168.1.2:9630, etc…) by using layer3+4, these flows can be distributed across the aggregated interfaces, even though they all go to the same host.

In the case of an Access Point, for example, a client on the “main” VLAN (i.e. an iPad) can stream a movie using one link, while another 10–15 clients on the IoT VLAN can transfer data over the second link. This provides two transmission channels, helping to avoid slowdowns, queued packets, or packet drops. On the AP this is usually where I notice the greatest improvements, and also the highest number of distributed flows (as shown by the switch statistics).

Another useful example is a server backing up to a NAS: if one connection saturates a link, opening another connection from a Mac Mini can still use the other link, so traffic gets distributed and tasks continue without slowdowns.

Another positive effect of bonding is that if a cable gets disconnected or a port fails, the connection will still work through the other link. It acts as a small failover system, even though in a home environment the chance of a single port failure is quite low.

I hope this explains the LAG easily.

The nanoPi R5S

The FriendlyElec NanoPi R5S is the new router I upgraded to from the R4S, as it comes with 2.5 Gb/s Ethernet ports, I can connect it to the MikroTik CSS318-16G-2S+IN switch via the SFP+ ports in a LAG, creating up to 5 Gb/s of aggregated bandwidth across the two links.

setup2

SFP+ to RJ45

These are the SFP+ to RJ45 connectors needed to connect to the switch at 2.5+2.5 Gb/s. I got them on Aliexpress for ~€30 each and they perfectly negotiate the speed. (The original MikroTik ones cost ~€70 each)

sfp

Reinstalling OpenWrt

First, I wanted to reinstall OpenWrt while keeping all my settings, but there is a problem between R4S and R5S because the network interface names are different, since the R5S has one more port.

So, to start I launched a:

opkg list-installed | awk '{ printf "%s ",$1 }'

In order to list my actual opkg packages, then I copied the list and generated a new firmware for the R5S using the very handy OpenWrt Firmware Selector to be flashed to the microSD.

Then I booted the R5S and I immediately found it at 192.168.1.1.

💡

I use 192.168.1.2 as the default IP for my router, so the .1 is always free for this purpose

After that, I downloaded the backup of my old nanoPi R4S, opened the tar.gz, removed the network configuration file and repackaged the archive without the network configuration file.

PS: macOS has some issue with tar.gz packages because it creates lots of hidden ._ files, so I hade to delete them before packing it, using:

find etc -name "._*" -delete
find etc -name ".DS_Store" -delete
COPYFILE_DISABLE=1 tar -czvf r4s-to-r5s-backup.tar.gz etc

Then I restored the backup on the R5S, disconnected the R4S, entered into the R5S and I set the default IP 192.268.1.2 on it.

Now all is the same as the old R4S, except for the network setup, so I started to build it.

LAG on the nanoPi R5S

Now the R5S configuration is the same as the R4S, except for the network interfaces of course, so it’s time to start configuring the bound.

To use the network interfaces with bonding, the first step was installing the package kmod-bonding.

ℹ️

Check if the file /etc/modules.d/40-bonding is present, this ensure the module will be loaded at boot

Then a reboot is required for it to appear as an option in LuCI (I have no idea why, reload LuCi alone, won’t work).

bond-device

You can configure the bond options (like the layer3+4 or LACP rate under the “Adavanced device options” tab

Once that was done, I created the bond0 including the two new ports: eth1 and eth2.

After setting up the bond, I took the old network configuration file and copied it over the entire previous configuration, obviously adjusting the VLAN tags, which would no longer be tagged on eth1:t but instead on bond0:t.

So my new /etc/config/network file has become this:

config interface 'loopback'
	option device 'lo'
	option proto 'static'
	list ipaddr '127.0.0.1/8'

config globals 'globals'
	option ula_prefix 'fd3d:ec9d:f74b::/48'
	option packet_steering '1'

config device
	option name 'br-lan'
	option type 'bridge'
	option ipv6 '0'
	list ports 'bond0'

config device
	option name 'eth1'
	option macaddr '...'

config device
	option name 'eth2'
	option macaddr '…'

config interface 'lan'
	option device 'br-lan.1'
	option proto 'static'
	option ipaddr '192.168.1.2'
	option netmask '255.255.255.0'
	list DNS '192.168.1.4'
	option delegate '0'

config device
	option name 'eth0'
	option macaddr '...'

config interface 'WAN'
	option device 'eth0'
	option proto 'pppoe'
	option username 'aliceadsl'
	option password 'aliceadsl'
	option peerdns '0'
	list DNS '192.168.1.4'
	option ipv6 '0'

config interface 'wan6'
	option device 'eth0'
	option proto 'dhcpv6'

config device
	option type 'bonding'
	option name 'bond0'
	list ports 'eth1'
	list ports 'eth2'
	option policy '802.3ad'
	option all_ports_active '1'
	option xmit_hash_policy 'layer3+4'
	option ipv6 '0'
	option lacp_rate 'fast'
	option ad_select 'stable'

config interface 'wg0'
	option proto 'wireguard'
	option private_key '...'
	option listen_port '51820'
	list addresses '10.4.0.1/32'
	option delegate '0'

config wireguard_wg0
	option public_key '...'
	list allowed_ips '10.4.0.2/32'
	option route_allowed_ips '1'
	option description 'iPhone'

config wireguard_wg0
	option public_key '...'
	list allowed_ips '10.4.0.3/32'
	option route_allowed_ips '1'
	option description 'iPad Pro'

config wireguard_wg0
	option public_key '...'
	option route_allowed_ips '1'
	list allowed_ips '10.4.0.4/32'
	option description 'MacBook Air'

config interface 'modem'
	option proto 'static'
	option device 'eth0'
	option ipaddr '192.168.2.2'
	option netmask '255.255.255.0'

config bridge-VLAN
	option device 'br-lan'
	option VLAN '1'
	list ports 'bond0'

config bridge-VLAN
	option device 'br-lan'
	option VLAN '10'
	list ports 'bond0:t'

config bridge-VLAN
	option device 'br-lan'
	option VLAN '20'
	list ports 'bond0:t'

config bridge-VLAN
	option device 'br-lan'
	option VLAN '50'
	list ports 'bond0:t'

config interface 'guest'
	option proto 'static'
	option device 'br-lan.20'
	option ipaddr '192.168.20.1'
	option netmask '255.255.255.0'
	list DNS '192.168.1.4'

config interface 'iot'
	option proto 'static'
	option device 'br-lan.50'
	option ipaddr '192.168.50.1'
	option netmask '255.255.255.0'
	list DNS '192.168.1.4'

config device
	option name 'br-lan.20'
	option type '8021q'
	option ifname 'br-lan'
	option vid '20'
	option ipv6 '0'
	option acceptlocal '1'

config device
	option name 'br-lan.1'
	option type '8021q'
	option ifname 'br-lan'
	option vid '1'
	option ipv6 '0'
	option acceptlocal '1'

config device
	option name 'br-lan.50'
	option type '8021q'
	option ifname 'br-lan'
	option vid '50'
	option acceptlocal '1'
	option ipv6 '0'

After that, I changed the switch (MikroTik CSS318-16G-2S+IN) ports to “active” unde LAG tab and I assigned the VLANs to both ports of the LAG.

vlans

After rebooting the router the bond came up, but only on one port on the switch side was inside the LAG.

So, while trying to figure out why, I tweaked a few simple settings and noticed that as soon as the network interfaces were reloaded, the bond came back up on both ports on the switch.

So I figured that at boot the interfaces were coming up too quickly to form the bond, and I added a 5 second delay to OpenWrt startup:

sleep 5
/etc/init.d/network restart

This way, the bond comes up fine every boot without any issue.

switch-lag

Checking it on the R5S:

root@R5S:~# cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v6.6.93

Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer3+4 (1)
MII Status: up
MII Polling Interval (ms): 0
Up Delay (ms): 0
Down Delay (ms): 0
Peer Notification Delay (ms): 0

802.3ad info
LACP active: on
LACP rate: fast
Min links: 0
Aggregator selection policy (ad_select): stable
System priority: 65535
System Mac address: 02:72:83:dc:9f:2e
Active Aggregator Info:
        Aggregator ID: 1
        Number of ports: 2
        Actor Key: 11
        Partner Key: 3
        Partner Mac Address: f4:1e:57:95:ff:00

Slave Interface: eth1
MII Status: up
Speed: 2500 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 02:72:83:dc:9f:2e
Slave queue ID: 0
Aggregator ID: 1
Actor Churn State: none
Partner Churn State: none
Actor Churned Count: 0
Partner Churned Count: 0
details actor lacp pdu:
    system priority: 65535
    system Mac address: 02:72:83:dc:9f:2e
    port key: 11
    port priority: 255
    port number: 1
    port state: 63
details partner lacp pdu:
    system priority: 32768
    system Mac address: f4:1e:57:95:ff:00
    oper key: 3
    port priority: 32768
    port number: 17
    port state: 61

Slave Interface: eth2
MII Status: up
Speed: 2500 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 02:72:83:dc:9f:2e
Slave queue ID: 0
Aggregator ID: 1
Actor Churn State: none
Partner Churn State: none
Actor Churned Count: 0
Partner Churned Count: 0
details actor lacp pdu:
    system priority: 65535
    system Mac address: 02:72:83:dc:9f:2e
    port key: 11
    port priority: 255
    port number: 2
    port state: 63
details partner lacp pdu:
    system priority: 32768
    system Mac address: f4:1e:57:95:ff:00
    oper key: 3
    port priority: 32768
    port number: 18
    port state: 61

from microSD to eMMC and LEDs

Once I was sure everything was working, the final step was switching from the microSD to the eMMC installed inside the nanoPi R5S.

It’s fast and easy: I uploaded my packages (including the new kmod-bonding) to the OpenWrt firmware selector and downloaded the generated squashfs file (and of course, I also made a backup of my current configurations).

Then I powered off the R5S, removed the microSD, rebooted, changed my Mac’s IP to the 192.168.2.x subnet (because FriendlyWrt uses 192.168.2.1 as the default IP), logged into FriendlyWrt LuCI and from there flashed the OpenWrt squashfs firmware using “eMMC Tools” (under “System” in LuCi).

After rebooting, everything worked fine, and it even felt slightly more responsive (the boot was noticeably faster). But there was still one last thing: the LEDs.

I love the Ethernet port LEDs… but the front LEDs of the R5S stopped working after uploading the old backup from the R4S and creating the bond. So I checked the settings and modified the file to use the new eth1 and eth2 ports:

config led 'led_WAN'
	option name 'WAN'
	option sysfs 'green:WAN'
	option trigger 'netdev'
	option mode 'link tx rx'
	option dev 'eth0'

config led 'led_eth1'
        option name 'LAN1'
        option sysfs 'green:lan-1'
        option trigger 'netdev'
        option dev 'eth1'
        option mode 'link tx rx'

config led 'led_eth2'
        option name 'LAN2'
        option sysfs 'green:lan-2'
        option trigger 'netdev'
        option dev 'eth2'
        option mode 'link tx rx'

config led
	option sysfs 'red:sys'
	option name 'SYS'
	option trigger 'heartbeat'

Restarted the LEDs using: /etc/init.d/led restart and here we are, the front LEDs of LAN1 and LAN2 started blinking again just the way I like =)

LAG on the Access Point

My access point is a Netgear WAX206 that has 4 LAN port, so two of them can be used as LAG.

Premise: The access point carries 3 VLANs over the wireless network: Main (5 GHz), IoT (2.4 GHz) and Guest (5 GHz), and the AP is connected to the switch on ports eth3 and eth4.

First, I installed the kernel module kmod-bonding also here.

Then I created the bond1 which includes port lan3 and lan4 (I called it bond1 just to remember that is the access point and not the router)

config device
	option type 'bonding'
	option name 'bond1'
	list ports 'lan3'
	list ports 'lan4'
	option policy '802.3ad'
	option xmit_hash_policy 'layer3+4'
	option all_ports_active '1'
	option ipv6 '0'

After, I included tbe bond1 inside the LAN bridge with the VLANs:

config device
	option name 'br-lan'
	option type 'bridge'
	option VLAN_filtering '1'
	option ipv6 '0'
	list ports 'bond1'

and then the VLANs obviously now use bond1 as the referral interface:

config bridge-VLAN
	option device 'br-lan'
	option VLAN '1'
	list ports 'bond1:u'

config bridge-VLAN
	option device 'br-lan'
	option VLAN '10'
	list ports 'bond1:t'

config bridge-VLAN
	option device 'br-lan'
	option VLAN '20'
	list ports 'bond1:t'

config bridge-VLAN
	option device 'br-lan'
	option VLAN '50'
	list ports 'bond1:t'

In the end, the LAN interface, which includes the 2 ports lan3 and lan4 and are obviously seen as a single port with one IP:

config interface 'lan'
	option proto 'static'
	option device 'br-lan.1'
	option ipaddr '192.168.1.3'
	option netmask '255.255.255.0'
	option gateway '192.168.1.2'
	list dns '192.168.1.4'

Tip: To avoid losing the connection to the AP if something goes wrong, you can configure another port outside the bridge with a different address, so you can always connect to it if the bond doesn’t come up. Here’s my port1 used as a backup connection in this example:

config device 'lan1dev'
        option name 'lan1'
        option ipv6 '0'

config interface 'backup'
        option device 'lan1'
        option proto 'static'
        option ipaddr '192.168.1.33'
        option netmask '255.255.255.0'

💪

You can create the same LAG using all 4 ports if you want to try ludicrous mode, but is not guaranteed to see any improvement

Now i configured the ports where the two switch cables end to “active” also here, and after a reboot of the AP, the LAG with its trunk was automatically formed switch side.

Back to the AP and check if the LAG works fine:

root@WAX206:~# cat /proc/net/bonding/bond1
Ethernet Channel Bonding Driver: v6.6.93

Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer3+4 (1)
MII Status: up
MII Polling Interval (ms): 0
Up Delay (ms): 0
Down Delay (ms): 0
Peer Notification Delay (ms): 0

802.3ad info
LACP active: on
LACP rate: slow
Min links: 0
Aggregator selection policy (ad_select): stable
System priority: 65535
System MAC address: 80:cc:9c:eb:8d:37
Active Aggregator Info:
        Aggregator ID: 2
        Number of ports: 2
        Actor Key: 9
        Partner Key: 2
        Partner Mac Address: f4:1e:57:95:ff:00

Slave Interface: lan3
MII Status: up
Speed: 1000 Mb/s
Duplex: full
Link Failure Count: 0
Permanent HW addr: 80:cc:9c:eb:8d:37
Slave queue ID: 0
Aggregator ID: 2
Actor Churn State: monitoring
Partner Churn State: monitoring
Actor Churned Count: 0
Partner Churned Count: 0
details actor lacp pdu:
    system priority: 65535
    system Mac address: 80:cc:9c:eb:8d:37
    port key: 9
    port priority: 255
    port number: 1
    port state: 61
details partner lacp pdu:
    system priority: 32768
    system Mac address: f4:1e:57:95:ff:00
    oper key: 2
    port priority: 32768
    port number: 2
    port state: 61

Slave Interface: lan4
MII Status: up
Speed: 1000 Mb/s
Duplex: full
Link Failure Count: 0
Permanent HW addr: 80:cc:9c:eb:8d:37
Slave queue ID: 0
Aggregator ID: 2
Actor Churn State: monitoring
Partner Churn State: monitoring
Actor Churned Count: 0
Partner Churned Count: 0
details actor lacp pdu:
    system priority: 65535
    system Mac address: 80:cc:9c:eb:8d:37
    port key: 9
    port priority: 255
    port number: 2
    port state: 61
details partner lacp pdu:
    system priority: 32768
    system Mac address: f4:1e:57:95:ff:00
    oper key: 2
    port priority: 32768
    port number: 1
    port state: 61

LAG on the servers

My servers are two miniPCs (see the previous post) running Alpine Linux, and configuring the LAG here was simpler than on the OpenWrt devices.

First, I enabled and loaded the module bonding

echo bonding >> /etc/modules
modprobe bonding

After, I set the parameters of the module in /etc/modprobe.d/bonding.conf with these simple settings:

options bonding mode=802.3ad miimon=100 lacp_rate=1 xmit_hash_policy=layer3+4

And inside the file /etc/network/interfaces i configured the bond on the two interfaces eth0 and eth1:

auto lo
iface lo inet loopback

allow-hotplug eth0
iface eth0 inet manual

allow-hotplug eth1
iface eth1 inet manual

auto bond0
iface bond0 inet static
    address 192.168.1.7
    netmask 255.255.255.0
    gateway 192.168.1.2

    pre-up IP link add bond0 type bond || true
    post-up IP link set eth0 master bond0 || true
    post-up IP link set eth1 master bond0 || true

P.S. I had to add the pre-up and post-up scripts because, otherwise, Alpine would not correctly generate the bond on reboot. I tried various options but it’s the only one that works. If you want a clean /etc/network/interfaces file, you can also use the /etc/network/if-pre-up.d/ to create the bond and add the interfaces.

On the switch, I set the ports to “active”, and upon rebooting the two servers, the trunk immediately came up. Checking it on the server side:

archive:~# cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v6.12.44-0-lts

Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer3+4 (1)
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0
Peer Notification Delay (ms): 0

802.3ad info
LACP active: on
LACP rate: fast
Min links: 0
Aggregator selection policy (ad_select): stable
System priority: 65535
System Mac address: e8:ff:1e:dc:fa:64
Active Aggregator Info:
        Aggregator ID: 2
        Number of ports: 2
        Actor Key: 9
        Partner Key: 2
        Partner Mac Address: f4:1e:57:95:ff:00

Slave Interface: eth0
MII Status: up
Speed: 1000 Mb/s
Duplex: full
Link Failure Count: 0
Permanent HW addr: e8:ff:1e:dc:fa:64
Slave queue ID: 0
Aggregator ID: 2
Actor Churn State: none
Partner Churn State: none
Actor Churned Count: 0
Partner Churned Count: 0
details actor lacp pdu:
    system priority: 65535
    system Mac address: e8:ff:1e:dc:fa:64
    port key: 9
    port priority: 255
    port number: 1
    port state: 63
details partner lacp pdu:
    system priority: 32768
    system Mac address: f4:1e:57:95:ff:00
    oper key: 2
    port priority: 32768
    port number: 13
    port state: 61

Slave Interface: eth1
MII Status: up
Speed: 1000 Mb/s
Duplex: full
Link Failure Count: 0
Permanent HW addr: e8:ff:1e:dc:fa:65
Slave queue ID: 0
Aggregator ID: 2
Actor Churn State: none
Partner Churn State: none
Actor Churned Count: 0
Partner Churned Count: 0
details actor lacp pdu:
    system priority: 65535
    system Mac address: e8:ff:1e:dc:fa:64
    port key: 9
    port priority: 255
    port number: 2
    port state: 63
details partner lacp pdu:
    system priority: 32768
    system Mac address: f4:1e:57:95:ff:00
    oper key: 2
    port priority: 32768
    port number: 14
    port state: 61

Testing

This is a test made simultaneously from the two servers to the R5S using: iperf3 -c 192.168.1.2 -p 5201 -P 8 -t 30 and iperf3 -c 192.168.1.2 -p 5202 -P 8 -t 30

iperf3multistreams-multiclients

If you try the same wihout a LAG, obviously, the bandwith will be splitted between the two client (about 450Mb/s each) , with a LAG, both the servers can receive a full gigabit speed.

And a test from one server to the R5S using 8 flows and sequential ports to saturate the LAG: iperf3 -c 192.168.1.2 -p 5201 -P 8 -t 10 --cport 3000

iperf3-single

Another benefit of the LAG, is having a medium lower ping, because by pinging every 30 seconds, many times an interface can be sature and with small slowdowns, by having two interfaces where the packet can be routed the latency will generally be lower on la long and constant pinging.

I indeed noticed it over the long term.

ping

Ping over a week

Another advantage I’ve noticed (which I hope is not a placebo effect) is a very silly one: when I come home and press the “I’m home” scene button in Home app, instead of the 5/6 lights turning on with a few delay (tenths of a second obviously), they all turn on together simultaneously. Nice.

Heat issue on the switch

On the switch, after inserting the SFP+ connectors and the LAGs, I noticed that the temperatures ramped up by over 10 degrees (it’s perfectly normal) so I decided to “add” some heatsinks to to the chips, which only has passive heatsinks.

switch-heat

Obviously, the improvement is minimal, 6-8 °C less, but it allows the switch not to exceed 75 °C even when the air conditioner is off (before this modification it reached about 80 °C). The best option would be to find push-pin heatsinks with the same pitch only taller and larger, something like this.

heatsinks-temp

From two RPi to a single nanoPi R4S

Unsure of what to do with the old nanoPi R4S, I decided to repurpose it as a server for Pi-Hole and Homebridge.

setup3

Before, the two services were running on two Raspberry Pi 4 (2GB) booting from SSD via USB3, but I decided to streamline the hardware and use a single device instead of two: the nanoPi R4S with root on SSD via USB3. For the following reasons:

the nanoPi R4S is more powerful than a Raspberry Pi 4 (comparison: BCM2711 vs Rockchip RK3399 [cpubenchmark.net] by PassMark Software)
the nanoPi R4S has two “real” Ethernet ports, while the Raspberry Pi 4 has one port shared with the USB controller
a single nanoPi R4S consumes less than two Raspberry Pis with 2 SSDs
it’s a device designed specifically to act as a network appliance

I decided to use one Ethernet port per service, which caused me some issues with TCP packets routing, since it also acts as a DNS server. But I’ll dig into that later, now I want to describe all the issues I encountered using the external USB3 drive as root.

I used the usual DietPI distro, which already includes a build for the R4S. So I flashed the microSD and, after inserting and booting, the R4S was immediately reachable on the network. No problem.

I then intend to use an external USB 3 drive to circumvent the microSD wear and slowness. But…

USB3, compatibility issues:

I tested 2 different USB3 enclosures, that was working perfectly on the Raspberry Pi, but the R4S only recognized the devices as USB2. I have no idea why:

root@R4S:~# lsusb -t
/:  Bus 001.Port 001: Dev 001, Class=root_hub, Driver=xhci-hcd/1p, 480M
    |__ Port 001: Dev 002, If 0, Class=Mass Storage, Driver=uas, 480M
/:  Bus 002.Port 001: Dev 001, Class=root_hub, Driver=ohci-platform/1p, 12M
/:  Bus 003.Port 001: Dev 001, Class=root_hub, Driver=ohci-platform/1p, 12M
/:  Bus 004.Port 001: Dev 001, Class=root_hub, Driver=ehci-platform/1p, 480M
/:  Bus 005.Port 001: Dev 001, Class=root_hub, Driver=xhci-hcd/1p, 5000M
/:  Bus 006.Port 001: Dev 001, Class=root_hub, Driver=ehci-platform/1p, 480M
/:  Bus 007.Port 001: Dev 001, Class=root_hub, Driver=xhci-hcd/1p, 480M
/:  Bus 008.Port 001: Dev 001, Class=root_hub, Driver=xhci-hcd/1p, 5000M

Same for both USB3 ports and for both SATA-USB3 adapters.

Finally, with the third (and last one I have) SATA-USB3 adapter, it was detected as USB3:

root@R4S:~# lsusb -t
/:  Bus 001.Port 001: Dev 001, Class=root_hub, Driver=xhci-hcd/1p, 480M
/:  Bus 002.Port 001: Dev 001, Class=root_hub, Driver=ohci-platform/1p, 12M
/:  Bus 003.Port 001: Dev 001, Class=root_hub, Driver=ehci-platform/1p, 480M
/:  Bus 004.Port 001: Dev 001, Class=root_hub, Driver=ohci-platform/1p, 12M
/:  Bus 005.Port 001: Dev 001, Class=root_hub, Driver=ehci-platform/1p, 480M
/:  Bus 006.Port 001: Dev 001, Class=root_hub, Driver=xhci-hcd/1p, 5000M
    |__ Port 001: Dev 002, If 0, Class=Mass Storage, Driver=usb-storage, 5000M
/:  Bus 007.Port 001: Dev 001, Class=root_hub, Driver=xhci-hcd/1p, 480M
/:  Bus 008.Port 001: Dev 001, Class=root_hub, Driver=xhci-hcd/1p, 5000M

Now, on the nanoPi R4S it’s not possible to boot from USB because, when it boots, it searches for the U-Boot only on the microSD, unfortunately.

When DietPI is installed, using DietPI-setting under Update MMC bootloader : Flash current U-Boot to /dev/sda it’s possible to flash the bootloader onto the disk, but the R4S won’t start anyway without the microSD.

So I had to follow these steps:

Formatted the USB disk as ext4,

Then I copied the microSD onto the disk:

rsync -aAXH --exclude={"/proc/*","/sys/*","/dev/*","/run/*","/mnt/*","/tmp/*"} / /mnt/ssd

After I edited the file /boot/dietpiEnv.txt in order to point the rootdev to the exterternal drive:

rootdev=/dev/sda1
rootfstype=ext4

And also /etc/fstab to point to the external drive’s UUID:

UUID=c07600d2-4da9-49d2-83df-fb7e8adaf71b / ext4 noatime,lazytime,rw 0 1

(To read the UUID just use lsblk -f)

Now I rebooted and checked if the system was running from the external drive:

root@R4S:~# mount | grep ' / '
/dev/sda1 on / type ext4 (rw,noatime,lazytime)

All fine, just a speedtest to confirm:

root@R4S:~# hdparm -t /dev/sda

/dev/sda:
 Timing buffered disk reads: 776 MB in  3.00 seconds = 258.59 MB/sec

Now that DietPI is running fine from the external drive, I moved on to the network configuration. Remember that I wanted to use a different IP per port, with one service (Pi-Hole) assigned to one port (eth0) and the other (Homebridge) to the other (eth1).

To to this I configured the /etc/network/interfaces in this way:

root@R4S:~# cat /etc/network/interfaces
# /etc/network/interfaces

source interfaces.d/*

auto lo
iface lo inet loopback

# Pi-hole — table 100
auto eth0
iface eth0 inet static
    address 192.168.1.4
    netmask 255.255.255.0
    gateway 192.168.1.2
    dns-nameservers 192.168.1.4

# Homebridge — table 101
auto eth1
iface eth1 inet static
    address 192.168.1.5
    netmask 255.255.255.0
    dns-nameservers 192.168.1.4

Restarted the R4S and everything was working. Then, to assign an IP to each service, it’s easy to do it from each service’s webUI.

The problem is that the Linux kernel doesn’t play well with two IPs on the same subnet in the same system. And if one of them is the DNS server that the other service needs to contact to resolve addresses, it gets even worse. The result was that sometimes it worked and sometimes it didn’t, depending on how casually the packets were routed.

So I had to give it static routes for each address/port. To do that, just use: /etc/iproute2/rt_tables using two routing tables:

root@R4S:~# cat /etc/iproute2/rt_tables
100 eth0table
101 eth1table

Which are then applied from the directory /etc/network/if-up.d (192.168.1.2 is the default gateway):

root@R4S:~# cat /etc/network/if-up.d/10-policy-routing
#!/bin/sh
[ "$IFACE" = "eth0" ] && ip rule add from 192.168.1.4 table 100 2>/dev/null && ip route replace 192.168.1.0/24 dev eth0 src 192.168.1.4 table 100 && ip route replace default via 192.168.1.2 dev eth0 table 100
[ "$IFACE" = "eth1" ] && ip rule add from 192.168.1.5 table 101 2>/dev/null && ip route replace 192.168.1.0/24 dev eth1 src 192.168.1.5 table 101 && ip route replace default via 192.168.1.2 dev eth1 table 101

Then you need to specify the kernel to accept packets by reading the values from all routing tables. This can be made permanent by setting rp_filter=2 (instead of 1) in the kernel at boot:

root@R4S:~# cat /etc/sysctl.d/99-rpfilter.conf
net.ipv4.conf.all.rp_filter=2
net.ipv4.conf.eth0.rp_filter=2
net.ipv4.conf.eth1.rp_filter=2

After a reboot, everything started working correctly and consistently, and packets were no longer being dropped or sent out through a different interface than the one they came in on.

After several weeks of use, I can confirm that the nanoPi R4S is definitely faster overall (updates, Homebridge restarts, compilation, etc.), and DNS response times are also slightly lower (though we’re talking about 2–3 milliseconds).

🤔

Even though everything has been working perfectly after a month and many updates, I don’t really like this solution because it involves many system modifications, and the more changes there are, the higher the chance that something might break. Anyway, for now I’ll keep using it, hoping that nothing breaks.

Archive server updates

I’ve made some small tweaks to the Archive server, just some optimizations.

Update Docker containers

Since I was tired of manually updating the Docker containers every time, I decided to use a very handy service called Dockcheck.

Essentially, it’s a bash script that utilizes cron to check for updates, updates any outdated containers, prunes previous images, and sends a notification to me on Pushover with the names of the updated containers.

Here’s a modified push notification, altered slightly to suit my tastes:

push

pushicon

I also found a perfect icon for this Pushover service =)

Very convenient. I configured the script to run every six hours using cron.

# Dockcheck updates
0 */6 * * * /usr/local/bin/Dockcheck/dockcheck.sh -y -i -p -x 6

n8n workflow

For n8n I created a workflow that sends me an email every morning with the page visits to this blog, for each day, week, and month:

n8n-umami

The resulting email is this:

n8n-umami-mail

Conclusions

After about a month of using this setup with LAGs, I’ve noticed an improved responsiveness of the home network, or rather, it’s always the same but there are fewer delays and more smoothness. Those rare moments when something in the network stalls for 2/3 seconds are no longer there. And the best thing is being able to work on a server without ever experiencing slowdowns while it’s performing a backup or other similar network tasks.

And I like the idea of using the old nanoPi R4S as a server for both Pi-Hole and Homebridge, it’s faster then two Raspberry PIs 4, I saved some watts and heat and I have also two more spare Raspberry PI with SSDs.