Self-Made Router-Powered Repository Mirror

on Kde, Opensuse, Linux

After migrating most desktop PCs at work, at home and of some friends to openSUSE with upstream KDE repo, it was always a time-consuming task to keep all these systems up-to-date. I was most bothered by downloading the same files over and over again, which happened sometimes to be quite slow, especially via WLAN.

Fortunatly I got a couple of month ago the Linux-based router AVM Fritz!Box 7270, which can be rooted with a modified firmware. The customizable firmware is provided by the Freetz project and allows you to combine different modular packages to add functionality.

After playing a little bit with Freetz, I even got a lighttpd webserver with ruby and caching capabilities running. Of course, the original firmware as well as Freetz allow the configuration of dynamic DNS services. :wink:

I ended up with a configuration including:

  • dropbear ssh server (allows remote login via ssh – otherwise only telnet is supported)
  • automount-scripts supporting ext3 and ext4 (latter is not supported by AVM firmware)
  • nfs including a CGI configuration web page
  • rsync

I formatted an external hard drive to use ext4 and attached it to the router. The file mirror-rsync.sh needs to be copied to an arbitrary folder on the external drive.

# file: "code-block.js"
#!/bin/sh
set -o errexit

cd /var/media/ftp/uStor02/opensuse

echo -e "\nII: Start time: $(date)\nII: directory: $(pwd)" 2>&1 | tee -a log.txt

trap - INT

# rsync -vaH --delete --delete-after --progress --partial --size-only --delay-updates --include-from=include.txt --exclude-from=exclude.txt --timeout=300 ftp-1.gwdg.de::pub/opensuse/update/12.1/ update/12.1 2>&1 | tee -a log.txt
rsync -vaH --delete --delete-after --progress --partial --size-only --delay-updates --include-from=include.txt --exclude-from=exclude.txt --timeout=300 ftp.halifax.rwth-aachen.de::opensuse/update/12.1/ update/12.1 2>&1 | tee -a log.txt

# rsync -vaH --delete --delete-after --progress --partial --size-only --delay-updates --include-from=include.txt --exclude-from=exclude.txt --timeout=300 ftp5.gwdg.de::pub/opensuse/repositories/KDE:/Release:/48/openSUSE_12.1/ repositories/KDE:/Release:/48/openSUSE_12.1 2>&1 | tee -a log.txt
rsync -vaH --delete --delete-after --progress --partial --size-only --delay-updates --include-from=include.txt --exclude-from=exclude.txt --timeout=300 ftp.halifax.rwth-aachen.de::opensuse/repositories/KDE:/Release:/48/openSUSE_12.1/ repositories/KDE:/Release:/48/openSUSE_12.1 2>&1 | tee -a log.txt

rsync -vaH --delete --delete-after --progress --partial --size-only --delay-updates --include-from=include.txt --exclude-from=exclude.txt --timeout=300 packman.inode.at::packman/suse/12.1/Essentials/ repositories/packman/12.1/Essentials 2>&1 | tee -a log.txt

To lower the data transfer, I tuned my rsync commands to download only x86_64 packages as well as only German language packages.

# file: 'include.txt'
*l10n-de*x86_64*
*l10n-de*noarch*
*wine*

*libqt4-debuginfo*x86_64*
*libkdepimlibs4-debuginfo*x86_64*
*libkdecore4-debuginfo*x86_64*
*libkde4-debuginfo*x86_64*
*kdepim4-debuginfo*x86_64*
*libqt4-x11-debuginfo*x86_64*
*amarok-debuginfo*x86_64*
*kmail-debuginfo*x86_64*
*libakonadi4-debuginfo*x86_64*

# file: 'exclude.txt'
src
*i686.rpm
*i686.drpm
*i586.rpm
*i586.drpm
*buildsymbols*
ia64
*debuginfo*
*debugsource*
*l10n*

To finish the router setup, the only thing left is to add a Cron job (supported by Freetz interface) which runs the script daily – preferably during the night.

The original firmware provides SMB access to files on the hard drive. This can be quite slow. So I configured NFS for access these repositories. There’s a German blog post with some benchmarks to compare SMB and NFS.

Finally, the new repositories have to be activated. The priority is set to 90 and this way overranks the original repos, which don’t have to be disabled.

# file: 'client.sh'
#!/bin/sh

zypper ar --check --no-keep-packages --no-gpgcheck --type rpm-md --name fritzbox-12.1-update nfs://fritz.box:/var/media/ftp/uStor02/opensuse/update/12.1?mountoptions=vers=3 fritzbox-12.1-update
zypper ar --check --no-keep-packages --no-gpgcheck --type rpm-md --name fritzbox-12.1-kde48 nfs://fritz.box:/var/media/ftp/uStor02/opensuse/repositories/KDE:/Release:/48/openSUSE_12.1?mountoptions=vers=3 fritzbox-12.1-kde48
zypper ar --check --no-keep-packages --no-gpgcheck --type rpm-md --name fritzbox-12.1-packman nfs://fritz.box:/var/media/ftp/uStor02/opensuse/repositories/packman/12.1/Essentials?mountoptions=vers=3 fritzbox-12.1-packman

zypper mr -p 90 fritzbox-12.1-update
zypper mr -p 90 fritzbox-12.1-kde48
zypper mr -p 90 fritzbox-12.1-packman

The most important thing is here to append ?mountoptions=vers=3 to all URLs. The Freetz NFS build doesn’t support NFS version 4 and zypper fails to auto-detect this.

So the next time I want to bring the latest KDE release to a friend, I just have to unplug my hard-drive and have everything with me. At work, I don’t have to bring my own hard drive with me – they have their own one :p with an equal script which can be triggered from time to time.