Making and testing RedHat rpms for ntop Burton M. Strauss III June 2002 / April 2005 =============================================== ==== W a r n i n g ============================ =============================================== First off, this is information on how I personally create rpms for ntop. It's actually quite simple, but some of the details will refer to how-tos for other programs. I use a PIII-800 w/ 128 MB of RAM, named "tigger", which is running RedHat 7.2 updated to be reasonably current with various packages. tigger is also my ntop development machine. All of which is irrelevant for the purposes of creating rpms! NB: Jul2002 - I upgraded to 384MB - found a cheap 256MB DIMM. Performance is a lot better, but nothing really changes. NB: Apr2005 - Tigger is now dual P3-1000 (SMP) w 832MB of RAM running Fedora Core 2. The underlying platform is still largely irrelevant. You have THREE basic choices... 1) UML (which I used for 2.1 and earlier) 2) buildmeister (which I used through 3.0) 3) VPC (3.1 and following) Most of the UML setup was done by trial and error - there may be a better way. But it worked well enough for the purpose. However, I don't always know what I did or how I got it to work. If you have questions about rpm or user mode linux, don't ask on the ntop lists, ask the rpm or uml people! Also - one real warning... REMEMBER WHICH MACHINE YOU ARE ON when issuing commands! It's likely that the rpm won't build right on the underlying (real) computer - I once spent three hours trying to (re)install things and fixing things because I was trying to build on tigger, not ur-tigger! ALSO: DO NOT SHUTDOWN THE REAL MACHINE THINKING IT IS THE UML MACHINE! It causes BAD things like file system corruption to the uml machines! Trust me on both of these... I have not updated the UML instructions since 2.1 days. But it you go this route, all of the changes in ntop on a native machine still apply. ============================================================================================== ==== U M L =================================================================================== ============================================================================================== =============================================== ==== E n v i r o n m e n t ==================== =============================================== Building rpms works best if you are running as root. And on a system without complications (other services running, etc). Accordingly, I've used User Mode Linux (uml), which is available at http://user-mode-linux.sourceforge.net/ On tigger, I created two uml machines, called ur-tigger and ur-tigger2. Both are RedHat 7.2 machines - just follow the uml instructions. You may find it easier to use the prebuilt pristine filesystems available at that site. I did. I found that I had trouble with the rpm database, and I rebuilt it using this procedure: http://hermes.afaa.asso.fr/users/pascal/linux/rpm-rebuilddb But you should be able to download a working, pristine version and not have the problem. I then stripped them down so that ur-tigger is really just a development machine - no extra services (it runs ssh and the development tools). And ur-tigger2 is a basic user machine, such as one might install ntop on. Using uml and only 128MB of ram, but with a PIII-800, it's not fast, but it's certainly good enough. I start ur-tigger, create the rpm, move it to shared storage, shut down ur-tigger, start ur-tigger2, install the rpm and still have 30-45 minutes of test time left out of an hour. Since the machine is somewhat underpowered, you will see serious performance hits on your session on the real machine while the uml machines are up. If you really have to do this, I *guess* (but haven't tried it) that you could nice the uml machine - at the expense of performance there. For both uml machines, I created startup files (I put these in /etc/init.d for no good reason, since the uml machines are run under my personal id, not root). I've listed one of them at the bottom. The only thing unusual is this line: args="con1=tty:/dev/tty4 Which puts the uml (ur-tigger) console on tty4 of tigger's console. I strongly suggest you don't try and do this with a COW file. The two machines are pretty different. While the file sizes LOOK scary, they're really sparse, and quite small: [bstrauss@tigger ur-tigger]$ ls -ls 4 -r-xr-xr-x 1 bstrauss bstrauss 3191 May 12 18:23 control 268 -rwxrwxr-x 1 bstrauss bstrauss 266876 Apr 28 10:04 initrd-um 4 drwxrwxr-x 2 bstrauss bstrauss 4096 Apr 28 09:51 mnt 617580 -rwxrwxr-x 1 bstrauss bstrauss 1073741824 Jun 19 10:16 rootfs 12252 -rwxrwxr-x 1 bstrauss bstrauss 134217728 Jun 18 17:27 swapfs ^^^^^^actual effective ^^^^^^^^^^ So that you can easily exchange files between the two uml machines, follow this to setup host file access: http://user-mode-linux.sourceforge.net/hostfs.html (If you're using a machine that is this tight on memory, running two umls simultaneously isn't advised). =============================================== ==== C r e a t i n g ========================== =============================================== ur-tigger is used to create the rpms. You'll need to have available all the tools you need to compile and run ntop - via trial and error, I developed the following list (it's in the ntop.spec file): BuildPrereq: glibc, glibc-devel, gcc, cpp, gawk, autoconf, automake, openssl, openssl-devel, gdbm, gdbm-devel, libpcap, mysql, mysql-devel Key version# information (These aren't REQUIRED, it's just what I use): Kernel: 2.4.18-15um autoconf-2.13-14 automake-1.4p5-2 cpp-2.96-98 gawk-3.1.0-3 gcc-2.96-98 gdbm-1.8.0-10 gdbm-devel-1.8.0-10 glibc-2.2.4-13 glibc-common-2.2.4-13 glibc-devel-2.2.4-13 libpcap-0.6.2-9 mysql-3.23.41-1 mysql-devel-3.23.41-1 openssl-0.9.6b-8 openssl-devel-0.9.6b-8 Login to the real machine as you. Startup the uml machine. Running the /etc/init.d/ur-tigger start command will seize your session, so I usually connect via ssh from my workstation which is a separate (Windows) machine and just leave the session up. But I could just as easily define two uml consoles and use them. Login to the uml machine as root. Under /usr/src, create the following structure (this is standard): ./redhat ./redhat/BUILD ./redhat/RPMS ./redhat/RPMS/athlon ./redhat/RPMS/i386 ./redhat/RPMS/i486 ./redhat/RPMS/i586 ./redhat/RPMS/i686 ./redhat/RPMS/noarch ./redhat/SOURCES ./redhat/SPECS ./redhat/SRPMS Put the ntop source into ./redhat/SOURCES, along with any other files you need: -rw-r--r-- 1 root root 225 Jun 4 13:10 force-i386.patch -rw-rw-r-- 1 ntop ntop 1828895 Jun 4 10:28 ntop-2.0.99-rc2.tgz -rw-r--r-- 1 root root 3602 May 14 14:04 ntop-config.in -rw-r--r-- 1 root root 1246 May 14 14:03 ntop.init -rw-r--r-- 1 root root 159 Nov 23 2001 ntop.logrotate -rw-r--r-- 1 root root 3267 Jun 18 18:12 ntop.spec Also put the ntop.spec file into ./redhat/SPECS The spec file is the key to the build process (This is the RC3 version, other than directory structure and comments, it should be the 2.1 version): Summary: ntop shows the network usage Name: ntop Version: 2.0.99RC3 Release: 20020704 Source: ntop-2.0.99RC3-20020704.tgz Source1: ntop.init Source2: ntop.logrotate Patch1: force-i386.patch Patch2: version.patch Copyright: GPL Group: Applications/System BuildPrereq: glibc, glibc-devel, gcc, cpp, gawk, autoconf, automake, openssl, openssl-devel, gdbm, gdbm-devel, libpcap, mysql-devel Requires: glibc, openssl, gdbm, ncurses, libpcap Buildroot: %{_tmppath}/%{name}-root Prereq: /sbin/chkconfig, /sbin/ldconfig %description ntop is a network and traffic analyzer that provides a wealth of information on various networking hosts and protocols. ntop is primarily accessed via a built-in web interface. Optionally, data may be stored into a database for analysis or extracted from the web server in formats suitable for manipulation in perl or php. Note that the command line version, intop, is largely unsupported. See docs/1STRUN.txt for the 1st time startup procedure! This version - RC3 - includes a fix for the https:// hangs, see --use-sslwatchdog. SSLv3 is enabled. %prep %setup -q -c ${NAME}${VERSION} %patch 1 -P %build unset RPM_OPT_FLAGS %undefine optflags # ntop craps out here with gcc2.96-98 and full optimization! # First, build static libraries - use the new buildAll script cd gdchart0.94c ./buildAll.sh cd ../ntop # Now, configure and build ntop %configure --enable-optimize --bindir=%{_sbindir} --enable-sslv3 make patch -p0 < ../../../SOURCES/version.patch make %install cd ntop mkdir -p $RPM_BUILD_ROOT/etc/rc.d/init.d \ $RPM_BUILD_ROOT/%{_sbindir} \ $RPM_BUILD_ROOT/etc/logrotate.d make install DESTDIR=$RPM_BUILD_ROOT install -c -m0755 %{SOURCE1} $RPM_BUILD_ROOT/etc/rc.d/init.d/ntop install -c -m0644 %{SOURCE2} $RPM_BUILD_ROOT/etc/logrotate.d/ntop %post /sbin/chkconfig --add ntop /sbin/ldconfig /sbin/service ntop condrestart > /dev/null 2>&1 %preun if [ "$1" = "0" ]; then /sbin/service ntop stop > /dev/null 2>&1 /sbin/chkconfig --del ntop fi %postun if [ "$1" -ge "1" ]; then /sbin/service ntop condrestart > /dev/null 2>&1 fi /sbin/ldconfig %clean rm -rf $RPM_BUILD_ROOT %files %defattr(-,root,root) %doc ntop/AUTHORS %doc ntop/CONTENTS %doc ntop/MANIFESTO %doc ntop/COPYING %doc ntop/ChangeLog %doc ntop/docs/FAQ %doc ntop/docs/HACKING %doc ntop/docs/KNOWN_BUGS %doc ntop/docs/TODO %doc ntop/docs/1STRUN.txt %doc ntop/NEWS %doc ntop/PORTING %doc ntop/README %doc ntop/SUPPORT_NTOP.txt %doc ntop/THANKS %config %{_sysconfdir}/rc.d/init.d/ntop %config %{_sysconfdir}/logrotate.d/ntop %{_sbindir}/ntop %{_sbindir}/ntop-cert.pem %{_sbindir}/intop %{_datadir}/%{name} %{_mandir}/man1/intop.1.gz %{_mandir}/man8/ntop.8.gz %{_libdir}/ntop %{_libdir}/plugins %{_libdir}/libntop* %{_libdir}/libicmpPlugin* %changelog * Thu Jul 03 2002 Burton M. Strauss III - Updated to RC3 * Thu Jun 20 2002 Burton M. Strauss III - Updated to RC2A - unofficial post RC2 build. * Tue Jun 04 2002 Burton M. Strauss III - Updated to RC2 * Fri May 17 2002 Burton M. Strauss III - Updated to RC1 * Mon May 13 2002 Burton M. Strauss III - Updated for 2.0.99 - beta - from 13May2002 snapshot * Fri Nov 23 2001 Hal Burgiss - Updated for 2.0 beta. * Thu Oct 11 2001 Hal Burgiss - Updated for 2.0 pre-release snapshots. Many changes. It's documented in many places, it's a standard rpm file. You can see where I stole it from (Thanks Hal), all I did was fixup a few things. Remember to update the key values each time: Version: 2.0.99RC2 Release: 20020604 Source: http://luca.ntop.org/ntop-2.0.99-rc2.tgz and the changelog. Luca's .tgz source files don't always follow the same structure, so you may have to change the cd commands, cd RC2/gdchart0.94c and cd RC2/ntop (in %build and %install respectively). The one patch forces an i386 build for maximum compatibility (that's also why I build for MySQL): --- RC2/gdchart0.94c/buildAll.sh Tue Apr 16 18:05:18 2002 +++ RC2/gdchart0.94c/buildAll.sh Mon May 13 13:09:31 2002 @@ -1,6 +1,6 @@ #!/bin/sh -./configure +./configure i386-redhat-linux cd zlib-1.1.4/ ./configure make (You may also have to adjust the directory structure on the --- and +++ lines). To actually do the build, cd to the SPECS directory and run rpm -ba ntop.spec When things finish, there should be a pair of .rpm files, the binary one in ./redhat/RPMS/i386 and the source in ./redhat/SRPMS Copy or move them to the host (real) machine. If you have errors, it's usually missing tools or the directory structure. Best bet is to take the name of the failing file and look at it (they are all standard shell scripts created by rpm and in /var/tmp) Shutdown the uml machine (as root, type shutdown -h now) =============================================== ==== T e s t i n g ============================ =============================================== ur-tigger2 is used to test the rpms. You'll need to have available all the tools you need to run ntop - via trial and error, I developed the following list (it's in the ntop.spec file): Requires: glibc, openssl, gdbm, ncurses, libpcap ^^^^^^^ intop only Login to the real machine as you. Startup the uml machine. Same warning as above about seizing your session. Login to the uml machine as root. To test the rpm, all you do is uninstall the old one rpm -e ntop and install the new one rpm -i ntop-xxxx.i386.rpm Then fire up ntop (remember docs/1STRUN.txt) and let it run as your test. ============================================================ =============================================== ==== /etc/init.d/ur-tigger - startup file====== =============================================== #!/bin/sh # $Id: RedHat-rpmbuild-HOWTO.txt 2757 2005-04-04 13:39:46Z burton $ ### ### Various settings ### name="ur-tigger" dirname=/home/ntop/uml/$name mc=$name mcp=$HOME/.uml/$name/pid memsize="64m" fs=" ubd0=rootfs root=/dev/ubd/0" swap="ubd7=swapfs" args="con1=tty:/dev/tty4" net="eth0=tuntap,,," hostiface="eth0" ### ### Change to the base directory ### cd $dirname ### ### Look for uml binary ### PATH=.:$PATH if [ -x $name ] then linux=$name else linux=linux fi ### ### figure out host ip address ### PATH=/sbin:/usr/sbin:$PATH ip=`LANG="C" LC_ALL="C" ifconfig $hostiface | grep 'inet addr:' | sed 's/.*inet addr:\([0-9.]*\).*/\1/'` net="$net$ip" ### ### look for initial ramdisk ### initrd="" if [ -f initrd-um ] ; then initrd="initrd=initrd-um" ; fi ### ### Process command ### case $1 in start) shift exec $linux $initrd umid="$name" $fs $swap mem=$memsize $net $args "$@" ;; stop) uml_mconsole "$mc" halt exit ;; kill) if [ -f $mcp ] then echo "Killing pid `cat $mcp`" kill -9 "`cat $mcp`" else echo "Instance $name doesn't appear to be running" fi exit ;; console|mconsole) exec uml_mconsole $mc ;; cad|ctrl-alt-delete) exec uml_mconsole $mc cad ;; reboot) uml_mconsole $mc reboot exit ;; install) # undocumented option used to install stuff originally shift rm -f installfs/success $linux rw umid="$name" $fs mem=64m ubd7=installfs $net root=/dev/ubd/7 init=/uml_stage2install.py $args "$@" if [ -f installfs/success ] then exitcode=0 echo "Install succeeded" else echo "Install failed" exitcode=1 fi rm -f installfs/opts.py 2>/dev/null # contains root password if [ -f installfs/initrd-um ] then cp installfs/initrd-um . fi find installfs -type d -print | xargs chmod 777 2>/dev/null rm -rf installfs 2>/dev/null exit $exitcode ;; mount) mount rootfs mnt/ -o loop exit ;; umount) cd ~ umount mnt/ exit ;; status) pid=`cat $mc/pid` echo "pid is $pid" ps -ef | grep $pid uml_mconsole $mc/mconsole version exit ;; df|du) ls -lhs exit ;; *) echo "`basename $0` (start|stop|console|reboot|status|kill|mount|umount)" echo "start starts new session" echo "stop stops current session (hard halt) - safer to use" echo " init 0 inside session" echo "reboot reboots current session (hard reboot) - safer to use" echo " init 6 inside session" echo "cad sends ctrl-alt-delete" echo "console starts mconsole" echo "status current status" echo "kill kill -9 the uml session" echo "mount mounts filesystems as loopback on mnt, you must be root" echo "umount unmounts the above" echo "du shows actual disk space usage" exit 1 esac ========================================== Because somebody always wants to know, here is what's on the system. Kernel: 2.4.18-15um Installed packages (90% of which are un-necessary): 2.4.18-15um 4Suite-0.11-2 MAKEDEV-3.2-5 PyXML-0.6.5-4 SysVinit-2.78-19 a2ps-4.13b-15 alchemist-1.0.18-1 anacron-2.3-17 ash-0.3.7-2 at-3.1.8-20 authconfig-4.1.19-1 autoconf-2.13-14 automake-1.4p5-2 basesystem-7.0-2 bash-2.05-8 binutils-2.11.90.0.8-9 bzip2-1.0.1-4 bzip2-libs-1.0.1-4 chkconfig-1.2.24-1 console-tools-19990829-36 cpio-2.4.2-23 cpp-2.96-98 cracklib-2.7-12 cracklib-dicts-2.7-12 crontabs-1.10-1 cyrus-sasl-1.5.24-20 cyrus-sasl-md5-1.5.24-20 cyrus-sasl-plain-1.5.24-20 db1-1.85-7 db2-2.4.14-7 db3-3.2.9-4 dev-3.2-5 dhcpcd-1.3.18pl8-13 diffutils-2.7.2-2 dosfstools-2.7-1 e2fsprogs-1.23-2 ed-0.2-21 emacs-20.7-41 expat-1.95.1-7 file-3.35-2 filesystem-2.1.6-2 fileutils-4.1-4 findutils-4.1.7-1 freetype-2.0.3-7 gawk-3.1.0-3 gcc-2.96-98 gcc-c++-2.96-98 gdbm-1.8.0-10 gdbm-devel-1.8.0-10 glib-1.2.10-5 glibc-2.2.4-13 glibc-common-2.2.4-13 glibc-devel-2.2.4-13 gmp-3.1.1-4 grep-2.4.2-7 groff-1.17.2-3 groff-perl-1.17.2-3 grub-0.90-11 gzip-1.3-15 hdparm-4.1-2 hotplug-2001_04_24-11 indexhtml-7.2-1 info-4.0b-3 initscripts-6.40-1 ipchains-1.3.10-10 iproute-2.2.4-14 iptables-1.2.3-1 iputils-20001110-6 kbdconfig-1.9.14-1 kernel-2.4.7-10 kernel-BOOT-2.4.7-10 kernel-debug-2.4.7-10 kernel-enterprise-2.4.7-10 kernel-headers-2.4.7-10 kernel-smp-2.4.7-10 kon2-fonts-0.3.9b-7 krb5-devel-1.2.2-13 krb5-libs-1.2.2-13 krb5-workstation-1.2.2-13 ksymoops-2.4.1-1 kudzu-0.99.23-1 less-358-21 libpcap-0.6.2-9 libstdc++-2.96-98 libstdc++-devel-2.96-98 libtermcap-2.0.8-28 libxml2-2.4.2-1 libxslt-1.0.1-3 lilo-21.4.4-14 lockdev-1.0.0-14 logrotate-3.5.9-1 lokkit-0.50-6 losetup-2.11g-5 m4-1.4.1-5 mailcap-2.1.6-1 mailx-8.1.1-22 make-3.79.1-8 man-1.5i2-6 mingetty-0.9.4-18 mkbootdisk-1.4.2-3 mkinitrd-3.2.6-1 mktemp-1.5-11 modutils-2.4.6-4 mount-2.11g-5 mouseconfig-4.23-1 mpage-2.5.1-7 mysql-3.23.41-1 mysql-devel-3.23.41-1 ncurses-5.2-12 ncurses-devel-5.2-12 net-tools-1.60-3 netconfig-0.8.11-7 newt-0.50.33-1 nkf-1.92-6 ntsysv-1.2.24-1 openldap-2.0.11-13 openssh-2.9p2-7 openssh-clients-2.9p2-7 openssh-server-2.9p2-7 openssl-0.9.6b-8 openssl-devel-0.9.6b-8 openssl-perl-0.9.6b-8 pam-0.75-14 parted-1.4.16-8 passwd-0.64.1-7 patch-2.5.4-10 pcre-3.4-2 perl-5.6.0-17 perl-DBI-1.18-1 perl-DateManip-5.39-5 perl-Digest-MD5-2.13-1 perl-HTML-Parser-3.25-2 perl-HTML-Tagset-3.03-3 perl-MIME-Base64-2.12-6 perl-NKF-1.71-2 perl-Parse-Yapp-1.04-3 perl-Storable-0.6.11-6 perl-URI-1.12-5 perl-XML-Dumper-0.4-5 perl-XML-Encoding-1.01-2 perl-XML-Grove-0.46alpha-3 perl-XML-Parser-2.30-7 perl-XML-Twig-2.02-2 perl-libnet-1.0703-6 perl-libwww-perl-5.53-3 perl-libxml-enno-1.02-5 perl-libxml-perl-0.07-5 popt-1.6.4-7x procmail-3.21-1 procps-2.0.7-11 psmisc-20.1-2 psutils-1.17-13 pwdb-0.61.1-3 python-1.5.2-35 quota-3.01pre9-3 readline-4.2-2 redhat-logos-1.1.3-1 redhat-release-7.2-1 rootfiles-7.2-1 rpm-4.0.4-7x rpm-build-4.0.4-7x sed-3.02-10 setserial-2.17-4 setup-2.5.7-1 setuptool-1.8-2 sh-utils-2.0.11-5 shadow-utils-20000902-4 slang-1.4.4-4 slocate-2.6-1 sysklogd-1.4.1-4 syslinux-1.52-2 tar-1.13.19-6 tcsh-6.10-6 termcap-11.0.1-10 textutils-2.0.14-2 time-1.7-14 timeconfig-3.2.2-1 tmpwatch-2.8-2 unzip-5.42-1 utempter-0.5.2-6 util-linux-2.11f-9 vim-common-5.8-7 vim-minimal-5.8-7 vixie-cron-3.0.1-63 wget-1.7-3 which-2.12-3 words-2-17 zlib-1.1.3-24 ============================================================================================== ==== B u i l d m e i s t e r ================================================================= ============================================================================================== Pretty standard for RedHat. You create a userid, buildmeister, plus the necessary directories and use rpmbuild, just as if you were root. The rpm build package automatically does the translation. The above instructions for UML are still the way to do it. You create the directories: ./BUILD ./RPMS ./RPMS/i386 ./RPMS/noarch ./SOURCES ./SPECS ./SRPMS in buildmeister's home instead of /usr/src. Use the spec file in ntop's cvs - packages/RedHat/ntop.spec - and that's where you will find the other files for SOURCES. ============================================================================================== ==== V P C =================================================================================== ============================================================================================== Microsoft's VPC 2004 (Virtual PC) - allows you to create multiple Linuxes that run at acceptable speed. You use standard VPC tools to create the VM (Virtual Machine), capture the CD-ROM for the OS you want to install and 'boot'. The key site to reference is "What works and what doesn't" at http://vpc.visualwin.com/. Two cautions: 1) The standard Fedora Core 3 kernel has problems - follow the workaround in the instructions at vpc.visualwin.com. 2) Use an external tool to capture the CD-ROM into a .iso image file. I've had problems with physical CDs hanging. But basically, using VPC is just like using a real machine. You install the OS - typically configuring it as a minimal development machine. Add the extra libraries ntop requires (libpcap, gdbm, etc.) and whatever native tools you want for package building. Thus I usually combine VPC + Buildmeister for .rpm based systems. Although real memory will limit the number of simultaneously running VMs, they boot and shutdown quickly, so the boot, scp the new files, rpmbuild, test the rpm, scp the generated file somewhere and shutdown cycle doesn't take much longer than (and can even be shorter) than on a native machine. Plus, for the bandwidth of downloading a few ISOs, you can have a new Linux. As of Apr2005, I have VPCs for: Debian, FC2, FC3, FreeBSD 4.11, FreeBSD 4.10, FreeBSD 5.3, Gentoo, RedHat 9, RedHat 8.0, SuSE and Solaris 10.