FOSSology Installation Documentation

Introduction
============
This document is designed to help you get FOSSology installed and ready
to use. It's intended audience is the system administrator who wants to
quickly get a local install up and running, or a distribution developer
looking to create packages.

For extended discussion on how to use or tune the software, please see
the User Documentation, available on the website.

This document is arranged  as follows:

Section 1 - Short Installation Instructions
Section 2 - Full Installation Instructions
         2.1 - Preparing your system
           2.1.1 Disk space
           2.1.2 Dependencies
           2.1.3 Adjusting the Kernel
           2.1.4 Preparing Postgresql
           2.1.5 Configuring PHP
           2.1.6 Configuring Apache
         2.2 - Building and Installing FOSSology
           2.2.1 Build and Install Instructions
           2.2.2 Automated Post-Installation
           2.2.3 Detailed Post-Install Instructions
           2.2.4 Checking the Default Configuration Files
           2.2.5 Start the Scheduler
Section 3 - After Installation 

Section 1 - Short Installation Instructions
===========================================
(In examples, # is as root, $ is mortal)
1. If you have any old FOSSology installs you want to remove from the
   system you can run the fo-cleanold script
   # utils/fo-cleanold
   If you have an existing install that you want to transition, be sure
   back up your repository, database, and config files before proceeding.
2. install dependencies
   # utils/fo-installdeps
3. Adjust the kernel shmmax (described in detail in section 2.1.3)
   # echo 512000000 > /proc/sys/kernel/shmmax
   # echo "kernel.shmmax=512000000" >> /etc/sysctl.conf
4. Adjust postgresql config (described in detail in section 2.1.4)
5. Adjust PHP config (described in detail in section 2.1.5)
6. Add apache config for FOSSology (described in detail in section 2.1.6)
  There is an example config provided in ui/fo-apache-example.conf.
  After adding the config restart apache.
7. Build FOSSology
   $ make
8. Install FOSSology
   # make install
9. Run the postinstall script
   # /usr/local/lib/fossology/fo-postinstall
10. Test that things are installed correctly
   # /usr/local/lib/fossology/fossology-scheduler -t
11. Start the fossology scheduler daemon
   # /etc/init.d/fossology start

You're done, point your web browser at the new install
(http://yourhostname/repo/) and start using FOSSology!


Section 2 - Full Installation Instructions
==========================================
The rest of this file is the much more verbose instructions. Consult
them if you have questions or need more explanation. If you are doing
a multi-system install please also see the INSTALL.multi document.

Section 2.1 - Preparing your system
***********************************

2.1.1 - Disk Space
------------------
FOSSology stores uploaded data in a filesystem repository.  As you
upload and analyze packages via FOSSology, the repository can grow very
large.  The default location for a single system repository is
/srv/fossology/repository/ however this can be adjusted by the system
administrator to another location if desired.

It is recommended that the area you choose to keep the repository in,
be a separate mount point with at least 4x the size of the unpacked
data you intend to scan.  For a small system intended to just scan a
few small personal projects this might mean gigabytes, but for systems
intended for scanning large collections of software including Linux
distributions, this probably means hundreds of gigabytes to terabytes.
If you are using multiple hosts to store the repository, it is best to
spread the repository data evenly across the hosts.  See the User Guide
for more information about using multiple hosts.

2.1.2 - Dependencies
--------------------
FOSSology uses lots of different existing tools and software and
expects to find it on the system.

FOSSology depends on recent versions of the following software,
dependencies needed when building fossology are labeled "build:" and
dependencies needed at runtime are labeled "runtime:".

Where we know them, commands are listed for installing on various
distributions.
* For Debian nearly all packages can be found in main, unless you want
   the non-free version of unrar
* For Ubuntu you will need a universe apt source setup
* For RHEL you will need packages from EPEL
    http://fedoraproject.org/wiki/EPEL

We provide a script for helping install dependencies:
   utils/fo-installdeps
To install just the build dependencies use the -b flag.
To install just the runtime dependencies use the -r flag.
To install just the agent runtime dependencies use the -a flag
   (and see INSTALL.multi)
For more information run it with the --help flag.
If this utility doesn't support the distro you are running (or fails
to install everything that's needed) please report a bug.


2.1.2.1 Service Dependencies
````````````````````````````
Postgresql:
 8.1 or newer
 On Debian (or a Debian derived distribution) you should be able to use 
 something like:
  runtime: apt-get install postgresql-8.1
  build: apt-get install postgresql-server-dev-8.1
 On Fedora/RHEL:
  runtime: yum install postgresql postgresql-server
  build: yum install postgresql-devel
 On Mandriva:
  runtime: urpmi postgresql
  build: urpmi postgresql-devel

Apache2:
 FOSSology is currently developed using Apache2 version 2.2, but any
 version should work.
 On Debian (or a Debian derived distribution) you should be able to use 
 something like:
   runtime: apt-get install apache2
 On Fedora/RHEL:
   runtime: yum install httpd
 On Mandriva:
  runtime: urpmi httpd

PHP5:
 On Debian (or a Debian derived distribution) you should be able to use 
 something like:
  runtime: apt-get install php5 php5-pgsql php-pear libapache2-mod-php5
 On Fedora:
  runtime: yum install php php-pear php-pgsql
 On Mandriva:
  runtime: urpmi php php-pear php-pgsql


2.1.2.2  Library Dependencies
`````````````````````````````
* libmagic - for determining file types, from the "file" software
   ftp://ftp.astron.com/pub/file/file-4.02.tar.gz 
* libxml2 - GNOME XML library
   http://gnome.org
* libextractor - GNU file meta-data extractor
   http://www.gnunet.org/libextractor/

 On Debian (or a Debian derived distribution) you should be able to use
 something like:
  runtime: apt-get install libmagic1 libxml2 libextractor1c2a \
	libextractor-plugins
  build: apt-get install libmagic-dev libxml2-dev libextractor-dev
 On Fedora:
  runtime: yum install file-libs libxml2 libextractor libextractor-plugins-base
  build: file-libs libxml2 libextractor-devel
 On RHEL, install libextractor from upstream sources and:
  runtime: yum install file-libs libxml2 
  build: file-libs libxml2
 On Mandriva:
  runtime: urpmi libextractor libextractor-plugins libxml2 
  build: urpmi libextractor-devel libmagic-devel libxml2 
  

2.1.2.3 External Commands
`````````````````````````
* ar - for extracting archives, from the binutils software
   http://www.gnu.org/software/binutils/
* bzcat - bz2 decompressor, from the bzip2 software
   http://www.bzip.org/
* cabextract - extractor for Microsoft Cabinet files
   http://www.cabextract.org.uk/
* cpio - for extracting cpio archives
   http://www.gnu.org/software/cpio/
* icat and fls - forensics tools from the sleuthkit software
   http://sourceforge.net/projects/sleuthkit/
* isoinfo - read metadata info from ISO9660 images, from the
   mkisofs/cdrtools/cdrkit implementations
   cdrkit implementation http://debburn.alioth.debian.org/
* pdftotext - from the xpdf software
   http://www.foolabs.com/xpdf
* rpm and rpm2cpio - for extracting software and metadata from rpm packages
   http://www.rpm.org/
* sendmail - for sending notification emails (optional)
   If you want to be able to receive notification emails about
   fossology processing, you need an /usr/sbin/sendmail command that
   fossology can use to send mail. All mail transport agents(MTA) (like
   postfix, exim, sendmail, etc.) provide this command and you probably
   already have it on your system, but you may need to configure the
   MTA to be able to send the mail where you want it to go.
* tar - tape archive decompressor
   http://www.gnu.org/software/tar/
* upx-ucl - an executable compressor/decompressor
   http://upx.sourceforge.net
* unrar-free - Unarchiver for .rar files
   https://gna.org/projects/unrar/
* unzip - De-archiver for .zip files
   ftp://ftp.info-zip.org/pub/infozip/src/
* zcat - for uncompressing .gz and .Z files, from the gzip software
   http://www.gzip.org/

 On Debian (or a Debian derived distribution) you should be able to use 
   something like:
   runtime: apt-get install binutils bzip2 cabextract cpio sleuthkit \
        mkisofs xpdf-utils rpm tar upx-ucl unrar unzip
   NOTE: If you are on etch, if you use unrar-free instead of unrar you
   will need to create a symlink from unrar to unrar-free. Post-etch
   both unrar packages install as "unrar".
 
 On Fedora, install sleuthkit and unrar from upstream sources and:
   runtime: yum install binutils bzip2 cabextract cpio genisoimage \
         poppler-utils rpm tar upx unzip gzip
 On RHEL, install cabextract, sleuthkit, upx, and unrar from upstream
  sources and:
   yum install binutils bzip2 cpio mkisofs poppler-utils rpm tar unzip gzip
 On Mandriva:
  runtime: urpmi cabextract genisoimage upx sleuthkit binutils bzip2 cpio \
                 mkisofs poppler-utils rpm tar unzip gzip

For those distributions where you can't get cabextract, unrar, or upx
as prebuilt packages, you can choose to do without those, but if fossology
encounters any of those container types it will be unable to scan them,
possibly missing important results (especially in the case of important
things like license scanning).


2.1.2.4 -  Additional Build Dependencies
````````````````````````````````````````
The fossology build requires gcc, binutils, glibc headers, make, perl,
and other tools you'd expect to find on a development system. In
addition, the Text::Template perl module is needed for processing files
in the build tree, and subversion is needed to embed version control
numbers into the build.

 On Debian (or a Debian derived distribution) you should be able to use 
 something like:
   build: apt-get install build-essential libtext-template-perl subversion
 On Fedora/RHEL the equivalent is:
   build: yum groupinstall 'Development Tools'
          yum install perl-Text-Template subversion
 On Mandriva:
   build: urpmi perl-Text-Template subversion gcc make perl 

2.1.3 Adjusting the Kernel
--------------------------
On modern large memory systems(>4gb), the linux kernel needs to be
adjusted to give postgresql more SysV shared memory.

To set on a running system:
  # echo 512000000 > /proc/sys/kernel/shmmax
To make sure it gets set on boot
  # echo "kernel.shmmax=512000000" >> /etc/sysctl.conf

This number is the number of pages (usually 4k each), and is based on a
fairly complicated formula, please see the postgresql tuning part of the
user guide.

2.1.4 Preparing Postgresql
--------------------------
Your postgresql install should be configured and running. If you need
help doing that consult the user documentation at
http://www.postgresql.org/docs/manuals/. If you are using SSL in
particular see the section Secure TCP/IP Connections with SSL to set it up.

Edit /etc/postgresql/<version>/main/postgresql.conf:
The tuning and preferences in the config file will vary depending on your
installation. We don't provide an automated way to do this because it
is complicated and specific to your particular install goals. Here are
the results of a diff between the default config file provided by
Debian and the one used for the http://repo.fossology.org database:

 #hba_file = 'ConfigDir/pg_hba.conf'	# host-based authentication file
 #ident_file = 'ConfigDir/pg_ident.conf'	# IDENT configuration file
 #external_pid_file = '(none)'		# write an extra pid file
 listen_addresses = '*'
 max_connections = 50
 #shared_buffers = 1000			# min 16 or max_connections*2, 8KB each
 shared_buffers = 32768
 work_mem = 102400 
 max_fsm_pages = 100000			# min max_fsm_relations*16, 6 bytes each
 fsync = off   
 full_page_writes = off			#recover from partial page writes
 commit_delay = 1000 
 effective_cache_size = 25000
 log_min_duration_statement = -1	# -1 is disabled, 0 logs all statements
 log_line_prefix = '%t %h %c'		# prepend a time stamp to all log entries

2.1.5 Configuring PHP
---------------------
Some php config variables may need to be adjusted for FOSSology. We don't
provide an automated way to do this because it can be specific to your
particular install if you are using php for other things. Edit your
php.ini file for apache (location dependent on your install, but
probably something like /etc/php5/apache2/php.ini) and make adjustments
that will work for your system and usage. Here are some things to
consider:

max_execution_time = 90
  This controls how long, in seconds, a php process is allowed to run.
  For "one shot" license analysis, particularly large jobs, or if your
  system is slow you may need to increase this (the default is 30).

memory_limit = 702M
post_max_size = 701M
upload_max_filesize = 700M
  These control the size of file you will be able to upload via the
  web interface. For very large uploads (for example DVD images) we
  recommend using the command line upload method, but you might want
  to increase these to handle up to CDROM sized uploads. (the defaults
  are 128M, 8M, and 2M)

2.1.6 Configuring Apache
------------------------
1) You need to add something like the following to the apache config,
  and this will depend on
 A) How you have apache configured. You might be creating a new site
    config using apache's "sites-available"/a2ensite(8) mechanism or
    editing and existing config you have setup. For example, on a
    Debian apache2 install you would have site config files in
    /etc/apache2/sites-available/ and you might be editing the default
    one or creating a new one.
 B) The path you want the FOSSology UI to appear on the server, this
   example uses "/repo/"
 C) Where your FOSSology is installed, this example assumes the default
   local sysadmin share prefix of /usr/local/share/
 D) What other things you might be using apache for on the system.
 for these reasons we can't provide an automated way of doing this.

========================================================================
	Alias /repo/ /usr/local/share/fossology/www/
        <Directory "/usr/local/share/fossology/www">
                AllowOverride None
                Options FollowSymLinks MultiViews
                Order allow,deny
                Allow from all
		# uncomment to turn on php error reporting 
		#php_flag display_errors on
		#php_value error_reporting 2039
        </Directory>
========================================================================
NOTE: included in the above example are some commented lines used for
enabling php error reporting. If you are having problems you might
choose to enable these to help determine the problem. Normally you
probably want them turned off so they don't report confusing error
messages to your end users or reveal information about your system
configuration to potential attackers.

2) Because this software dynamically generates web pages based on the
   database, you may want to tell web robots not to index pages.  You
   can do this with a robots.txt file in your DocumentRoot.  Here is
   a sample that tells all agents to ignore your /repo urls:
========================================================================
    User-agent: *
    Disallow: /repo
========================================================================

Once you have installed the configuration you can test it by running
(as root):
  apache2ctl configtest
and if it tests ok, then you can restart the server with the new config
by running (as root):
  apache2ctl graceful
Note: the site won't work yet until we install FOSSology below.


Your system should now be ready for installing FOSSology!


Section 2.2 - Building and Installing FOSSology
***********************************************
In the following steps, you will need to use files included in the
fossology source.  If you haven't done so already, untar the
fossology-<version>.tar.gz tarball into a staging directory.

NOTE: FOSSology uses a system of variables to control where the build
will install things. The defaults are all set for a standard local
sysadmin install, but if you prefer other locations or are building
distribution or 3rd party packages you can adjust to meet your needs.
See the HACKING file for more info.

NOTE: some commands are required to be run as the root user and are
indicated so. You should be able to use sudo, "su -", a normal root
login, or your favorite root-obtaining utility.

2.2.1   Build and Install Instructions 
--------------------------------------
1) If you have any old fossology installs you want to remove from the
   system you can run the fo-cleanold script
   # utils/fo-cleanold
   If you have an existing install that you want to transition, be sure
   back up your repository, database, and config files before proceeding.
   For more information on what this utility can do use the --help flag.

2) cd into the directory unpacking the tarball created
   (fossology-<version>/).

3) To build the software run "make".
   If you are using an existing SVN tree that you have updated, you
   might want to run a "make clean" first to get to a known good state.
   The Makefiles are pretty good about declaring dependencies that
   will make sure things get rebuilt, but to be sure use "make clean".

4) To install the software run "make install". You will need the proper
   permissions to install to where the files will be install, which in
   most cases means this needs to be run by root.

2.2.2   Automated Post-Installation
-----------------------------------
As part of the install a script was placed in
  /usr/local/lib/fossology/fo-postinstall
(by default, or similar if you configured a different install location).
This script will help you finish the install, setting up things that
can't be done by merely installing files on the filesystem. Because of
the things it does, the script must be run as root. By default the
script is designed to do everything you will need for a single machine
install. If you are doing a multiple machine install, please see the
INSTALL.multi document.
Run the postinstall script

  # /usr/local/lib/fossology/fo-postinstall

2.2.3   Detailed Post-Install Instructions 
------------------------------------------
If you prefer to do these steps by hand or are curious of what the
fo-postinstall script is doing, continue reading, otherwise skip to
step 2.2.4.

1) Setting up Users and Groups
You are expected to already have a "postgres" user as part of the
system postgresql install, and a "www-data" user as part of the
apache2 install.

FOSSology requires a system user and a system group both named
"fossy".

The /etc/passwd entries for these user should look
something like (Note: your uid & gid values may be different):

  fossy:x:1001:1001:FOSSology:/srv/fossology:/bin/bash

and the /etc/group entry

  fossy:x:1001:fossy

On a system with the useradd and groupadd commands (all LSB systems
including Debian, Fedora, etc) you can create the above system user
and group with the following commands as root:
  groupadd fossy
  useradd -c FOSSology -d /srv/fossology -g fossy -s /bin/false fossy

Alternatively, you can use the adduser command:
  adduser --gecos "FOSSology" --home /srv/fossology --system --group fossy

2) Setting up the database
A sample database schema file can be found either on the installed system
at /usr/local/lib/fossology/test.d/fossologyinit.sql or in the tarball at
fossology-<version>/db/fossologyinit.sql.  This will create the
database when it is read in.  It will also create an owner for the
database "fossy" and owner's password "fossy".  For security reasons,
please change this password, after the database is created, using
the alter command. Here are the steps:

Check to see the database server is running
   A) Ensure that postgresql is running using this command:
      /etc/init.d/postgresql-8.1 status
      Version Cluster   Port Status Owner    Data directory            Log file
      8.1     main      5432 online postgres /var/lib/postgresql/8.1/main \
           /var/log/postgresql/postgresql-8.1-main.log

   If the status reported is "online", you can proceed to the next step.
   If the status is "down", use this command to start postgres:
   /etc/init.d/postgresql-8.1 start

   You should see the following message:
      Starting PostgreSQL 8.1 database server: main.

   B) The default database and owner is created by reading in the
   fossologyinit.sql file of sql statements to define the database schema.

   You must be logged in as user postgres to create the database schema
   and define the owner.  As root, you can su to user postgres (su
   postgres) and then run the psql command:
      psql  < fossologyinit.sql

   C) If any steps fail, check the postgres log file for errors:
      /var/log/postgresql/postgresql-8.1-main.log

   D) Make sure /etc/postgresql/8.1/main/pg_hba.conf is configured
   correctly to allow your connection.  This file controls: which hosts
   are allowed to connect, how clients are authenticated, which
   PostgreSQL user names they can use, which databases they can access.
   As a starting point, you will need something like the following for
   local connections:

    # local      DATABASE  USER  METHOD  [OPTION]
      local       all      all   md5

   See http://www.postgresql.org/docs/current/static/client-authentication.html
   for detailed information.
   If you do need to edit it then restart postgresql so the changes take
   effect:
        /etc/init.d/postgresql-<version> restart

   E) Once the database is defined, verify connection with
        psql -d fossology -U fossy

      use the default password "fossy".  You should connect and see the
      following:
        Welcome to psql 8.1.9, the PostgreSQL interactive terminal.

        Type:  \copyright for distribution terms
               \h for help with SQL commands
               \? for help with psql commands
               \g or terminate with semicolon to execute query
               \q to quit

        fossology=>
      If so then you successfully connected. Type \q to quit.

   F) If any steps fail, check the postgres log file for errors:
      /var/log/postgresql/postgresql-8.1-main.log


2.2.4   Checking the Default Configuration Files
------------------------------------------------
On a fresh install you start with default versions of configuration
files that contain reasonable defaults (where possible), or some files
are generated via the post install script. You need to review and edit
where needed.
   A) /usr/local/etc/fossology/RepPath.conf
     This is the path to the fossology repository. The repository can
     be a mount point, and should be a large disk space. If you plan to
     analyze ISOs, then consider a terabyte of disk or larger. For more
     discussion of this topic see the INSTALL.multi file and user
     documents on the website. If you change this file you may need
     to setup the new repo or run the fo-postinstall script to do it.
   B) /usr/local/etc/fossology/Hosts.conf
     This file determines how the file repository is split across
     hosts, the default is a single machine. For more into see the
     INSTALL.multi file.
   C) /usr/local/etc/fossology/Proxy.conf
     If you need to use a proxy to access the internet, you can use this
     file to tell the different parts of fossology that need internet
     access which proxy to use.
   D) /usr/local/etc/fossology/Depth.conf
     This file is magic, do not touch unless you are a wizard.
   E) /usr/local/etc/fossology/Db.conf
     This file is the information needed to access the database. It's
     ownership and permissions should be root:fossy 660 so that
     only root and people in the fossy group can read it. On a fresh
     install it contains the default database password, you should
     really change that here and in postgresql.
   F) /usr/local/etc/fossology/Scheduler.conf
     This file tells the scheduler how to send out work to the agents.
     It is generated by the mkschedconf utility. You can run
       /usr/local/lib/fossology/mkschedconf -h
     for help on using that utility. For more information on the
     scheduler config file format, see
       http://fossology.org/docs/scheduler#configuring_the_scheduler


2.2.5   Start the Scheduler
---------------------------
As root, start the scheduler with:
   /etc/init.d/fossology start

The scheduler logs to /var/log/fossology/fossology.log by default, you
can look there for scheduler activity or errors.


Section 3 - After Installation
==============================
Congratulations, FOSSology is now installed!

You can now point your web browser at the new install
(http://yourhostname/repo/) and start using FOSSology!

You should login to FOSSology with the default fossy/fossy user and
password and create yourself a user with administrative privileges and
then delete the fossy user (or at least change the password).

For extended discussion on how to use or tune the software, please see
the User Documentation, available on the website at

   http://fossology.org/user_documentation