==========================
Checking Your Installation
==========================

When debugging mod_wsgi or a WSGI application, it is import to be able to
understand how mod_wsgi has been installed, what Apache and/or Python it
uses and how those systems have been configured, plus under what
configuration the WSGI application is running.

This document details various such checks that can be made. The primary
purpose of providing this information is so that when people ask questions
on the mod_wsgi mailing list, they can be directed here to perform certain
checks as a way of collecting additional information needed to help debug
their problem.

If you are reading this document because you have been directed here from
the mailing list, then ensure that you actually provide the full amount of
detail obtained from the checks and not leave out information. When you
leave out information then it means guesses have to be made about your
setup which makes it harder to debug your problems.

Apache Build Information
------------------------

Information related to what version of Apache is being used and how it is
built is obtained in a number of ways. The primary means is from the
Apache 'httpd' executable itself using command line options. The main such
option is the ``-V`` option.

On most systems the standard Apache executable supplied with the operating
system is located at '/usr/sbin/httpd'. On MacOS X, for the operating system
supplied Apache the output from this is::

    $ /usr/sbin/httpd -V
    Server version: Apache/2.2.14 (Unix)
    Server built:   Feb 10 2010 22:22:39
    Server's Module Magic Number: 20051115:23
    Server loaded:  APR 1.3.8, APR-Util 1.3.9
    Compiled using: APR 1.3.8, APR-Util 1.3.9
    Architecture:   64-bit
    Server MPM:     Prefork
      threaded:     no
        forked:     yes (variable process count)
    Server compiled with....
     -D APACHE_MPM_DIR="server/mpm/prefork"
     -D APR_HAS_SENDFILE
     -D APR_HAS_MMAP
     -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
     -D APR_USE_FLOCK_SERIALIZE
     -D APR_USE_PTHREAD_SERIALIZE
     -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
     -D APR_HAS_OTHER_CHILD
     -D AP_HAVE_RELIABLE_PIPED_LOGS
     -D DYNAMIC_MODULE_LIMIT=128
     -D HTTPD_ROOT="/usr"
     -D SUEXEC_BIN="/usr/bin/suexec"
     -D DEFAULT_PIDLOG="/private/var/run/httpd.pid"
     -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
     -D DEFAULT_LOCKFILE="/private/var/run/accept.lock"
     -D DEFAULT_ERRORLOG="logs/error_log"
     -D AP_TYPES_CONFIG_FILE="/private/etc/apache2/mime.types"
     -D SERVER_CONFIG_FILE="/private/etc/apache2/httpd.conf"

The most important details here are:

  * The version of Apache from the 'Server version' entry.
  * The MPM which Apache has been compiled to use from the 'Server MPM' entry.

Although this has a section which appears to indicate what preprocessor
options the server was compiled with, it is a massaged list. What is often
more useful is the actual arguments which were supplied to the 'configure'
command when Apache was built.

To determine this information you need to do the following.

  * Work out where 'apxs2' or 'apxs' is installed.
  * Open this file and find setting for '$installbuilddir'.
  * Open the 'config.nice' file in the directory specified for build directory.

On MacOS X, for the operating system supplied Apache this file is located at
'/usr/share/httpd/build/config.nice'. The contents of the file is::

    #! /bin/sh
    #
    # Created by configure

    "/SourceCache/apache/apache-747.1/httpd/configure" \
    "--prefix=/usr" \
    "--enable-layout=Darwin" \
    "--with-apr=/usr" \
    "--with-apr-util=/usr" \
    "--with-pcre=/usr/local/bin/pcre-config" \
    "--enable-mods-shared=all" \
    "--enable-ssl" \
    "--enable-cache" \
    "--enable-mem-cache" \
    "--enable-proxy-balancer" \
    "--enable-proxy" \
    "--enable-proxy-http" \
    "--enable-disk-cache" \
    "$@"

Not only does this indicate what features of Apache have been compiled in,
it also indicates by way of the ``--enable-layout`` option what custom Apache
installation layout has been used.

Apache Modules Loaded
---------------------

Modules can be loaded into Apache statically, or can be loaded dynamically
at run time based on Apache configuration files.

If modules have been statically compiled into Apache, usually it would be
evident by what 'configure' arguments have been used when Apache was built.
To verify exactly what is compiled statically, you can use the ``-l``
option to the Apache executable.

On MacOS X, for the operating system supplied Apache the output from
running ``-l`` option is::

    $ /usr/sbin/httpd -l
    Compiled in modules:
      core.c
      prefork.c
      http_core.c
      mod_so.c

This indicates that the only module that is loaded statically is 'mod_so'.
This is actually the Apache module that handles the task of dynamically
loading other Apache modules.

For a specific Apache configuration, you can determine what Apache modules
will be loaded dynamically by using the ``-M`` option for the Apache executable.

On MacOS X, for the operating system supplied Apache the output from
running ``-M`` option, where the only additional module added is mod_wsgi,
is::

    $ /usr/sbin/httpd -M
    Loaded Modules:
     core_module (static)
     mpm_prefork_module (static)
     http_module (static)
     so_module (static)
     authn_file_module (shared)
     authn_dbm_module (shared)
     authn_anon_module (shared)
     authn_dbd_module (shared)
     authn_default_module (shared)
     authz_host_module (shared)
     authz_groupfile_module (shared)
     authz_user_module (shared)
     authz_dbm_module (shared)
     authz_owner_module (shared)
     authz_default_module (shared)
     auth_basic_module (shared)
     auth_digest_module (shared)
     cache_module (shared)
     disk_cache_module (shared)
     mem_cache_module (shared)
     dbd_module (shared)
     dumpio_module (shared)
     ext_filter_module (shared)
     include_module (shared)
     filter_module (shared)
     substitute_module (shared)
     deflate_module (shared)
     log_config_module (shared)
     log_forensic_module (shared)
     logio_module (shared)
     env_module (shared)
     mime_magic_module (shared)
     cern_meta_module (shared)
     expires_module (shared)
     headers_module (shared)
     ident_module (shared)
     usertrack_module (shared)
     setenvif_module (shared)
     version_module (shared)
     proxy_module (shared)
     proxy_connect_module (shared)
     proxy_ftp_module (shared)
     proxy_http_module (shared)
     proxy_ajp_module (shared)
     proxy_balancer_module (shared)
     ssl_module (shared)
     mime_module (shared)
     dav_module (shared)
     status_module (shared)
     autoindex_module (shared)
     asis_module (shared)
     info_module (shared)
     cgi_module (shared)
     dav_fs_module (shared)
     vhost_alias_module (shared)
     negotiation_module (shared)
     dir_module (shared)
     imagemap_module (shared)
     actions_module (shared)
     speling_module (shared)
     userdir_module (shared)
     alias_module (shared)
     rewrite_module (shared)
     bonjour_module (shared)
     wsgi_module (shared)
    Syntax OK

The names reflect that which would have been used with the LoadModule line
in the Apache configuration and not the name of the module file itself.

The order in which modules are listed can be important in some cases where
a module doesn't explicitly designate in what order a handler should be
applied relative to other Apache modules.

Global Accept Mutex
-------------------

Because Apache is a multi process server, it needs to use a global cross
process mutex to control which of the Apache child processes get the next
chance to accept a connection from a HTTP client.

This cross process mutex can be implemented using a variety of different
mechanisms and exactly which is used can vary based on the operating system.
Which mechanism is used can also be overridden in the Apache configuration
if absolutely required.

A simlar instance of a cross process mutex is also used for each mod_wsgi
daemon process group to mediate which process in the daemon process group
gets to accept the next request proxied to that daemon process group via the
Apache child processes.

The list of mechanisms which might be used to implement the cross process
mutex are as follows:

  * flock
  * fcntl
  * sysvsem
  * posixsem
  * pthread

In the event that there are issues which communicating between the Apache
child processes and the mod_wsgi daemon process in particular, it can be
useful to know what mechanism is used to implement the cross process mutex.

By default, the Apache configuration files would not specify a specific
mechanism, and instead which is used would be automatically selected by the
underlying Apache runtime libraries based on various build time and system
checks about what is the prefered mechanism for a particular operating
system.

Which mechanism is used by default can be determined from the build
information displayed by the ``-V`` option to the Apache executable described
previously. The particular entries of interest are those with 'SERIALIZE'
in the name of the macro.

On MacOS X, using operating system supplied Apache, the entries of interest
are::

    -D APR_USE_FLOCK_SERIALIZE
    -D APR_USE_PTHREAD_SERIALIZE

As the entries are used in order, what this indicates is that Apache will by
default use the 'flock' mechanism to implement the cross process mutex.

In comparison, on a Linux system, the entries of interest may be::

    -D APR_USE_SYSVSEM_SERIALIZE
    -D APR_USE_PTHREAD_SERIALIZE

which indicates that 'sysvsem' mechanism is instead used.

This mechanism is also what would be used by mod_wsgi by default as well for
the cross process mutex for daemon process groups.

This mechanism will be different where the AcceptMutex and WSGIAcceptMutex
directives are used.

If the AcceptMutex directive is defined in the Apache configuration file,
then what ever mechanism is specified will be used instead for Apache child
processes. Provided that Apache 2.2 or older is used, and WSGIAcceptMutex
is not specified, then when AcceptMutex is used, that will also then be used
by mod_wsgi daemon processes as well.

In the case of Apache 2.4 and later, AcceptMutex will no longer override the
default for mod_wsgi daemon process groups, and instead WSGIAcceptMutex must
be specified seperately if it needs to be overridden for both.

Either way, you should check the Apache configuration files as to whether
either AcceptMutex or WSGIAcceptMutex directives are used as they will
override the defaults calculated above. Under normal circumstances neither
should be set as default would always be used.

If wanting to look at overriding the default mechanism, what options exist
for what mechanism can be used will be dependent on the operating system
being used. There are a couple of ways this can be determined.

The first is to find the 'apr.h' header file from the Apache runtime library
installation that Apache was compiled against. In that you will find entries
similar to the 'USE' macros above. You will also find 'HAS' entries. In this
case we are interested in the 'HAS' entries.

On MacOS X, with the operating system supplied APR library, the entries in
'apr.h' are::

    #define APR_HAS_FLOCK_SERIALIZE           1
    #define APR_HAS_SYSVSEM_SERIALIZE         1
    #define APR_HAS_POSIXSEM_SERIALIZE        1
    #define APR_HAS_FCNTL_SERIALIZE           1
    #define APR_HAS_PROC_PTHREAD_SERIALIZE    0

The available mechanisms are those defined to be '1'.

Finding where the right 'apr.h' is located may be tricky, so an easier way
is to trick Apache into generating an error message to list what the available
mechanisms are. To do this, in turn, add entries into the Apache configuration
files, at global scope of::

    AcceptMutex xxx

and::

    WSGIAcceptMutex xxx

For each run the ``-t`` option on the Apache program executable.

On MacOS X, with the operating system supplied APR library, this yields::

    $ /usr/sbin/httpd -t
    Syntax error on line 501 of /private/etc/apache2/httpd.conf:
    xxx is an invalid mutex mechanism; Valid accept mutexes for this platform \
     and MPM are: default, flock, fcntl, sysvsem, posixsem.

for AcceptMutex and for WSGIAcceptMutex::

    $ /usr/sbin/httpd -t
    Syntax error on line 501 of /private/etc/apache2/httpd.conf:
    Accept mutex lock mechanism 'xxx' is invalid. Valid accept mutex mechanisms \
     for this platform are: default, flock, fcntl, sysvsem, posixsem.

The list of available mechanisms should normally be the same in both cases.

Using the value of 'default' indicates that which mechanism is used is left
up to the APR library.

Python Shared Library
---------------------

When mod_wsgi is built, the 'mod_wsgi.so' file should be linked against
Python via a shared library. If it isn't and it is linked against a static
library, various issues can arise. These include additional memory usage,
plus conflicts with mod_python if it is also loaded in same Apache.

To validate that 'mod_wsgi.so' is using a shared library for Python, on most
UNIX systems the 'ldd' command is used. For example::

    $ ldd mod_wsgi.so
     linux-vdso.so.1 =>  (0x00007fffeb3fe000)
     libpython2.5.so.1.0 => /usr/local/lib/libpython2.5.so.1.0 (0x00002adebf94d000)
     libpthread.so.0 => /lib/libpthread.so.0 (0x00002adebfcba000)
     libdl.so.2 => /lib/libdl.so.2 (0x00002adebfed6000)
     libutil.so.1 => /lib/libutil.so.1 (0x00002adec00da000)
     libc.so.6 => /lib/libc.so.6 (0x00002adec02dd000)
     libm.so.6 => /lib/libm.so.6 (0x00002adec0635000)
     /lib64/ld-linux-x86-64.so.2 (0x0000555555554000)

What you want to see is a reference to an instance of 'libpythonX.Y.so'.
Normally the operating system shared library version suffix would always be
'1.0'. What it is shouldn't really matter though.

This reference should refer to the actual Python shared library for your
Python installation.

Do note though, that 'ldd' will take into consideration any local user
setting of the 'LD_LIBRARY_PATH' environment variable. That is, 'ldd' will
also search any directories listed in that environment variable for shared
libraries.

Although that environment variable may be defined in your user account, it
will not normally be defined in the environment of the account that Apache
starts up as. Thus, it is important that you unset the 'LD_LIBRARY_PATH'
environment variable when running 'ldd'.

If you run the check with and without 'LD_LIBRARY_PATH' set and find that
without it that a different, or no Python shared library is found, then you
will likely have a problem. For the case of it not being found, Apache will
fail to start. For where it is found but it is a different installation to
that which you want used, subtle problems could occur due to C extension
modules for Python being used which were compiled against that installation.

For example, if 'LD_LIBRARY_PATH' contained the directory '/usr/local/lib'
and you obtained the results above, but when you unset it, it picked up
shared library from '/usr/lib' instead, then you may end up with problems
if for a different installation. In this case you would see::

    $ unset LD_LIBRARY_PATH
    $ ldd mod_wsgi.so
     linux-vdso.so.1 =>  (0x00007fffeb3fe000)
     libpython2.5.so.1.0 => /usr/lib/libpython2.5.so.1.0 (0x00002adebf94d000)
     libpthread.so.0 => /lib/libpthread.so.0 (0x00002adebfcba000)
     libdl.so.2 => /lib/libdl.so.2 (0x00002adebfed6000)
     libutil.so.1 => /lib/libutil.so.1 (0x00002adec00da000)
     libc.so.6 => /lib/libc.so.6 (0x00002adec02dd000)
     libm.so.6 => /lib/libm.so.6 (0x00002adec0635000)
     /lib64/ld-linux-x86-64.so.2 (0x0000555555554000)

Similarly, if not found at all, you would see::

    $ unset LD_LIBRARY_PATH
    $ ldd mod_wsgi.so
     linux-vdso.so.1 =>  (0x00007fffeb3fe000)
     libpython2.5.so.1.0 => not found
     libpthread.so.0 => /lib/libpthread.so.0 (0x00002adebfcba000)
     libdl.so.2 => /lib/libdl.so.2 (0x00002adebfed6000)
     libutil.so.1 => /lib/libutil.so.1 (0x00002adec00da000)
     libc.so.6 => /lib/libc.so.6 (0x00002adec02dd000)
     libm.so.6 => /lib/libm.so.6 (0x00002adec0635000)
     /lib64/ld-linux-x86-64.so.2 (0x0000555555554000)

If you have this problem, then it would be necessary to set 'LD_RUN_PATH'
environment variable to include directory containing where Python library
resides when building mod_wsgi, or set 'LD_LIBRARY_PATH' in startup file
for Apache such that it is also set for Apache when run. For standard
Apache installation the latter would be done in 'envvars' file in same
directory as Apache program executable. For some Linux installations would
need to be done in init scripts for Apache.

Note that MacOS X doesn't use 'LD_LIBRARY_PATH' nor have 'ldd'. On MacOS X,
instead of 'ldd' you can use 'otool -L'::

    $ otool -L mod_wsgi.so 
    mod_wsgi.so:
      /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 125.2.0)
      /System/Library/Frameworks/Python.framework/Versions/2.6/Python (compatibility version 2.6.0, current version 2.6.1)

If using standard MacOS X compilers and not using Fink or !MacPorts, there
generally should not ever be any issues with whether it is a shared library
or not as everything should just work.

The only issue with MacOS X is that for whatever reason, the location
dependency for the shared library (framework) isn't always encoded into
'mod_wsgi.so' correctly. This seems to vary between what Python installation
was used and what MacOS X operating system version. In this case, if
multiple installations of same version of Python in different locations,
may find the system installation rather than your custom installation.

In that situation you may need to use the ``--disable-framework`` option to
'configure' script for mod_wsgi. This doesn't actually disable use of the
framework, but does change how it links to use a more traditional library
style linking rather than framework linking. This seems to resolve the
problems in most cases.

Python Installation In Use
--------------------------

Although the 'mod_wsgi.so' file may be finding a specific Python shared
library and that may be from the correct installation, the Python library
when initialised doesn't actually know from where it came. As such, it uses
a series of checks to try and determine where the Python installation is
actually located.

This check has various subtleties and how it works varies depending on the
platform used. At its simplest though, on most UNIX systems it will check
all directories listed in the 'PATH' environment variable of the process.
In each of those directories it will look for the 'python' program. When it
finds such a file, it will then look for a corresponding 'lib' directory
containing a valid Python installation for the same version of Python as is
being run.

When it finds such a directory, the home for the Python installation will
be taken as being the parent directory of the directory containing the
'python' program file found.

Because this search is dependent on the 'PATH' environment variable, which
is likely set to a minimal set of directories for the Apache user, then if
you are using a Python installation in a non standard location, then it may
not properly find the location of that installation.

The easiest way to validate which Python installation is being used is to
use a test WSGI script to output the value of 'sys.prefix'::

    import sys 

    def application(environ, start_response):
        status = '200 OK'

        output = u''
        output += u'sys.version = %s\n' % repr(sys.version)
        output += u'sys.prefix = %s\n' % repr(sys.prefix)

        response_headers = [('Content-type', 'text/plain'),
                            ('Content-Length', str(len(output)))]
        start_response(status, response_headers)

        return [output.encode('UTF-8')]

For standard Python installation on a Linux system, this would produce
something like::

    sys.version = "'2.6.1 (r261:67515, Feb 11 2010, 00:51:29) \\n[GCC 4.2.1 (Apple Inc. build 5646)]'"
    sys.prefix = '/usr'

Thus, if you were expecting to pick up a separate Python installation
located under '/usr/local' or elsewhere, this would be indicative of a
problem.

It can also be worthwhile to check that the Python module search path also
looks correct. This can be done by using a test WSGI script to output the
value of 'sys.path'::

    import sys

    def application(environ, start_response):
        status = '200 OK'
        output = u'sys.path = %s' % repr(sys.path)

        response_headers = [('Content-type', 'text/plain'),
                            ('Content-Length', str(len(output)))]
        start_response(status, response_headers)

        return [output.encode('UTF-8')]

In both cases, even if incorrect location is being used for Python
installation and even if there is no actual Python installation of the
correct version under that root directory, then these test scripts should
still run as 'sys' module is a builtin module which can be satisified via
just the Python library.

If debugging, whether there is a Python installation underneath that root
directory, the subdirectory which you would want to look for is
'lib/pythonX.Y' corresponding to version of Python being used.

If the calculated directory is wrong, then you will need to use the
WSGIPythonHome directory to set the location to the correct value. The value
to use is what 'sys.prefix' is set to when the correct Python is run from
the command line and 'sys.prefix' output::

    >>> import sys
    >>> print sys.prefix
    /usr/local

Thus for case where installed under '/usr/local', would use::

    WSGIPythonHome /usr/local

Embedded Or Daemon Mode
-----------------------

WSGI applications can run in either embedded mode or daemon mode. In the
case of embedded mode, the WSGI application runs within the Apache child
processes themselves. In the case of daemon mode, they run within a
separate set of processes managed by mod_wsgi.

To determine what mode a WSGI application is running under, replace its
WSGI script with the test WSGI script as follows::

    import sys

    def application(environ, start_response):
        status = '200 OK'
        output = u'mod_wsgi.process_group = %s' % repr(environ['mod_wsgi.process_group']) 
        response_headers = [('Content-type', 'text/plain'),
                            ('Content-Length', str(len(output)))]
        start_response(status, response_headers)

        return [output.encode('UTF-8')]

If the configuration is such that the WSGI application is running in embedded
mode, then you will see::

    mod_wsgi.process_group = ''

This actually corresponds to the directive::

    WSGIProcessGroup %{GLOBAL}

having being used, or the same value being used to the 'process-group'
directive of WSGIScriptAlias. Do note though that these are also actually
the defaults for these if not explicitly defined.

If the WSGI application is actually running in daemon mode, then a non
empty string will instead be shown corresponding to the name of the daemon
process group used.

Sub Interpreter Being Used
--------------------------

As well as WSGI application being able to be delegated to run in either
embedded mode or daemon mode, within the process it ends up running in, it
can be delegated to a specific Python sub interpreter.

To determine which Python sub interpreter is being used within the process
the WSGI application is being run use the test WSGI script of::

    import sys

    def application(environ, start_response):
        status = '200 OK'
        output = u'mod_wsgi.application_group = %s' % repr(environ['mod_wsgi.application_group'])

        response_headers = [('Content-type', 'text/plain'),
                            ('Content-Length', str(len(output)))]
        start_response(status, response_headers)

        return [output.encode('UTF-8')]

If being run in the main interpreter, ie., the first interpreter created by
Python, this will output::

    mod_wsgi.application_group = ''

This actually corresponds to the directive::

    WSGIApplicationGroup %{GLOBAL}

having being used, or the same value being used to the 'application-group'
directive of WSGIScriptAlias.

The default for these if not defined is actually '%{RESOURCE}'. This will
be a value made up from the name of the virtual host or server, the port
on which connection was accepted and the mount point of the WSGI application.
The port however is actually dropped where port is 80 or 443.

An example of what you would expect to see is::

    mod_wsgi.application_group = 'tests.example.com|/interpreter.wsgi'

This corresponds to server name of 'tests.example.com' with connection
received on either port 80 or 443 and where WSGI application was mounted at
the URL of '/interpreter.wsgi'.

Single Or Multi Threaded
------------------------

Apache supports differing Multiprocessing Modules (MPMs) having different
attributes. One such difference is whether a specific Apache child process
uses multiple threads for handling requests or whether a single thread is
instead used.

Depending on how you configure a daemon process group when using daemon
mode will also dictate whether single or multithreaded. By default, if
number of threads is not explicitly specified for a daemon process group,
it will be multithreaded.

Whether a WSGI application is executing within a multithreaded environment
is important to know. If it is, then you need to ensure that your own code
and any framework you are using is also thread safe.

A test WSGI script for validating whether WSGI application running in
multithread configuration is as follows::

    import sys

    def application(environ, start_response):
        status = '200 OK'
        output = u'wsgi.multithread = %s' % repr(environ['wsgi.multithread'])

        response_headers = [('Content-type', 'text/plain'),
                            ('Content-Length', str(len(output)))]
        start_response(status, response_headers)

        return [output.encode('UTF-8')]

If multithreaded, this will yield::

    wsgi.multithread = True

Multithreaded would usually be true on Windows, on UNIX if running in embedded
mode and worker MPM is used by Apache, or if using daemon mode and number of
threads not explicitly set, or number of threads explicitly set to value other
than '1'.