\chapter{Deploying}

I am now going to try to get you to setup a small, tiny, little version of a good
deployment that matches the configuration of the site at \url{http://mongrel2.org}, with
all the examples running.  This configuration will give you all the tools you
need to make automated and managed deployments, but it is using small scale tools.
The idea is that you learn what is involved in a nice, easy-to-manage setup, using simple
things first, then you can extrapolate that out into your own setup or something
better.


\section{Mongrel2 Deployment Requirements}

It may seem obvious, but I'll go over the things you need in order to continue
on in this section:

\begin{description}
\item [Mongrel2] I know, hard to believe, but you actually need to have Mongrel2 installed.
\item [m2sh] Again, not sure why, but some folks think they don't need this.  Unless you've
    written your own, you need \shell{m2sh}.
\item [Python] Some systems (like Debian) don't install all of Python.  Make sure your Python setup is good.
\item [root] You'll need root access on your box.  Either through \shell{sudo} or some other means.
\item [Basic Python coding] Right now, you should be able to do some basic Python.
\end{description}

That will get you going at first and, as we go, we'll do various other setups to
get our application working.

\begin{aside}{Learning Python}
Why should you learn programming?  The trend is that if you are a system administrator
who can't code, you are on your way out.  Eventually, you'll be in charge of automating
systems; not manually managing them, and if you don't believe me then what do you think all
those managed service companies are doing?  Alright, so you need to learn to code, but
most of the books suck for really learning if you know nothing.

This is why I started my own book: \href{http://learnpythonthehardway.org}{Learn
Python The Hard Way}, for people who know nothing about programming but need or
want to learn.  It teaches Python, but it mostly teaches all the things
programmers actually learn before they learn programming.  When you're done
with my book, you'll have your ``programming brown belt''.  That means you can
then move onto one of many other free online books and \emph{really} learn
programming, and have a higher chance of actually learning it.

If you can't code Python then you can probably muddle through this and you may
learn something, but learning Python will be important later.

But \emph{don't} read ``Dive Into Python''. It is a horrible introduction.
\end{aside}

\subsection{Introducing procer}

When I started working on this little manual, I wanted to get you into setting up
a well-managed and automated deployment system.  The \shell{m2sh} program does
much of the automation you need, but Mongrel2 also has to talk to quite a few
separate little pieces that run as separate processes.  Trying to juggle all these
processes without a tool to help is a nightmare.  You end up writing init scripts
and merging them into your boot process and all sorts of crazy antics just so you
can run a stupid hello world demo.

What I needed was a ``user space process manager''.  These are programs that run other
programs, but, more importantly, try to keep those other programs running without much
human intervention.  When you need to deploy a ton of processes that all have to
be running, these USPMs are fantastic.  They usually read some startup profile describing
what needs to start and what they depend on, and then it kicks everything into gear
and watches them.  If any of the processes crash, they try to restart them.  Very simple.

There's just one catch: all of them suck.  There's \href{http://cr.yp.to/daemontools.html}{daemontools},
which barely builds (if at all) and then assumes that daemons don't fork.  Stupid.  There's
\href{http://www.fefe.de/minit/}{minit}, which bafflingly required dietlibc to even compile
and assumed it was going to be the one true init (not user space at all).  There's
\href{http://www.nico.schottelius.org/software/cinit/}{cinit}, which got through a compile,
then barfed on its documentation, and the end result is some huge number of weird
shell scripts to make it work, and, again, it wants to be the one true init.  Finally ,
\href{http://smarden.org/runit/}{runit} is some of the worst C code I've seen in years and
has the same weird design as daemontools.

After trying every single one, I just gave up.  Either they didn't build, were too complex,
expected to be the one true init, poorly documented, not maintained, and definitely not
going to work for this manual.  My only choice was to shave a yak and write my own.

The end result is \shell{procer}, which lives in \file{tools/procer} and does
most of what you need in a USPM\@.  It works a lot like daemontools or minit, but
is much simpler, with these differences:

\begin{enumerate}
\item It is much simpler, with only a single command to start all your stuff and
    keep it running.
\item It will build anywhere Mongrel2 builds, because it reuses the \file{libm2.a}
    library from the Mongrel2 project.
\item It doesn't want to be the one true init, or even expect to be running constantly.
    You can start it and stop it and it will only run what's not already running.
\item It assumes that programs will always daemonize and create a PID file.  This turns
    out to be way easier to manage than what daemontools does, so I'm sort of baffled why
    daemontools is how it is.
\item It has dependency management so that you can have processes start only after others
    have finished.
\item It still uses simple files to configure itself that are in separate directories.
\item It can be run as root and, like Mongrel2, it will drop privileges to the owner
    of the profile directory before it runs the command.  This is incredibly useful because
    it lets you setup scripts that run as other users without much configuration or fuss.
\item It is dinky, tiny and well written so you can understand it, even though it's written
    in C.
\item Best of all, I can use it in this book and you won't go insane trying to install it
    or use it like the others.
\end{enumerate}

Of course, if you have something else you like then, please, use it.  Anything that automates
process management will be your friend.  In this manual, to keep things simple and easily
understood, I'll be using \shell{procer} to tell you how to setup everything.

\begin{aside}{Alternatives to procer}
I wrote procer mostly for this book, but I also use it for my Mongrel2 deployments.
It works for me but you can try other solutions.  By default, Mongrel2 will work with
either daemontools/runit style, or init.d style launchers.  If Mongrel2 runs as a
regular user, it assumes that you want runit style (don't fork, write to stdout/stderr).
If you run as root, it assumes you want init.d style like what procer uses (fork, drop priv, 
chroot, etc.).

You should check out \href{http://github.com/peterkeen/proclaunch}{proclaunch} as another
alternative that is similar to procer, and inspired by procer, but written in Perl with a
few more features.

Either way, Mongrel2 is practical, and does generally the right thing with today's tools.
Want to use daemontools?  Fine, just run it \shell{mongrel2 config.sqlite server\_uuid} and it'll
work right.  Want to put it in init.d or use procer or similar?  Fine, run it as root.
\end{aside}


\subsection{Installing procer}

Installing procer is very easy.  It's a single little binary and it lives in
\file{tools/procer} in the Mongrel2 source.  Here's how you'd install it
totally from scratch as if you hadn't even built Mongrel2 yet:

\begin{code}{Install procer}
<< d['inputs/install_procer.sh|pyg|l'] >>
\end{code}

That's the entire install process, and now procer is in \file{/usr/local/bin}
so you can use it.  In the rest of this chapter you'll learn how to use \shell{procer}
by just setting up the Mongrel2 demo completely and messing around with it.



\section{The Plan}

We need to plan this deployment to make sure we get the end result correct:

\begin{enumerate}
\item Create a deployment area where everything will live.
\item Create a config.sqlite that will work with the demos in \shell{examples}.
\item Setup procer to run Mongrel2 and the three demo Python scripts for chat, handlertest, and mp3stream,
    and have it run the fake backend web.py project so we have something to proxy to.
\item Get all the static file content working.
\item Test out that procer is keeping things running and play with taking things down and up and using
    \shell{m2sh} to work with the deployment.
\end{enumerate}

Once you have this setup working, you can then start to make your own
deployments and tweak things as you need for your own applications.  Remember
that the goal is to get you to \emph{automate} everything as much as possible,
so you can go further than this then do it.

\section{Step 1: The Deployment Area}

We'll need a place to put all this stuff and run it so that Mongrel2 can chroot
there, procer knows where its profiles are, and its all nice and clean.  For these
instructions, we're just going to make some directories in your home directory,
but feel free to change this up later if you find a better way.

\begin{code}{Make Deployment Directories}
<< d['inputs/make_deployment_directories.sh|pyg|l'] >>
\end{code}

Hopefully, you're starting to see how you could easily automate this so that you don't
have to do this all the time.  I'm just showing you how to ``make the sausage'' so that
you know where everything goes.  Future versions of \shell{m2sh} will most likely
create deployment directories like this automatically.

What we've done here is the following:

\begin{enumerate}
  \item Setup a \file{\~{}/deployment} directory we'll put everything in.
  \item Created \file{run}, \file{tmp}, \file{logs}, and \file{profiles} that Mongrel2 and procer need to run.
  \item In profiles we started dirs for \file{chat}, \file{mp3stream}, \file{handlertest}, \file{web} and \file{mongrel2},
    that procer will read files out of to get all our gear up and running.
  \item Copied the \file{mongrel2.conf} example file over to our deployment so we can modify it.
  \item Initialized the \file{config.sqlite} file we'll be filling in with our modififed mongrel2.conf.
\end{enumerate}


\section{Step 2: The mongrel2.org Configuration}

Now we're ready to get the configuration working.  Here's the thing, though: you should
try to alter the configuration yourself.  I've already given you the file and you are
going to have to make the changes to meet the requirements for this deployment directory.
Here's what you have to change in \file{mongrel2.conf} to make everything work right:

\begin{enumerate}
\item Get rid of the \ident{test\_directory} handler, since we won't need it, and
    any routes that mention it.
\item Change the \ident{base} of \ident{chat\_demo\_dir} to \file{'static/chatdemo/'}, which we'll setup at the end.
\item Modify the server \ident{chroot} so that it's \file{/home/YOU/deployment/}.
\item Use the \shell{m2sh uuid} command to make some new UUIDs for all the
    existing ones.  This is optional, but probably a good idea to get in the
    habit now.
\item Change the \ident{port} for \ident{web\_app\_proxy} so it points to 8080 instead of 80.
\item Finally, change any mention of ``mongrel2.org'' into ``localhost'' so that you
    can run it locally.
\end{enumerate}

Once you have that all edited, you should be able to run \shell{m2sh load -db
config.sqlite -config mongrel2.conf} and it'll just load it up.  Try using
\shell{m2sh servers} and \shell{m2sh hosts} to take a peek.


To test it out at this stage you can just run the config.sqlite that you
did with these commands:

\begin{code}{Testing The Initial Configuration}
<< d['inputs/testing_initial_config.sh|pyg|l'] >>
\end{code}

That's enough to make sure it runs, but you've got nothing running,
so it mostly won't work at all.  Just start up and then kill it
right after.


\section{Step 3: Setup procer}

Now we want to make \shell{procer} start everything for us and keep
it running.  How \shell{procer} works is you put a few special
files into a directory in \file{profiles}.  This directory (say \file{chat})
is the profile for that app.  When you start \shell{procer}, you
point it at the main \file{profiles} directory and it tries to run it.
It's dead simple and very easy to automate, so we'll do it by hand
and then you can do some automation later.

Let's first setup a basic config that gets our skeleton profiles
and make sure procer can run everything:

\begin{code}{Skeleton procer Setup}
<< d['inputs/skeleton_procer_setup.sh|pyg|l'] >>
\end{code}

With all of that, you can then try to run \shell{procer} to watch
it fail but still try to run everything:

<< d['inputs/run_procer_first_time.sh|pyg|l'] >>

This is assuming that you are still in the \file{profiles} directory.
You should see the file \file{error.log} get created and
probably some messages printed to the screen.  Just ignore any
mention of Mongrel2 since that's probably just cruft from the \file{libm2.a}
we haven't removed.

Take a look in the \file{error.log} and you'll see it's not necessarily
errors but information on how things were run.  You should
see something like this for each profile:

\begin{code}{First Dummy Run Of procer}
\begin{lstlisting}
DEBUG procer.c:232: Loading 5 actions.
DEBUG procer.c:83: STARTED chat
ERROR Failed to open PID file /home/zedshaw/deployment/profiles/chat/chat.pid for reading.
ERROR Failed to open PID file /home/zedshaw/deployment/profiles/chat/chat.pid for reading.
INFO  No previous Mongrel2 running, continuing on.
DEBUG procer.c:37: ACTION: command=/home/zedshaw/deployment/profiles/chat/run, pid_file=/home/zedshaw/deployment/profiles/chat/chat.pid, restart=1, depends=(null)
DEBUG procer.c:56: WAITING FOR CHILD.
INFO  Now running as UID:1000, GID:1000
DEBUG procer.c:60: Command ran and exited successfully, now looking for the PID file.
ERROR chat didn't make pidfile /home/zedshaw/deployment/profiles/chat/chat.pid.
\end{lstlisting}
\end{code}

I've cleaned this up a bit and, again, ignore that it's saying ``Mongrel2'';
that's just cruft from the library since it was originally designed
for Mongrel2.  What you can see here is the following:

\begin{enumerate}
\item It starts up and says it found 5 profiles.
\item It starts chat, and says there's no PID file so it's good to continue.
\item It reports what ACTION it's running, so you can see the config.
\item It spawns off your \file{run} script, \emph{drops privilege}
     and says it's WAITING for your script to exit.
\item After your script runs, it looks for the PID file you gave in \file{pid\_file} and, if it's not there, it exits that action.
\item It does this for all of them and, since none of them run right, \shell{procer} exits.
\end{enumerate}

Next up, let's get Mongrel2 running inside \shell{procer}:

\begin{code}{procer Config For Mongrel2}
<< d['inputs/procer_config_for_mongrel2.sh|pyg|l'] >>
\end{code}

Obviously, you don't have to use a series of \shell{echo} commands to
make these scripts.  You can edit them just fine, we're just doing it
this way so that you can follow along easier.

Now, make sure you don't have any other Mongrel2 processes running,
and then start procer again to see if it starts this configuration
correctly.


\begin{code}{Using procer To Run Mongrel2}
<< d['inputs/using_procer_to_run_mongrel2.sh|pyg|l'] >>
\end{code}

To watch \shell{procer} in action, try doing \shell{m2sh stop -db config.sqlite
-host localhost -murder} and then look at \file{profiles/error.log} and watch
Mongrel2 come right back.


\subsection{The Python Examples}

We've got a good setup of \shell{procer} going and it keeps Mongrel2
running, so let's setup a similar thing for each of our little
Python demos that we'll need.  In order to do this, though, we sort
of have to ``hack in'' making them daemonize and create PID files with
a little shell script help.  Let's start with the \ident{chat} demo
and, assuming your mongrel2 source is in \file{\~{}/projects/mongrel2},
you will change \file{profiles/chat/run} to be like this:

\begin{code}{Run Script For Chat Demo}
<< d['inputs/procer_script_for_chat_demo.sh|pyg|l'] >>
\end{code}

This little script uses some funky features you might not be familiar
with, but which are nice to learn, so let's take a look:


\begin{enumerate}
\item The first trick is \shell{set -e}, which tells bash to bail if there's
    any errors in your script.  This is a \emph{huge} life saver in system
    scripts.
\item Next, you point some variables at where the deployment and Mongrel2 source
    live, remembering to not type YOU but your username.
\item After that, you run the \file{chat.py} using a program called \shell{nohup}.
    This basically daemonizes your script by redirecting output and
    preventing the program from exiting, and then you background it with \verb|&|.
\item The final thing we do is echo the magic variable \verb|$!| (the PID of the
  last process started in the background) to the \file{chat.pid} file in the
  profile directory.
\end{enumerate}

When you run this manually, you should see something like this:

<< d['inputs/run_procer_chat_demo.sh|pyg|l'] >>

After all that, you can then try out \shell{procer} again to see if it
properly runs the chat demo as well as mongrel2:

\begin{code}{Running procer With Chat Demo}
<< d['inputs/running_procer_with_chat_demo.sh|pyg|l'] >>
\end{code}

If you go look at \file{profiles/error.log}, you'll see that \shell{procer}
is also running each of them as the right user, with chat being run
as you, but Mongrel2 being run as root so it can chroot/drop privileges properly.

Rather than give you a walk through each of these setups, here's the
run scripts for the remaining files:

\begin{code}{Remaining Run Scripts}
  \file{profiles/handlertest/run}
\hrule

<< d['inputs/procer_handlertest_run.sh|pyg|l'] >>

\file{profiles/mp3stream/run}
\hrule

<< d['inputs/procer_mp3stream_run.sh|pyg|l'] >>

\file{profiles/web/run}
\hrule

<< d['inputs/procer_web_run.sh|pyg|l'] >>
\end{code}


\subsection{Testing The New Setup}

Once everything is running and \shell{procer} is maintaining it, you
just need to see if things work.  Here's some curl commands to try:

\begin{code}{Testing With Curl}
<< d['inputs/testing_procer_setup.sh|pyg|l'] >>
\end{code}


\subsection{Nice Features of Procer}

There's some nice subtle features you get from using \shell{procer}
to run your stuff:

\begin{description}
\item [Faster Development]  A great thing about \shell{procer} is once you get all of this setup,
    it cuts down on a lot of your setup time and development time because
    it will \emph{properly} restart things for you.  This means you can
    simply make changes to code or configs, and then just kill the process and
    procer will kick it back over automatically.
\item [Easy Automation]  You should start to see how you could automate creating
    profiles for new processes since the setup is consistent.
  \item [\file{profiles/run.log}]  All your commands will have their output sent to
    this file so you can see how they might be blowing up in your scripts.
\item [Restart State Maintained]  Since procer is just tracking PID files and
    processes, if you shut it down, it won't kill the world.  When you start
    it back up, it just starts new stuff or stuff it needs, then goes back to
    supervising.  This means you can change the configs for procer then just
    kick it over and it'll do the right thing.
\end{description}

The key thing, though, is that you now have the whole application for
the mongrel2.org demo up and running, including automated process management,
configuration, and managing everything.

\section{Step 4: Static Content}

The final thing we have to do is get the static content we need to try
out the chat demo:

\begin{code}{Setting Up Static Content}
<< d['inputs/setting_up_static_content.sh|pyg|l'] >>
\end{code}

If you get a good response then you should be able to go to
\url{http://localhost:6767/chatdemo/} and the chat should work.  Notice
also that you just killed mongrel2 with \shell{m2sh} and it came back
because of \shell{procer}.  If you do your curl check too fast,
you might miss it, so just wait a bit.

\section{Step 5: Testing And Troubleshooting}

You should have been testing the configuration as you went, but the
main things to test are:

\begin{enumerate}
\item The /chatdemo/ works and you can send messages. Try a few different
    browsers.
\item You can get a simple message from the /handlertest/ and that's about it.
\item See if you can get the mp3streamer to stream some mp3s.  Put a few
    in its directory, then kill it so \shell{procer} brings it back.  Then,
    point \shell{mplayer} at \url{http://localhost:6767/mp3stream} and
    it should work.
\item Check that you can make the proxy go to the web.py app you start
    in the chat demo's directory.
\item See if you can stop things and have \shell{procer} bring them back.
\item Stop \shell{procer} and then start it again to see if it properly
    doesn't step on things.
\end{enumerate}

If you run into problems, make sure that you can run each little
piece and that the files you were supposed to make are correct.
The best tool to use is \shell{diff}.

\section{Further Improvements}

That ends this chapter, and at this point you should know how to setup
nearly everything Mongrel2 has to offer right now.  You should have a good
idea of how \shell{procer} will work or not for your real deployments, and
how it's used by me for my own deployments.

A \emph{major} improvement that we may eventually make is automating
setup of \shell{procer} profiles, and just better overall management
of the profiles with \shell{m2sh}.  If you feel like hacking on that,
just go ahead and try and let us know.

Other than that, automate, automate, automate.

\section{Deployment Tips}

Mongrel2 enforces the correct behavior when you run as root, which is to drop priv and
chroot.  This makes the server more secure, and it also simplifies your deployments.
Since everything you do always runs in a chroot, you now just need to rsync that chroot
directory, or put it into a git or hg repo, and you're set.  You're literally forced to
make your deployments portable to different directories and systems.

As of the 1.0 push for Mongrel2, we haven't done much work on how you deploy all
the different languages.  They sort of sprung up during development and our plan is
to expand that out in the 2.0 version so that deployment is very well documented
for all the different languages we support.  That means you'll probably run into
some snags and things we didn't anticipate.

The following are some general points we've come up with while deploying our
own apps, with more to come as we work on the 2.0 version:

\begin{enumerate}
\item Don't run things as root if you can.  It's bad habit that everyone tries to
do their sysadmin completely as root, so Mongrel2 is designed to be run very easily
as under a regular user account.  The only time you really should be running as root
is when you do a quick -sudo to m2sh to start \file{mongrel2} up so it can chroot.

\item Use the chroot to keep your deployment simpler.  I literally do all my work
locally and then just rsync my changes up to my remote staging server.  Everything
has to live in the chroot anyway, and the chroot enforces that it is completely self-contained.

\item Use Python's virtualenv or anything similar to get yourself a totally local environment.
Too many systems, such as OSX, have very outdated packages and will change versions on you
without telling you.  The best way to make sure your software keeps working (and works as
one cohesive deployment) is to use a virtualenv inside your chroot.  It should even work
cross-platform if you don't have compiled packages in there.

\item Create a user for your application and live in there.  I don't have any root
access on my stuff.  \emph{Everything} is run as a user named after the website, and
is deployed right in the /home/USER directory.  I login as that user, manage as that user,
and I don't give them sudo access.  For the times when I need to sudo to restart or run
Mongrel2, I then use a separate login that I have open (with screen) and do it there.  This
reduces your risk of hacks, but also just simplifies things.  It's no problem for me to
move my configuration over to new machines with this setup, or deploy clusters.  I know that as
long as there's the right user on the target, I'm set.

\item Use \href{http://www.gnu.org/software/screen/}{GNU screen} or die.

\item Keep your config.sqlite and the .conf file in your chroot, and keep your content and
everything else under that.  This makes sure that the config isn't accessible outside your
content directories.  Mongrel2 helps you get this right by not allowing certain Dir
configurations that would expose your chroot to the world.
\end{enumerate}

There's a few additional tips for people who want to use alternative process supervision
like daemontools, runit, or init.d setups.  No matter what you use, you should probably
follow this advice:

\begin{enumerate}
\item Whatever you use for process management, make sure it can run stuff as \emph{not} root
and can do chroot for you.  If you're running your Mongrel2 as root, you're doing it wrong.
Actually, if you're running any services as root that don't absolutely need to be, you're doing
it wrong.

\item Mongrel2 is happy to run as a regular user, and assumes that if you do not run as root,
then you probably want to run under daemontools or similar.  It won't chroot or drop priv and
logs to stdout/stderr.

\item If you need to bind to port 80 but run under daemontools as a regular user, then use
\href{http://manpages.ubuntu.com/manpages/lucid/man1/privbind.1.html}{privbind} to do it.
This tool will run any command, like \file{mongrel2} but it does it in a way that lets the
executable grab ports below 1024.  This restriction on ports is actually really stupid so
don't worry about doing this.

\item Make sure your process monitor is not a single point of failure.  Some of
them out there will take your whole world down if they crash.  Try doing a
harsh \ident{kill} on your process manager and see how it behaves.  As much as
they like to tell you not to worry about this because they ``run forever'',
everything has bugs and stupid people tend to kill things they don't get.
If taking one process down nukes your whole server, then that's a bad
design.

\end{enumerate}

As we work on the next phase of Mongrel2 development, this will improve, so watch for
news about deployment and real applications.