Title: Unit Testing
Lamson provides the
"lamson.testing":http://lamsonproject.org/docs/api/lamson.testing-module.html
to help with writing unit tests. It doesn't do everything for you, but does
enough that you can TDD your email interactions with pretend users. It
includes features for checking messages get delivered to queues, checking
spelling, and running things through a fake or real relay.
h2. The Log Server
The first thing you need to do for testing is to run the "log server":
$ lamson log
The log server acts as the smart relay host you've configured in your @config/settings.py@
file by default. What it does is take all emails that your Lamson application
"sends out" and redeposits them into @run/queue@. This queue directory is then also
used by @lamson.testing@ for checking that emails were sent out. When anything goes
wrong, you can look in this directory and see what is getting sent out.
bq. An alternative to this setup would be to have all of the sending/routing/relaying done
internal to the whole testing framework, similar to how @lamson.testing.RelayConversation@ works.
However, I found that this made testing that your server actually sends proper emails much
too difficult.
h2. Test Organization
Lamson organizes tests into directories that match the things you're testing:
librelist $ ls -l tests/
total 16
drwxr-xr-x 6 zedshaw staff 204 Aug 19 17:01 handlers
drwxr-xr-x 8 zedshaw staff 272 Aug 18 15:50 model
drwxr-xr-x 3 zedshaw staff 102 Aug 18 15:50 templates
In most of the projects I've only rarely used template tests, but I'll cover
them below. Model tests make sure that any @app.model@ classes work right, and
handler tests make sure that any @app.handler@ classes work. Nice and simple.
You can add any other directories you want to this, and you can also use
doctests if you want. This comes free with
"nosetests":http://somethingaboutorange.com/mrl/projects/nose/0.11.1/ and is
very handy.
h2. Handler Tests
We'll use an example from the "librelist.com":http://librelist.com/ code base
that validates that a user can subscribe, unsubscribe, and post a message to a
mailing list. This test was primarily written in a TDD style, since generally
interactions and usability testing works better
"TDD":http://en.wikipedia.org/wiki/Test-driven_development style.
First you have a common preamble of modules that you need to include:
from nose.tools import *
from lamson.testing import *
from config import settings
import time
from app.model import archive, confirmation
Right away you notice we just include everything from @nose.tools@ and
@lamson.testing@ so that we can use it directly. Yes, this violates Python
style guidelines, but practicality is more important than dogmatic slavery to
supposed standards.
After that we include the @config.settings@, and two modules from @app.model@ that we'll
use to check that everything was working.
bq. Notice that we don't include anything from @app.handlers@ directly. These tests are meant
to be from the perspective of a user interacting with the handler via emails.
Once we have that we do a little setup to clear set some common variables and clear
out some queues we'll need to check:
queue_path = archive.store_path('test.list', 'queue')
sender = "sender-%s@sender.com" % time.time()
host = "librelist.com"
list_name = "test.list"
list_addr = "test.list@%s" % host
client = RouterConversation(sender, 'Admin Tests')
def setup():
clear_queue("run/posts")
clear_queue("run/spam")
Most of these are just variables used in tests later, but the big one is the
@client@ variable. It's a
"lamson.testing.RouterConversation":http://lamsonproject.org/docs/api/lamson.testing.RouterConversation-class.html
class that lets you simulate delivering email to your Lamson project.
bq. There's also a
"lamson.testing.TestConversation":http://lamsonproject.org/docs/api/lamson.testing.TestConversation-class.html
class that actually uses your real @config/settings.py@ to connect to a Relay.
This isn't used so much, but is intended for running "smoke tests" against a
newly deployed server.
With that we're ready to write out first handler test:
def test_new_user_subscribes_with_invalid_name():
client.begin()
client.say('test-list@%s' % host, "I can't read!", 'noreply')
client.say('test=list@%s' % host, "I can't read!", 'noreply')
clear_queue()
client.say('unbounce@%s' % host, "I have two email addresses!")
assert not delivered('noreply')
assert not delivered('unbounce')
client.say('noreply@%s' % host, "Dumb dumb.")
assert not delivered('noreply')
This is the longest of the tests, and shows all the various things you can do
with the @lamson.testing@ gear. Here's what we're doing in order:
# Call @client.begin@ to clear out queues and state and start fresh.
# Use @client.say@ to send an email from that client to your Lamson application. Notice that you configured the RelayConversation to pretend to be one person with each email getting the same subject line.
# Use @lamson.testing.clear_queue@ when you want to make sure the queue is clean.
# Use @lamson.testing.delivered@ to check if a certain message from someone is in the queue.
With that you can do pretty much everything you need to send an email and make
sure you get proper replies.
Here's another example:
def test_new_user_subscribes():
client.begin()
msg = client.say(list_addr, "Hey I was wondering how to fix this?",
list_name + '-confirm')
client.say(msg['Reply-To'], 'Confirmed I am.', 'noreply')
clear_queue()
Notice in this example we have a fourth parameter @list_name + '-confirm'@ and
we get a @msg@ back from our call to @client.say@. This basically combines
@client.say@ with @delivered@ to do it in one shot. Very commonly, you'll want
to say something to your server and make sure you got a certain response, and
then do something with that response. This is how you do that.
We then use this '-confirm' email message to actually subscribe the fake user.
Finally, here's two more examples:
def test_existing_user_unsubscribes():
test_new_user_subscribes()
msg = client.say(list_name + "-unsubscribe@%s" % host,
"I would like to unsubscribe.", 'confirm')
client.say(msg['Reply-To'], 'Confirmed yes I want out.', 'noreply')
def test_existing_user_posts_message():
test_new_user_subscribes()
msg = client.say(list_addr, "Howdy folks, I was wondering what this is?",
list_addr)
# make sure it gets archived
assert delivered(list_addr, to_queue=queue(queue_path))
In @test_existing_user_unsubscribes@ what we do is call
@test_new_user_subscribes@ to go through that process again, and then we chain
off that to do an unsubscribe. There's really nothing new here other than that
little trick.
In @test_existing_user_posts_message@ we do the usual send a message and expect
a reply, but then we *also* make sure that this message was delivered to the
archiver queue.
Apart from those methods and techniques, there's really nothing more to doing a
handler test. The only additional thing would be using
"assert_in_state":http://lamsonproject.org/docs/api/lamson.testing-module.html#assert_in_state
to make sure that your handler is in a particular state. I'd recommend against
doing that too much in a handler test, since it will make your tests brittle.
I only do it when the state is very important, such as when checking that they
are in a SPAMMING or BOUNCING state that I need to enforce.
h2. Model Tests
There's less functionality available in @lamson.testing@ for doing your models. The theory
is that your models will be classes, modules, and ORM that you need to perform the majority
of your storage and analysis. Since has very little to do with email you probably won't
use @lamson.testing@ as much.
About the only things you might use are APIs for checking that queues get certain messages
in them, and that certain users are in certain states.
Here's a quick example from "librelist.com":http://librelist.com again that tests how
archives work:
from nose.tools import *
from lamson.testing import *
from lamson.mail import MailRequest, MailResponse
from app.model import archive, mailinglist
import simplejson as json
import shutil
queue_path = archive.store_path('test.list', 'queue')
json_path = archive.store_path('test.list', 'json')
def setup():
clear_queue(queue_path)
shutil.rmtree(json_path)
def teardown():
clear_queue(queue_path)
shutil.rmtree(json_path)
def test_archive_enqueue():
msg = MailResponse(From="zedshaw@zedshaw.com", To="test.list@librelist.com",
Subject="test message", Body="This is a test.")
archive.enqueue('test.list', msg)
assert delivered('zedshaw', to_queue=queue(queue_path))
This is the usual initial setup, and then some extras to make sure that the "JSON archives":http://librelist.com/browser/ is working. You'll notice that we hand construct various messages, call methods on the @app.model.archive@ module, and then use @delivered@ to make sure they're correctly delivered.
h2. Template Tests
Typically you really can only test that your templates are spelled right, or that your templates
render when given certain locals. I've found that automated testing of templates isn't incredibly
useful yet, so the only one I've written is from the "oneshotblog":http://oneshotblog.com/ example:
from nose.tools import *
from lamson.testing import *
from lamson import view
import os
from glob import glob
def test_spelling():
message = {}
original = {}
for path in glob("app/templates/mail/*.msg"):
template = "mail/" + os.path.basename(path)
result = view.render(locals(), template)
spelling(template, result)
This uses @lamson.testing.spelling@ to make sure that each template renders and
that it is spelled correctly. This uses
"PyEnchant":http://www.rfk.id.au/software/pyenchant/ to do the checking, which
turns out to be rather annoying. If you are interested in improving the
template testing setup, then feel free to talk about your ideas on "the lamson
mailing list":mailto:lamson@librelist.com (but bring code, talk is cheap).
h2. Conclusion
Hopefully you'll be able to develop your application using good testing techniques with
the @lamson.testing@ API. If you find additional testing patterns that could be included
then "talk about them on the lamson mailing list":mailto:lamson@librelist.com to see
if they're general enough for others.