File: customizing.txt

package info (click to toggle)
roundup 1.2.1-10%2Betch1
links: PTS
area: main
in suites: etch
size: 4,764 kB
ctags: 3,756
sloc: python: 30,296; sh: 1,497; perl: 23; makefile: 22
file content (4632 lines) | stat: -rw-r--r-- 176,648 bytes
parent folder | download | duplicates (2)
===================
Customising Roundup
===================

:Version: $Revision$

.. This document borrows from the ZopeBook section on ZPT. The original is at:
   http://www.zope.org/Documentation/Books/ZopeBook/current/ZPT.stx

.. contents::
   :depth: 1

What You Can Do
===============

Before you get too far, it's probably worth having a quick read of the Roundup
`design documentation`_.

Customisation of Roundup can take one of six forms:

1. `tracker configuration`_ changes
2. database, or `tracker schema`_ changes
3. "definition" class `database content`_ changes
4. behavioural changes, through detectors_
5. `security / access controls`_
6. change the `web interface`_

The third case is special because it takes two distinctly different forms
depending upon whether the tracker has been initialised or not. The other two
may be done at any time, before or after tracker initialisation. Yes, this
includes adding or removing properties from classes.


Trackers in a Nutshell
======================

Trackers have the following structure:

=================== ========================================================
Tracker File        Description
=================== ========================================================
config.ini          Holds the basic `tracker configuration`_                 
schema.py           Holds the `tracker schema`_                              
initial_data.py     Holds any data to be entered into the database when the
                    tracker is initialised.
db/                 Holds the tracker's database                             
db/files/           Holds the tracker's upload files and messages            
db/backend_name     Names the database back-end for the tracker            
detectors/          Auditors and reactors for this tracker                   
extensions/         Additional web actions and templating utilities.
html/               Web interface templates, images and style sheets         
=================== ======================================================== 


Tracker Configuration
=====================

The ``config.ini`` located in your tracker home contains the basic
configuration for the web and e-mail components of roundup's interfaces.

Changes to the data captured by your tracker is controlled by the `tracker
schema`_.  Some configuration is also performed using permissions - see the 
`security / access controls`_ section. For example, to allow users to
automatically register through the email interface, you must grant the
"Anonymous" Role the "Email Access" Permission.

The following is taken from the `Python Library Reference`__ (May 20, 2004)
section "ConfigParser -- Configuration file parser":

 The configuration file consists of sections, led by a "[section]" header
 and followed by "name = value" entries, with line continuations on a
 newline with leading whitespace. Note that leading whitespace is removed
 from values. The optional values can contain format strings which
 refer to other values in the same section. Lines beginning with "#" or ";"
 are ignored and may be used to provide comments. 

 For example::

   [My Section]
   foodir = %(dir)s/whatever
   dir = frob

 would resolve the "%(dir)s" to the value of "dir" ("frob" in this case)
 resulting in "foodir" being "frob/whatever".

__ http://docs.python.org/lib/module-ConfigParser.html

Section **main**
 database -- ``db``
  Database directory path. The path may be either absolute or relative
  to the directory containig this config file.

 templates -- ``html``
  Path to the HTML templates directory. The path may be either absolute
  or relative to the directory containig this config file.

 admin_email -- ``roundup-admin``
  Email address that roundup will complain to if it runs into trouble. If
  the email address doesn't contain an ``@`` part, the MAIL_DOMAIN defined
  below is used.

 dispatcher_email -- ``roundup-admin``
  The 'dispatcher' is a role that can get notified of new items to the
  database. It is used by the ERROR_MESSAGES_TO config setting. If the
  email address doesn't contain an ``@`` part, the MAIL_DOMAIN defined
  below is used.

 email_from_tag -- default *blank*
  Additional text to include in the "name" part of the From: address used
  in nosy messages. If the sending user is "Foo Bar", the From: line
  is usually: ``"Foo Bar" <issue_tracker@tracker.example>``
  the EMAIL_FROM_TAG goes inside the "Foo Bar" quotes like so:
  ``"Foo Bar EMAIL_FROM_TAG" <issue_tracker@tracker.example>``

 new_web_user_roles -- ``User``
  Roles that a user gets when they register with Web User Interface.
  This is a comma-separated list of role names (e.g. ``Admin,User``).

 new_email_user_roles -- ``User``
  Roles that a user gets when they register with Email Gateway.
  This is a comma-separated string of role names (e.g. ``Admin,User``).

 error_messages_to -- ``user``
  Send error message emails to the ``dispatcher``, ``user``, or ``both``?
  The dispatcher is configured using the DISPATCHER_EMAIL setting.
  Allowed values: ``dispatcher``, ``user``, or ``both``

 html_version -- ``html4``
  HTML version to generate. The templates are ``html4`` by default.
  If you wish to make them xhtml, then you'll need to change this
  var to ``xhtml`` too so all auto-generated HTML is compliant.
  Allowed values: ``html4``, ``xhtml``

 timezone -- ``0``
  Numeric timezone offset used when users do not choose their own
  in their settings.

 instant_registration -- ``yes``
  Register new users instantly, or require confirmation via
  email?
  Allowed values: ``yes``, ``no``

 email_registration_confirmation -- ``yes``
  Offer registration confirmation by email or only through the web?
  Allowed values: ``yes``, ``no``

 indexer_stopwords -- default *blank*
  Additional stop-words for the full-text indexer specific to
  your tracker. See the indexer source for the default list of
  stop-words (e.g. ``A,AND,ARE,AS,AT,BE,BUT,BY, ...``).

Section **tracker**
 name -- ``Roundup issue tracker``
  A descriptive name for your roundup instance.

 web -- ``http://host.example/demo/``
  The web address that the tracker is viewable at.
  This will be included in information sent to users of the tracker.
  The URL MUST include the cgi-bin part or anything else
  that is required to get to the home page of the tracker.
  You MUST include a trailing '/' in the URL.

 email -- ``issue_tracker``
  Email address that mail to roundup should go to.

Section **web**
 http_auth -- ``yes``
  Whether to use HTTP Basic Authentication, if present.
  Roundup will use either the REMOTE_USER or HTTP_AUTHORIZATION
  variables supplied by your web server (in that order).
  Set this option to 'no' if you do not wish to use HTTP Basic
  Authentication in your web interface.

 use_browser_language -- ``yes``
  Whether to use HTTP Accept-Language, if present.
  Browsers send a language-region preference list.
  It's usually set in the client's browser or in their
  Operating System.
  Set this option to 'no' if you want to ignore it.

 debug -- ``no``
  Setting this option makes Roundup display error tracebacks
  in the user's browser rather than emailing them to the
  tracker admin."),

Section **rdbms**
 Settings in this section are used by Postgresql and MySQL backends only

 name -- ``roundup``
  Name of the database to use.

 host -- ``localhost``
  Database server host.

 port -- default *blank*
  TCP port number of the database server. Postgresql usually resides on
  port 5432 (if any), for MySQL default port number is 3306. Leave this
  option empty to use backend default.

 user -- ``roundup``
  Database user name that Roundup should use.

 password -- ``roundup``
  Database user password.

Section **logging**
 config -- default *blank*
  Path to configuration file for standard Python logging module. If this
  option is set, logging configuration is loaded from specified file;
  options 'filename' and 'level' in this section are ignored. The path may
  be either absolute or relative to the directory containig this config file.

 filename -- default *blank*
  Log file name for minimal logging facility built into Roundup.  If no file
  name specified, log messages are written on stderr. If above 'config'
  option is set, this option has no effect. The path may be either absolute
  or relative to the directory containig this config file.

 level -- ``ERROR``
  Minimal severity level of messages written to log file. If above 'config'
  option is set, this option has no effect.
  Allowed values: ``DEBUG``, ``INFO``, ``WARNING``, ``ERROR``

Section **mail**
 Outgoing email options. Used for nosy messages, password reset and
 registration approval requests.

 domain -- ``localhost``
  Domain name used for email addresses.

 host -- default *blank*
  SMTP mail host that roundup will use to send mail

 username -- default *blank*
  SMTP login name. Set this if your mail host requires authenticated access.
  If username is not empty, password (below) MUST be set!

 password -- default *blank*
  SMTP login password.
  Set this if your mail host requires authenticated access.

 tls -- ``no``
  If your SMTP mail host provides or requires TLS (Transport Layer Security)
  then you may set this option to 'yes'.
  Allowed values: ``yes``, ``no``

 tls_keyfile -- default *blank*
  If TLS is used, you may set this option to the name of a PEM formatted
  file that contains your private key. The path may be either absolute or
  relative to the directory containig this config file.

 tls_certfile -- default *blank*
  If TLS is used, you may set this option to the name of a PEM formatted
  certificate chain file. The path may be either absolute or relative
  to the directory containig this config file.

 charset -- utf-8
  Character set to encode email headers with. We use utf-8 by default, as
  it's the most flexible. Some mail readers (eg. Eudora) can't cope with
  that, so you might need to specify a more limited character set
  (eg. iso-8859-1).

 debug -- default *blank*
  Setting this option makes Roundup to write all outgoing email messages
  to this file *instead* of sending them. This option has the same effect
  as environment variable SENDMAILDEBUG. Environment variable takes
  precedence. The path may be either absolute or relative to the directory
  containig this config file.

Section **mailgw**
 Roundup Mail Gateway options

 keep_quoted_text -- ``yes``
  Keep email citations when accepting messages. Setting this to ``no`` strips
  out "quoted" text from the message. Signatures are also stripped.
  Allowed values: ``yes``, ``no``

 leave_body_unchanged -- ``no``
  Preserve the email body as is - that is, keep the citations *and*
  signatures.
  Allowed values: ``yes``, ``no``

 default_class -- ``issue``
  Default class to use in the mailgw if one isn't supplied in email subjects.
  To disable, leave the value blank.

 subject_prefix_parsing -- ``strict``
  Controls the parsing of the [prefix] on subject lines in incoming emails.
  ``strict`` will return an error to the sender if the [prefix] is not
  recognised. ``loose`` will attempt to parse the [prefix] but just
  pass it through as part of the issue title if not recognised. ``none``
  will always pass any [prefix] through as part of the issue title.

 subject_suffix_parsing -- ``strict``
  Controls the parsing of the [suffix] on subject lines in incoming emails.
  ``strict`` will return an error to the sender if the [suffix] is not
  recognised. ``loose`` will attempt to parse the [suffix] but just
  pass it through as part of the issue title if not recognised. ``none``
  will always pass any [suffix] through as part of the issue title.

 subject_suffix_delimiters -- ``[]``
  Defines the brackets used for delimiting the commands suffix in a subject
  line.

 subject_content_match -- ``always``
  Controls matching of the incoming email subject line against issue titles
  in the case where there is no designator [prefix]. ``never`` turns off
  matching. ``creation + interval`` or ``activity + interval`` will match
  an issue for the interval after the issue's creation or last activity.
  The interval is a standard Roundup interval.

Section **nosy**
 Nosy messages sending

 messages_to_author -- ``no``
  Send nosy messages to the author of the message.
  Allowed values: ``yes``, ``no``, ``new``

 signature_position -- ``bottom``
  Where to place the email signature.
  Allowed values: ``top``, ``bottom``, ``none``

 add_author -- ``new``
  Does the author of a message get placed on the nosy list automatically?
  If ``new`` is used, then the author will only be added when a message
  creates a new issue. If ``yes``, then the author will be added on
  followups too. If ``no``, they're never added to the nosy.
  Allowed values: ``yes``, ``no``, ``new``
  
 add_recipients -- ``new``
  Do the recipients (``To:``, ``Cc:``) of a message get placed on the nosy
  list?  If ``new`` is used, then the recipients will only be added when a
  message creates a new issue. If ``yes``, then the recipients will be added
  on followups too. If ``no``, they're never added to the nosy.
  Allowed values: ``yes``, ``no``, ``new``

 email_sending -- ``single``
  Controls the email sending from the nosy reactor. If ``multiple`` then
  a separate email is sent to each recipient. If ``single`` then a single
  email is sent with each recipient as a CC address.

You may generate a new default config file using the ``roundup-admin
genconfig`` command.

Configuration variables may be referred to in lower or upper case. In code,
variables not in the "main" section are referred to using their section and
name, so "domain" in the section "mail" becomes MAIL_DOMAIN. The
configuration variables available are:


Tracker Schema
==============

.. note::
   if you modify the schema, you'll most likely need to edit the
   `web interface`_ HTML template files and `detectors`_ to reflect
   your changes.

A tracker schema defines what data is stored in the tracker's database.
Schemas are defined using Python code in the ``schema.py`` module of your
tracker.

The ``schema.py`` module
------------------------

The ``schema.py`` module contains two functions:

**open**
  This function defines what your tracker looks like on the inside, the
  **schema** of the tracker. It defines the **Classes** and **properties**
  on each class. It also defines the **security** for those Classes. The
  next few sections describe how schemas work and what you can do with
  them.
**init**
  This function is responsible for setting up the initial state of your
  tracker. It's called exactly once - but the ``roundup-admin initialise``
  command.  See the start of the section on `database content`_ for more
  info about how this works.


The "classic" schema
--------------------

The "classic" schema looks like this (see section `setkey(property)`_
below for the meaning of ``'setkey'`` -- you may also want to look into
the sections `setlabelprop(property)`_ and `setorderprop(property)`_ for
specifying (default) labelling and ordering of classes.)::

    pri = Class(db, "priority", name=String(), order=String())
    pri.setkey("name")

    stat = Class(db, "status", name=String(), order=String())
    stat.setkey("name")

    keyword = Class(db, "keyword", name=String())
    keyword.setkey("name")

    user = Class(db, "user", username=String(), organisation=String(),
        password=String(), address=String(), realname=String(),
        phone=String())
    user.setkey("username")

    msg = FileClass(db, "msg", author=Link("user"), summary=String(),
        date=Date(), recipients=Multilink("user"),
        files=Multilink("file"))

    file = FileClass(db, "file", name=String(), type=String())

    issue = IssueClass(db, "issue", topic=Multilink("keyword"),
        status=Link("status"), assignedto=Link("user"),
        priority=Link("priority"))
    issue.setkey('title')


What you can't do to the schema
-------------------------------

You must never:

**Remove the users class**
  This class is the only *required* class in Roundup.

**Remove the "username", "address", "password" or "realname" user properties**
  Various parts of Roundup require these properties. Don't remove them.

**Change the type of a property**
  Property types must *never* be changed - the database simply doesn't take
  this kind of action into account. Note that you can't just remove a
  property and re-add it as a new type either. If you wanted to make the
  assignedto property a Multilink, you'd need to create a new property
  assignedto_list and remove the old assignedto property.


What you can do to the schema
-----------------------------

Your schema may be changed at any time before or after the tracker has been
initialised (or used). You may:

**Add new properties to classes, or add whole new classes**
  This is painless and easy to do - there are generally no repurcussions
  from adding new information to a tracker's schema.

**Remove properties**
  Removing properties is a little more tricky - you need to make sure that
  the property is no longer used in the `web interface`_ *or* by the
  detectors_.



Classes and Properties - creating a new information store
---------------------------------------------------------

In the tracker above, we've defined 7 classes of information:

  priority
      Defines the possible levels of urgency for issues.

  status
      Defines the possible states of processing the issue may be in.

  keyword
      Initially empty, will hold keywords useful for searching issues.

  user
      Initially holding the "admin" user, will eventually have an entry
      for all users using roundup.

  msg
      Initially empty, will hold all e-mail messages sent to or
      generated by roundup.

  file
      Initially empty, will hold all files attached to issues.

  issue
      Initially empty, this is where the issue information is stored.

We define the "priority" and "status" classes to allow two things:
reduction in the amount of information stored on the issue and more
powerful, accurate searching of issues by priority and status. By only
requiring a link on the issue (which is stored as a single number) we
reduce the chance that someone mis-types a priority or status - or
simply makes a new one up.


Class and Items
~~~~~~~~~~~~~~~

A Class defines a particular class (or type) of data that will be stored
in the database. A class comprises one or more properties, which gives
the information about the class items.

The actual data entered into the database, using ``class.create()``, are
called items. They have a special immutable property called ``'id'``. We
sometimes refer to this as the *itemid*.


Properties
~~~~~~~~~~

A Class is comprised of one or more properties of the following types:

* String properties are for storing arbitrary-length strings.
* Password properties are for storing encoded arbitrary-length strings.
  The default encoding is defined on the ``roundup.password.Password``
  class.
* Date properties store date-and-time stamps. Their values are Timestamp
  objects.
* Number properties store numeric values.
* Boolean properties store on/off, yes/no, true/false values.
* A Link property refers to a single other item selected from a
  specified class. The class is part of the property; the value is an
  integer, the id of the chosen item.
* A Multilink property refers to possibly many items in a specified
  class. The value is a list of integers.

All Classes automatically have a number of properties by default:

*creator*
  Link to the user that created the item.
*creation*
  Date the item was created.
*actor*
  Link to the user that last modified the item.
*activity*
  Date the item was last modified.


FileClass
~~~~~~~~~

FileClasses save their "content" attribute off in a separate file from
the rest of the database. This reduces the number of large entries in
the database, which generally makes databases more efficient, and also
allows us to use command-line tools to operate on the files. They are
stored in the files sub-directory of the ``'db'`` directory in your
tracker.


IssueClass
~~~~~~~~~~

IssueClasses automatically include the "messages", "files", "nosy", and
"superseder" properties.

The messages and files properties list the links to the messages and
files related to the issue. The nosy property is a list of links to
users who wish to be informed of changes to the issue - they get "CC'ed"
e-mails when messages are sent to or generated by the issue. The nosy
reactor (in the ``'detectors'`` directory) handles this action. The
superseder link indicates an issue which has superseded this one.

They also have the dynamically generated "creation", "activity" and
"creator" properties.

The value of the "creation" property is the date when an item was
created, and the value of the "activity" property is the date when any
property on the item was last edited (equivalently, these are the dates
on the first and last records in the item's journal). The "creator"
property holds a link to the user that created the issue.


setkey(property)
~~~~~~~~~~~~~~~~

Select a String property of the class to be the key property. The key
property must be unique, and allows references to the items in the class
by the content of the key property. That is, we can refer to users by
their username: for example, let's say that there's an issue in roundup,
issue 23. There's also a user, richard, who happens to be user 2. To
assign an issue to him, we could do either of::

     roundup-admin set issue23 assignedto=2

or::

     roundup-admin set issue23 assignedto=richard

Note, the same thing can be done in the web and e-mail interfaces. 

setlabelprop(property)
~~~~~~~~~~~~~~~~~~~~~~

Select a property of the class to be the label property. The label
property is used whereever an item should be uniquely identified, e.g.,
when displaying a link to an item. If setlabelprop is not specified for
a class, the following values are tried for the label: 

 * the key of the class (see the `setkey(property)`_ section above)
 * the "name" property
 * the "title" property
 * the first property from the sorted property name list

So in most cases you can get away without specifying setlabelprop
explicitly.

setorderprop(property)
~~~~~~~~~~~~~~~~~~~~~~

Select a property of the class to be the order property. The order
property is used whenever using a default sort order for the class,
e.g., when grouping or sorting class A by a link to class B in the user
interface, the order property of class B is used for sorting.  If
setorderprop is not specified for a class, the following values are tried
for the order property:

 * the property named "order"
 * the label property (see `setlabelprop(property)`_ above)

So in most cases you can get away without specifying setorderprop
explicitly.

create(information)
~~~~~~~~~~~~~~~~~~~

Create an item in the database. This is generally used to create items
in the "definitional" classes like "priority" and "status".


A note about ordering
~~~~~~~~~~~~~~~~~~~~~

When we sort items in the hyperdb, we use one of a number of methods,
depending on the properties being sorted on:

1. If it's a String, Number, Date or Interval property, we just sort the
   scalar value of the property. Strings are sorted case-sensitively.
2. If it's a Link property, we sort by either the linked item's "order"
   property (if it has one) or the linked item's "id".
3. Mulitlinks sort similar to #2, but we start with the first Multilink
   list item, and if they're the same, we sort by the second item, and
   so on.

Note that if an "order" property is defined on a Class that is used for
sorting, all items of that Class *must* have a value against the "order"
property, or sorting will result in random ordering.


Examples of adding to your schema
---------------------------------

TODO


Detectors - adding behaviour to your tracker
============================================
.. _detectors:

Detectors are initialised every time you open your tracker database, so
you're free to add and remove them any time, even after the database is
initialised via the ``roundup-admin initialise`` command.

The detectors in your tracker fire *before* (**auditors**) and *after*
(**reactors**) changes to the contents of your database. They are Python
modules that sit in your tracker's ``detectors`` directory. You will
have some installed by default - have a look. You can write new
detectors or modify the existing ones. The existing detectors installed
for you are:

**nosyreaction.py**
  This provides the automatic nosy list maintenance and email sending.
  The nosy reactor (``nosyreaction``) fires when new messages are added
  to issues. The nosy auditor (``updatenosy``) fires when issues are
  changed, and figures out what changes need to be made to the nosy list
  (such as adding new authors, etc.)
**statusauditor.py**
  This provides the ``chatty`` auditor which changes the issue status
  from ``unread`` or ``closed`` to ``chatting`` if new messages appear.
  It also provides the ``presetunread`` auditor which pre-sets the
  status to ``unread`` on new items if the status isn't explicitly
  defined.
**messagesummary.py**
  Generates the ``summary`` property for new messages based on the message
  content.
**userauditor.py**
  Verifies the content of some of the user fields (email addresses and
  roles lists).

If you don't want this default behaviour, you're completely free to change
or remove these detectors.

See the detectors section in the `design document`__ for details of the
interface for detectors.

__ design.html


Detector API
------------

Auditors are called with the arguments::

    audit(db, cl, itemid, newdata)

where ``db`` is the database, ``cl`` is an instance of Class or
IssueClass within the database, and ``newdata`` is a dictionary mapping
property names to values.

For a ``create()`` operation, the ``itemid`` argument is None and
newdata contains all of the initial property values with which the item
is about to be created.

For a ``set()`` operation, newdata contains only the names and values of
properties that are about to be changed.

For a ``retire()`` or ``restore()`` operation, newdata is None.

Reactors are called with the arguments::

    react(db, cl, itemid, olddata)

where ``db`` is the database, ``cl`` is an instance of Class or
IssueClass within the database, and ``olddata`` is a dictionary mapping
property names to values.

For a ``create()`` operation, the ``itemid`` argument is the id of the
newly-created item and ``olddata`` is None.

For a ``set()`` operation, ``olddata`` contains the names and previous
values of properties that were changed.

For a ``retire()`` or ``restore()`` operation, ``itemid`` is the id of
the retired or restored item and ``olddata`` is None.


Additional Detectors Ready For Use
----------------------------------

Sample additional detectors that have been found useful will appear in
the ``'detectors'`` directory of the Roundup distribution. If you want
to use one, copy it to the ``'detectors'`` of your tracker instance:

**newissuecopy.py**
  This detector sends an email to a team address whenever a new issue is
  created. The address is hard-coded into the detector, so edit it
  before you use it (look for the text 'team@team.host') or you'll get
  email errors!
**creator_resolution.py**
  Catch attempts to set the status to "resolved" - if the assignedto
  user isn't the creator, then set the status to "confirm-done". Note that
  "classic" Roundup doesn't have that status, so you'll have to add it. If
  you don't want to though, it'll just use "in-progress" instead.
**email_auditor.py**
  If a file added to an issue is of type message/rfc822, we tack on the
  extension .eml.
  The reason for this is that Microsoft Internet Explorer will not open
  things with a .eml attachment, as they deem it 'unsafe'. Worse yet,
  they'll just give you an incomprehensible error message. For more 
  information, see the detector code - it has a length explanation.


Auditor or Reactor?
-------------------

Generally speaking, the following rules should be observed:

**Auditors**
  Are used for `vetoing creation of or changes to items`_. They might
  also make automatic changes to item properties.
**Reactors**
  Detect changes in the database and react accordingly. They should avoid
  making changes to the database where possible, as this could create
  detector loops.


Vetoing creation of or changes to items
---------------------------------------

Auditors may raise the ``Reject`` exception to prevent the creation of
or changes to items in the database.  The mail gateway, for example, will
not attach files or messages to issues when the creation of those files or
messages are prevented through the ``Reject`` exception. It'll also not create
users if that creation is ``Reject``'ed too.

To use, simply add at the top of your auditor::

   from roundup.exceptions import Reject

And then when your rejection criteria have been detected, simply::

   raise Reject


Generating email from Roundup
-----------------------------

The module ``roundup.mailer`` contains most of the nuts-n-bolts required
to generate email messages from Roundup.

In addition, the ``IssueClass`` methods ``nosymessage()`` and
``send_message()`` are used to generate nosy messages, and may generate
messages which only consist of a change note (ie. the message id parameter
is not required - this is referred to as a "System Message" because it
comes from "the system" and not a user).


Database Content
================

.. note::
   if you modify the content of definitional classes, you'll most
   likely need to edit the tracker `detectors`_ to reflect your changes.

Customisation of the special "definitional" classes (eg. status,
priority, resolution, ...) may be done either before or after the
tracker is initialised. The actual method of doing so is completely
different in each case though, so be careful to use the right one.

**Changing content before tracker initialisation**
    Edit the initial_data.py module in your tracker to alter the items
    created using the ``create( ... )`` methods.

**Changing content after tracker initialisation**
    As the "admin" user, click on the "class list" link in the web
    interface to bring up a list of all database classes. Click on the
    name of the class you wish to change the content of.

    You may also use the ``roundup-admin`` interface's create, set and
    retire methods to add, alter or remove items from the classes in
    question.

See "`adding a new field to the classic schema`_" for an example that
requires database content changes.


Security / Access Controls
==========================

A set of Permissions is built into the security module by default:

- Create (everything)
- Edit (everything)
- View (everything)

These are assigned to the "Admin" Role by default, and allow a user to do
anything. Every Class you define in your `tracker schema`_ also gets an
Create, Edit and View Permission of its own. The web and email interfaces
also define:

*Email Access*
  If defined, the user may use the email interface. Used by default to deny
  Anonymous users access to the email interface. When granted to the
  Anonymous user, they will be automatically registered by the email
  interface (see also the ``new_email_user_roles`` configuration option).
*Web Access*
  If defined, the user may use the web interface. All users are able to see
  the login form, regardless of this setting (thus enabling logging in).
*Web Roles*
  Controls user access to editing the "roles" property of the "user" class.
  TODO: deprecate in favour of a property-based control.

These are hooked into the default Roles:

- Admin (Create, Edit, View and everything; Web Roles)
- User (Web Access; Email Access)
- Anonymous (Web Access)

And finally, the "admin" user gets the "Admin" Role, and the "anonymous"
user gets "Anonymous" assigned when the tracker is installed.

For the "User" Role, the "classic" tracker defines:

- Create, Edit and View issue, file, msg, query, keyword 
- View priority, status
- View user
- Edit their own user record

And the "Anonymous" Role is defined as:

- Web interface access
- Create user (for registration)
- View issue, file, msg, query, keyword, priority, status

Put together, these settings appear in the tracker's ``schema.py`` file::

    #
    # TRACKER SECURITY SETTINGS
    #
    # See the configuration and customisation document for information
    # about security setup.

    #
    # REGULAR USERS
    #
    # Give the regular users access to the web and email interface
    db.security.addPermissionToRole('User', 'Web Access')
    db.security.addPermissionToRole('User', 'Email Access')

    # Assign the access and edit Permissions for issue, file and message
    # to regular users now
    for cl in 'issue', 'file', 'msg', 'query', 'keyword':
        db.security.addPermissionToRole('User', 'View', cl)
        db.security.addPermissionToRole('User', 'Edit', cl)
        db.security.addPermissionToRole('User', 'Create', cl)
    for cl in 'priority', 'status':
        db.security.addPermissionToRole('User', 'View', cl)

    # May users view other user information? Comment these lines out
    # if you don't want them to
    db.security.addPermissionToRole('User', 'View', 'user')

    # Users should be able to edit their own details -- this permission
    # is limited to only the situation where the Viewed or Edited item
    # is their own.
    def own_record(db, userid, itemid):
        '''Determine whether the userid matches the item being accessed.'''
        return userid == itemid
    p = db.security.addPermission(name='View', klass='user', check=own_record,
        description="User is allowed to view their own user details")
    db.security.addPermissionToRole('User', p)
    p = db.security.addPermission(name='Edit', klass='user', check=own_record,
        description="User is allowed to edit their own user details")
    db.security.addPermissionToRole('User', p)

    #
    # ANONYMOUS USER PERMISSIONS
    #
    # Let anonymous users access the web interface. Note that almost all
    # trackers will need this Permission. The only situation where it's not
    # required is in a tracker that uses an HTTP Basic Authenticated front-end.
    db.security.addPermissionToRole('Anonymous', 'Web Access')

    # Let anonymous users access the email interface (note that this implies
    # that they will be registered automatically, hence they will need the
    # "Create" user Permission below)
    # This is disabled by default to stop spam from auto-registering users on
    # public trackers.
    #db.security.addPermissionToRole('Anonymous', 'Email Access')

    # Assign the appropriate permissions to the anonymous user's Anonymous
    # Role. Choices here are:
    # - Allow anonymous users to register
    db.security.addPermissionToRole('Anonymous', 'Create', 'user')

    # Allow anonymous users access to view issues (and the related, linked
    # information)
    for cl in 'issue', 'file', 'msg', 'keyword', 'priority', 'status':
        db.security.addPermissionToRole('Anonymous', 'View', cl)

    # [OPTIONAL]
    # Allow anonymous users access to create or edit "issue" items (and the
    # related file and message items)
    #for cl in 'issue', 'file', 'msg':
    #   db.security.addPermissionToRole('Anonymous', 'Create', cl)
    #   db.security.addPermissionToRole('Anonymous', 'Edit', cl)


Automatic Permission Checks
---------------------------

Permissions are automatically checked when information is rendered
through the web. This includes:

1. View checks for properties when being rendered via the ``plain()`` or
   similar methods. If the check fails, the text "[hidden]" will be
   displayed.
2. Edit checks for properties when the edit field is being rendered via
   the ``field()`` or similar methods. If the check fails, the property
   will be rendered via the ``plain()`` method (see point 1. for subsequent
   checking performed)
3. View checks are performed in index pages for each item being displayed
   such that if the user does not have permission, the row is not rendered.
4. View checks are performed at the top of item pages for the Item being
   displayed. If the user does not have permission, the text "You are not
   allowed to view this page." will be displayed.
5. View checks are performed at the top of index pages for the Class being
   displayed. If the user does not have permission, the text "You are not
   allowed to view this page." will be displayed.


New User Roles
--------------

New users are assigned the Roles defined in the config file as:

- NEW_WEB_USER_ROLES
- NEW_EMAIL_USER_ROLES

The `users may only edit their issues`_ example shows customisation of
these parameters.


Changing Access Controls
------------------------

You may alter the configuration variables to change the Role that new
web or email users get, for example to not give them access to the web
interface if they register through email. 

You may use the ``roundup-admin`` "``security``" command to display the
current Role and Permission configuration in your tracker.


Adding a new Permission
~~~~~~~~~~~~~~~~~~~~~~~

When adding a new Permission, you will need to:

1. add it to your tracker's ``schema.py`` so it is created, using
   ``security.addPermission``, for example::

    self.security.addPermission(name="View", klass='frozzle',
        description="User is allowed to access frozzles")

   will set up a new "View" permission on the Class "frozzle".
2. enable it for the Roles that should have it (verify with
   "``roundup-admin security``")
3. add it to the relevant HTML interface templates
4. add it to the appropriate xxxPermission methods on in your tracker
   interfaces module

The ``addPermission`` method takes a couple of optional parameters:

**properties**
  A sequence of property names that are the only properties to apply the
  new Permission to (eg. ``... klass='user', properties=('name',
  'email') ...``)
**check**
  A function to be execute which returns boolean determining whether the
  Permission is allowed. The function has the signature ``check(db, userid,
  itemid)`` where ``db`` is a handle on the open database, ``userid`` is
  the user attempting access and ``itemid`` is the specific item being
  accessed.

Example Scenarios
~~~~~~~~~~~~~~~~~

See the `examples`_ section for longer examples of customisation.

**anonymous access through the e-mail gateway**
 Give the "anonymous" user the "Email Access", ("Edit", "issue") and
 ("Create", "msg") Permissions but do not not give them the ("Create",
 "user") Permission. This means that when an unknown user sends email
 into the tracker, they're automatically logged in as "anonymous".
 Since they don't have the ("Create", "user") Permission, they won't
 be automatically registered, but since "anonymous" has permission to
 use the gateway, they'll still be able to submit issues. Note that
 the Sender information - their email address - will not be available
 - they're *anonymous*.

**automatic registration of users in the e-mail gateway**
 By giving the "anonymous" user the ("Create", "user" Permission, any
 unidentified user will automatically be registered with the tracker
 (with no password, so they won't be able to log in through
 the web until an admin sets their password). This is the default
 behaviour in the tracker templates that ship with Roundup. The new user
 is given the Roles list defined in the "new_email_user_roles" config
 variable.

**only developers may be assigned issues**
 Create a new Permission called "Fixer" for the "issue" class. Create a
 new Role "Developer" which has that Permission, and assign that to the
 appropriate users. Filter the list of users available in the assignedto
 list to include only those users. Enforce the Permission with an
 auditor. See the example 
 `restricting the list of users that are assignable to a task`_.

**only managers may sign off issues as complete**
 Create a new Permission called "Closer" for the "issue" class. Create a
 new Role "Manager" which has that Permission, and assign that to the
 appropriate users. In your web interface, only display the "resolved"
 issue state option when the user has the "Closer" Permissions. Enforce
 the Permission with an auditor. This is very similar to the previous
 example, except that the web interface check would look like::

   <option tal:condition="python:request.user.hasPermission('Closer')"
           value="resolved">Resolved</option>
 
**don't give web access to users who register through email**
 Create a new Role called "Email User" which has all the Permissions of
 the normal "User" Role minus the "Web Access" Permission. This will
 allow users to send in emails to the tracker, but not access the web
 interface.

**let some users edit the details of all users**
 Create a new Role called "User Admin" which has the Permission for
 editing users::

    db.security.addRole(name='User Admin', description='Managing users')
    p = db.security.getPermission('Edit', 'user')
    db.security.addPermissionToRole('User Admin', p)

 and assign the Role to the users who need the permission.


Web Interface
=============

.. contents::
   :local:

The web interface is provided by the ``roundup.cgi.client`` module and
is used by ``roundup.cgi``, ``roundup-server`` and ``ZRoundup``
(``ZRoundup``  is broken, until further notice). In all cases, we
determine which tracker is being accessed (the first part of the URL
path inside the scope of the CGI handler) and pass control on to the
``roundup.cgi.client.Client`` class - which handles the rest of the
access through its ``main()`` method. This means that you can do pretty
much anything you want as a web interface to your tracker.



Repercussions of changing the tracker schema
---------------------------------------------

If you choose to change the `tracker schema`_ you will need to ensure
the web interface knows about it:

1. Index, item and search pages for the relevant classes may need to
   have properties added or removed,
2. The "page" template may require links to be changed, as might the
   "home" page's content arguments.


How requests are processed
--------------------------

The basic processing of a web request proceeds as follows:

1. figure out who we are, defaulting to the "anonymous" user
2. figure out what the request is for - we call this the "context"
3. handle any requested action (item edit, search, ...)
4. render the template requested by the context, resulting in HTML
   output

In some situations, exceptions occur:

- HTTP Redirect  (generally raised by an action)
- SendFile       (generally raised by ``determine_context``)
    here we serve up a FileClass "content" property
- SendStaticFile (generally raised by ``determine_context``)
    here we serve up a file from the tracker "html" directory
- Unauthorised   (generally raised by an action)
    here the action is cancelled, the request is rendered and an error
    message is displayed indicating that permission was not granted for
    the action to take place
- NotFound       (raised wherever it needs to be)
    this exception percolates up to the CGI interface that called the
    client


Determining web context
-----------------------

To determine the "context" of a request, we look at the URL and the
special request variable ``@template``. The URL path after the tracker
identifier is examined. Typical URL paths look like:

1.  ``/tracker/issue``
2.  ``/tracker/issue1``
3.  ``/tracker/@@file/style.css``
4.  ``/cgi-bin/roundup.cgi/tracker/file1``
5.  ``/cgi-bin/roundup.cgi/tracker/file1/kitten.png``

where the "tracker identifier" is "tracker" in the above cases. That means
we're looking at "issue", "issue1", "@@file/style.css", "file1" and
"file1/kitten.png" in the cases above. The path is generally only one
entry long - longer paths are handled differently.

a. if there is no path, then we are in the "home" context. See `the "home"
   context`_ below for more information about how it may be used.
b. if the path starts with "@@file" (as in example 3,
   "/tracker/@@file/style.css"), then the additional path entry,
   "style.css" specifies the filename of a static file we're to serve up
   from the tracker TEMPLATES (or STATIC_FILES, if configured) directory.
   This is usually the tracker's "html" directory. Raises a SendStaticFile
   exception.
c. if there is something in the path (as in example 1, "issue"), it
   identifies the tracker class we're to display.
d. if the path is an item designator (as in examples 2 and 4, "issue1"
   and "file1"), then we're to display a specific item.
e. if the path starts with an item designator and is longer than one
   entry (as in example 5, "file1/kitten.png"), then we're assumed to be
   handling an item of a ``FileClass``, and the extra path information
   gives the filename that the client is going to label the download
   with (i.e. "file1/kitten.png" is nicer to download than "file1").
   This raises a ``SendFile`` exception.

Both b. and e. stop before we bother to determine the template we're
going to use. That's because they don't actually use templates.

The template used is specified by the ``@template`` CGI variable, which
defaults to:

- only classname suplied:        "index"
- full item designator supplied: "item"


The "home" Context
------------------

The "home" context is special because it allows you to add templated
pages to your tracker that don't rely on a class or item (ie. an issues
list or specific issue).

Let's say you wish to add frames to control the layout of your tracker's
interface. You'd probably have:

- A top-level frameset page. This page probably wouldn't be templated, so
  it could be served as a static file (see `serving static content`_)
- A sidebar frame that is templated. Let's call this page
  "home.navigation.html" in your tracker's "html" directory. To load that
  page up, you use the URL:

    <tracker url>/home?@template=navigation


Serving static content
----------------------

See the previous section `determining web context`_ where it describes
``@@file`` paths.


Performing actions in web requests
----------------------------------

When a user requests a web page, they may optionally also request for an
action to take place. As described in `how requests are processed`_, the
action is performed before the requested page is generated. Actions are
triggered by using a ``@action`` CGI variable, where the value is one
of:

**login**
 Attempt to log a user in.

**logout**
 Log the user out - make them "anonymous".

**register**
 Attempt to create a new user based on the contents of the form and then
 log them in.

**edit**
 Perform an edit of an item in the database. There are some `special form
 variables`_ you may use.

**new**
 Add a new item to the database. You may use the same `special form
 variables`_ as in the "edit" action.

**retire**
 Retire the item in the database.

**editCSV**
 Performs an edit of all of a class' items in one go. See also the
 *class*.csv templating method which generates the CSV data to be
 edited, and the ``'_generic.index'`` template which uses both of these
 features.

**search**
 Mangle some of the form variables:

 - Set the form ":filter" variable based on the values of the filter
   variables - if they're set to anything other than "dontcare" then add
   them to :filter.

 - Also handle the ":queryname" variable and save off the query to the
   user's query list.

Each of the actions is implemented by a corresponding ``*XxxAction*`` (where
"Xxx" is the name of the action) class in the ``roundup.cgi.actions`` module.
These classes are registered with ``roundup.cgi.client.Client``. If you need
to define new actions, you may add them there (see `defining new
web actions`_).

Each action class also has a ``*permission*`` method which determines whether
the action is permissible given the current user. The base permission checks
for each action are:

**login**
 Determine whether the user has the "Web Access" Permission.
**logout**
 No permission checks are made.
**register**
 Determine whether the user has the ("Create", "user") Permission.
**edit**
 Determine whether the user has permission to edit this item. If we're
 editing the "user" class, users are allowed to edit their own details -
 unless they try to edit the "roles" property, which requires the
 special Permission "Web Roles".
**new**
 Determine whether the user has permission to create this item. No
 additional property checks are made. Additionally, new user items may
 be created if the user has the ("Create", "user") Permission.
**editCSV**
 Determine whether the user has permission to edit this class.
**search**
 Determine whether the user has permission to view this class.


Special form variables
----------------------

Item properties and their values are edited with html FORM
variables and their values. You can:

- Change the value of some property of the current item.
- Create a new item of any class, and edit the new item's
  properties,
- Attach newly created items to a multilink property of the
  current item.
- Remove items from a multilink property of the current item.
- Specify that some properties are required for the edit
  operation to be successful.
- Set up user interface locale.

These operations will only take place if the form action (the
``@action`` variable) is "edit" or "new".

In the following, <bracketed> values are variable, "@" may be
either ":" or "@", and other text "required" is fixed.

Two special form variables are used to specify user language preferences:

``@language``
  value may be locale name or ``none``. If this variable is set to
  locale name, web interface language is changed to given value
  (provided that appropriate translation is available), the value
  is stored in the browser cookie and will be used for all following
  requests.  If value is ``none`` the cookie is removed and the
  language is changed to the tracker default, set up in the tracker
  configuration or OS environment.

``@charset``
  value may be character set name or ``none``.  Character set name
  is stored in the browser cookie and sets output encoding for all
  HTML pages generated by Roundup.  If value is ``none`` the cookie
  is removed and HTML output is reset to Roundup internal encoding
  (UTF-8).

Most properties are specified as form variables:

``<propname>``
  property on the current context item

``<designator>"@"<propname>``
  property on the indicated item (for editing related information)

Designators name a specific item of a class.

``<classname><N>``
    Name an existing item of class <classname>.

``<classname>"-"<N>``
    Name the <N>th new item of class <classname>. If the form
    submission is successful, a new item of <classname> is
    created. Within the submitted form, a particular
    designator of this form always refers to the same new
    item.

Once we have determined the "propname", we look at it to see
if it's special:

``@required``
    The associated form value is a comma-separated list of
    property names that must be specified when the form is
    submitted for the edit operation to succeed.  

    When the <designator> is missing, the properties are
    for the current context item.  When <designator> is
    present, they are for the item specified by
    <designator>.

    The "@required" specifier must come before any of the
    properties it refers to are assigned in the form.

``@remove@<propname>=id(s)`` or ``@add@<propname>=id(s)``
    The "@add@" and "@remove@" edit actions apply only to
    Multilink properties.  The form value must be a
    comma-separate list of keys for the class specified by
    the simple form variable.  The listed items are added
    to (respectively, removed from) the specified
    property.

``@link@<propname>=<designator>``
    If the edit action is "@link@", the simple form
    variable must specify a Link or Multilink property.
    The form value is a comma-separated list of
    designators.  The item corresponding to each
    designator is linked to the property given by simple
    form variable.

None of the above (ie. just a simple form value)
    The value of the form variable is converted
    appropriately, depending on the type of the property.

    For a Link('klass') property, the form value is a
    single key for 'klass', where the key field is
    specified in schema.py.  

    For a Multilink('klass') property, the form value is a
    comma-separated list of keys for 'klass', where the
    key field is specified in schema.py.  

    Note that for simple-form-variables specifiying Link
    and Multilink properties, the linked-to class must
    have a key field.

    For a String() property specifying a filename, the
    file named by the form value is uploaded. This means we
    try to set additional properties "filename" and "type" (if
    they are valid for the class).  Otherwise, the property
    is set to the form value.

    For Date(), Interval(), Boolean(), and Number()
    properties, the form value is converted to the
    appropriate

Any of the form variables may be prefixed with a classname or
designator.

Two special form values are supported for backwards compatibility:

@note
    This is equivalent to::

        @link@messages=msg-1
        msg-1@content=value

    except that in addition, the "author" and "date" properties of
    "msg-1" are set to the userid of the submitter, and the current
    time, respectively.

@file
    This is equivalent to::

        @link@files=file-1
        file-1@content=value

    The String content value is handled as described above for file
    uploads.

If both the "@note" and "@file" form variables are
specified, the action::

        @link@msg-1@files=file-1

is also performed.

We also check that FileClass items have a "content" property with
actual content, otherwise we remove them from all_props before
returning.


Default templates
-----------------

The default templates are html4 compliant. If you wish to change them to be
xhtml compliant, you'll need to change the ``html_version`` configuration
variable in ``config.ini`` to ``'xhtml'`` instead of ``'html4'``.

Most customisation of the web view can be done by modifying the
templates in the tracker ``'html'`` directory. There are several types
of files in there. The *minimal* template includes:

**page.html**
  This template usually defines the overall look of your tracker. When
  you view an issue, it appears inside this template. When you view an
  index, it also appears inside this template. This template defines a
  macro called "icing" which is used by almost all other templates as a
  coating for their content, using its "content" slot. It also defines
  the "head_title" and "body_title" slots to allow setting of the page
  title.
**home.html**
  the default page displayed when no other page is indicated by the user
**home.classlist.html**
  a special version of the default page that lists the classes in the
  tracker
**classname.item.html**
  displays an item of the *classname* class
**classname.index.html**
  displays a list of *classname* items
**classname.search.html**
  displays a search page for *classname* items
**_generic.index.html**
  used to display a list of items where there is no
  ``*classname*.index`` available
**_generic.help.html**
  used to display a "class help" page where there is no
  ``*classname*.help``
**user.register.html**
  a special page just for the user class, that renders the registration
  page
**style.css.html**
  a static file that is served up as-is

The *classic* template has a number of additional templates.

Remember that you can create any template extension you want to,
so if you just want to play around with the templating for new issues,
you can copy the current "issue.item" template to "issue.test", and then
access the test template using the "@template" URL argument::

   http://your.tracker.example/tracker/issue?@template=test

and it won't affect your users using the "issue.item" template.


How the templates work
----------------------


Basic Templating Actions
~~~~~~~~~~~~~~~~~~~~~~~~

Roundup's templates consist of special attributes on the HTML tags.
These attributes form the `Template Attribute Language`_, or TAL.
The basic TAL commands are:

**tal:define="variable expression; variable expression; ..."**
   Define a new variable that is local to this tag and its contents. For
   example::

      <html tal:define="title request/description">
       <head><title tal:content="title"></title></head>
      </html>

   In this example, the variable "title" is defined as the result of the
   expression "request/description". The "tal:content" command inside the
   <html> tag may then use the "title" variable.

**tal:condition="expression"**
   Only keep this tag and its contents if the expression is true. For
   example::

     <p tal:condition="python:request.user.hasPermission('View', 'issue')">
      Display some issue information.
     </p>

   In the example, the <p> tag and its contents are only displayed if
   the user has the "View" permission for issues. We consider the number
   zero, a blank string, an empty list, and the built-in variable
   nothing to be false values. Nearly every other value is true,
   including non-zero numbers, and strings with anything in them (even
   spaces!).

**tal:repeat="variable expression"**
   Repeat this tag and its contents for each element of the sequence
   that the expression returns, defining a new local variable and a
   special "repeat" variable for each element. For example::

     <tr tal:repeat="u user/list">
      <td tal:content="u/id"></td>
      <td tal:content="u/username"></td>
      <td tal:content="u/realname"></td>
     </tr>

   The example would iterate over the sequence of users returned by
   "user/list" and define the local variable "u" for each entry. Using
   the repeat command creates a new variable called "repeat" which you
   may access to gather information about the iteration. See the section
   below on `the repeat variable`_.

**tal:replace="expression"**
   Replace this tag with the result of the expression. For example::

    <span tal:replace="request/user/realname" />

   The example would replace the <span> tag and its contents with the
   user's realname. If the user's realname was "Bruce", then the
   resultant output would be "Bruce".

**tal:content="expression"**
   Replace the contents of this tag with the result of the expression.
   For example::

    <span tal:content="request/user/realname">user's name appears here
    </span>

   The example would replace the contents of the <span> tag with the
   user's realname. If the user's realname was "Bruce" then the
   resultant output would be "<span>Bruce</span>".

**tal:attributes="attribute expression; attribute expression; ..."**
   Set attributes on this tag to the results of expressions. For
   example::

     <a tal:attributes="href string:user${request/user/id}">My Details</a>

   In the example, the "href" attribute of the <a> tag is set to the
   value of the "string:user${request/user/id}" expression, which will
   be something like "user123".

**tal:omit-tag="expression"**
   Remove this tag (but not its contents) if the expression is true. For
   example::

      <span tal:omit-tag="python:1">Hello, world!</span>

   would result in output of::

      Hello, world!

Note that the commands on a given tag are evaulated in the order above,
so *define* comes before *condition*, and so on.

Additionally, you may include tags such as <tal:block>, which are
removed from output. Its content is kept, but the tag itself is not (so
don't go using any "tal:attributes" commands on it). This is useful for
making arbitrary blocks of HTML conditional or repeatable (very handy
for repeating multiple table rows, which would othewise require an
illegal tag placement to effect the repeat).

.. _TAL:
.. _Template Attribute Language:
   http://dev.zope.org/Wikis/DevSite/Projects/ZPT/TAL%20Specification%201.4


Templating Expressions
~~~~~~~~~~~~~~~~~~~~~~

Templating Expressions are covered by `Template Attribute Language
Expression Syntax`_, or TALES. The expressions you may use in the
attribute values may be one of the following forms:

**Path Expressions** - eg. ``item/status/checklist``
   These are object attribute / item accesses. Roughly speaking, the
   path ``item/status/checklist`` is broken into parts ``item``,
   ``status`` and ``checklist``. The ``item`` part is the root of the
   expression. We then look for a ``status`` attribute on ``item``, or
   failing that, a ``status`` item (as in ``item['status']``). If that
   fails, the path expression fails. When we get to the end, the object
   we're left with is evaluated to get a string - if it is a method, it
   is called; if it is an object, it is stringified. Path expressions
   may have an optional ``path:`` prefix, but they are the default
   expression type, so it's not necessary.

   If an expression evaluates to ``default``, then the expression is
   "cancelled" - whatever HTML already exists in the template will
   remain (tag content in the case of ``tal:content``, attributes in the
   case of ``tal:attributes``).

   If an expression evaluates to ``nothing`` then the target of the
   expression is removed (tag content in the case of ``tal:content``,
   attributes in the case of ``tal:attributes`` and the tag itself in
   the case of ``tal:replace``).

   If an element in the path may not exist, then you can use the ``|``
   operator in the expression to provide an alternative. So, the
   expression ``request/form/foo/value | default`` would simply leave
   the current HTML in place if the "foo" form variable doesn't exist.

   You may use the python function ``path``, as in
   ``path("item/status")``, to embed path expressions in Python
   expressions.

**String Expressions** - eg. ``string:hello ${user/name}`` 
   These expressions are simple string interpolations - though they can
   be just plain strings with no interpolation if you want. The
   expression in the ``${ ... }`` is just a path expression as above.

**Python Expressions** - eg. ``python: 1+1`` 
   These expressions give the full power of Python. All the "root level"
   variables are available, so ``python:item.status.checklist()`` would
   be equivalent to ``item/status/checklist``, assuming that
   ``checklist`` is a method.

Modifiers:

**structure** - eg. ``structure python:msg.content.plain(hyperlink=1)``
   The result of expressions are normally *escaped* to be safe for HTML
   display (all "<", ">" and "&" are turned into special entities). The
   ``structure`` expression modifier turns off this escaping - the
   result of the expression is now assumed to be HTML, which is passed
   to the web browser for rendering.

**not:** - eg. ``not:python:1=1``
   This simply inverts the logical true/false value of another
   expression.

.. _TALES:
.. _Template Attribute Language Expression Syntax:
   http://dev.zope.org/Wikis/DevSite/Projects/ZPT/TALES%20Specification%201.3


Template Macros
~~~~~~~~~~~~~~~

Macros are used in Roundup to save us from repeating the same common
page stuctures over and over. The most common (and probably only) macro
you'll use is the "icing" macro defined in the "page" template.

Macros are generated and used inside your templates using special
attributes similar to the `basic templating actions`_. In this case,
though, the attributes belong to the `Macro Expansion Template
Attribute Language`_, or METAL. The macro commands are:

**metal:define-macro="macro name"**
  Define that the tag and its contents are now a macro that may be
  inserted into other templates using the *use-macro* command. For
  example::

    <html metal:define-macro="page">
     ...
    </html>

  defines a macro called "page" using the ``<html>`` tag and its
  contents. Once defined, macros are stored on the template they're
  defined on in the ``macros`` attribute. You can access them later on
  through the ``templates`` variable, eg. the most common
  ``templates/page/macros/icing`` to access the "page" macro of the
  "page" template.

**metal:use-macro="path expression"**
  Use a macro, which is identified by the path expression (see above).
  This will replace the current tag with the identified macro contents.
  For example::

   <tal:block metal:use-macro="templates/page/macros/icing">
    ...
   </tal:block>

   will replace the tag and its contents with the "page" macro of the
   "page" template.

**metal:define-slot="slot name"** and **metal:fill-slot="slot name"**
  To define *dynamic* parts of the macro, you define "slots" which may
  be filled when the macro is used with a *use-macro* command. For
  example, the ``templates/page/macros/icing`` macro defines a slot like
  so::

    <title metal:define-slot="head_title">title goes here</title>

  In your *use-macro* command, you may now use a *fill-slot* command
  like this::

    <title metal:fill-slot="head_title">My Title</title>

  where the tag that fills the slot completely replaces the one defined
  as the slot in the macro.

Note that you may not mix `METAL`_ and `TAL`_ commands on the same tag, but
TAL commands may be used freely inside METAL-using tags (so your
*fill-slots* tags may have all manner of TAL inside them).

.. _METAL:
.. _Macro Expansion Template Attribute Language:
   http://dev.zope.org/Wikis/DevSite/Projects/ZPT/METAL%20Specification%201.0

Information available to templates
----------------------------------

This is implemented by ``roundup.cgi.templating.RoundupPageTemplate``

The following variables are available to templates.

**context**
  The current context. This is either None, a `hyperdb class wrapper`_
  or a `hyperdb item wrapper`_
**request**
  Includes information about the current request, including:
   - the current index information (``filterspec``, ``filter`` args,
     ``properties``, etc) parsed out of the form. 
   - methods for easy filterspec link generation
   - *user*, the current user item as an HTMLItem instance
   - *form*
     The current CGI form information as a mapping of form argument name
     to value
**config**
  This variable holds all the values defined in the tracker config.ini
  file (eg. TRACKER_NAME, etc.)
**db**
  The current database, used to access arbitrary database items.
**templates**
  Access to all the tracker templates by name. Used mainly in
  *use-macro* commands.
**utils**
  This variable makes available some utility functions like batching.
**nothing**
  This is a special variable - if an expression evaluates to this, then
  the tag (in the case of a ``tal:replace``), its contents (in the case
  of ``tal:content``) or some attributes (in the case of
  ``tal:attributes``) will not appear in the the output. So, for
  example::

    <span tal:attributes="class nothing">Hello, World!</span>

  would result in::

    <span>Hello, World!</span>

**default**
  Also a special variable - if an expression evaluates to this, then the
  existing HTML in the template will not be replaced or removed, it will
  remain. So::

    <span tal:replace="default">Hello, World!</span>

  would result in::

    <span>Hello, World!</span>

**true**, **false**
  Boolean constants that may be used in `templating expressions`_
  instead of ``python:1`` and ``python:0``.
**i18n**
  Internationalization service, providing two string translation methods:

  **gettext** (*message*)
    Return the localized translation of message
  **ngettext** (*singular*, *plural*, *number*)
    Like ``gettext()``, but consider plural forms. If a translation
    is found, apply the plural formula to *number*, and return the
    resulting message (some languages have more than two plural forms).
    If no translation is found, return singular if *number* is 1;
    return plural otherwise.

    This function requires python2.3; in earlier python versions
    may not work as expected.

The context variable
~~~~~~~~~~~~~~~~~~~~

The *context* variable is one of three things based on the current
context (see `determining web context`_ for how we figure this out):

1. if we're looking at a "home" page, then it's None
2. if we're looking at a specific hyperdb class, it's a
   `hyperdb class wrapper`_.
3. if we're looking at a specific hyperdb item, it's a
   `hyperdb item wrapper`_.

If the context is not None, we can access the properties of the class or
item. The only real difference between cases 2 and 3 above are:

1. the properties may have a real value behind them, and this will
   appear if the property is displayed through ``context/property`` or
   ``context/property/field``.
2. the context's "id" property will be a false value in the second case,
   but a real, or true value in the third. Thus we can determine whether
   we're looking at a real item from the hyperdb by testing
   "context/id".

Hyperdb class wrapper
:::::::::::::::::::::

This is implemented by the ``roundup.cgi.templating.HTMLClass``
class.

This wrapper object provides access to a hyperb class. It is used
primarily in both index view and new item views, but it's also usable
anywhere else that you wish to access information about a class, or the
items of a class, when you don't have a specific item of that class in
mind.

We allow access to properties. There will be no "id" property. The value
accessed through the property will be the current value of the same name
from the CGI form.

There are several methods available on these wrapper objects:

=========== =============================================================
Method      Description
=========== =============================================================
properties  return a `hyperdb property wrapper`_ for all of this class's
            properties.
list        lists all of the active (not retired) items in the class.
csv         return the items of this class as a chunk of CSV text.
propnames   lists the names of the properties of this class.
filter      lists of items from this class, filtered and sorted. Two
            options are avaible for sorting:

            1. by the current *request* filterspec/filter/sort/group args
            2. by the "filterspec", "sort" and "group" keyword args.
               "filterspec" is ``{propname: value(s)}``. "sort" and
               "group" are an optionally empty list ``[(dir, prop)]``
               where dir is '+', '-' or None
               and prop is a prop name or None.

               The propname in filterspec and prop in a sort/group spec
               may be transitive, i.e., it may contain properties of
               the form link.link.link.name.

            eg. All issues with a priority of "1" with messages added in
            the last week, sorted by activity date:
            ``issue.filter(filterspec={"priority": "1",
            'messages.creation' : '.-1w;'}, sort=[('activity', '+')])``

filter_sql  **Only in SQL backends**

            Lists the items that match the SQL provided. The SQL is a
            complete "select" statement.

            The SQL select must include the item id as the first column.

            This function **does not** filter out retired items, add
            on a where clause "__retired__ <> 1" if you don't want
            retired nodes.

classhelp   display a link to a javascript popup containing this class'
            "help" template.

            This generates a link to a popup window which displays the
            properties indicated by "properties" of the class named by
            "classname". The "properties" should be a comma-separated list
            (eg. 'id,name,description'). Properties defaults to all the
            properties of a class (excluding id, creator, created and
            activity).

            You may optionally override the "label" displayed, the "width",
            the "height", the number of items per page ("pagesize") and
            the field on which the list is sorted ("sort").

            With the "filter" arg it is possible to specify a filter for
            which items are supposed to be displayed. It has to be of
            the format "<field>=<values>;<field>=<values>;...".

            The popup window will be resizable and scrollable.

            If the "property" arg is given, it's passed through to the
            javascript help_window function. This allows updating of a
            property in the calling HTML page.

            If the "form" arg is given, it's passed through to the
            javascript help_window function - it's the name of the form
            the "property" belongs to.

submit      generate a submit button (and action hidden element)
renderWith  render this class with the given template.
history     returns 'New node - no history' :)
is_edit_ok  is the user allowed to Edit the current class?
is_view_ok  is the user allowed to View the current class?
=========== =============================================================

Note that if you have a property of the same name as one of the above
methods, you'll need to access it using a python "item access"
expression. For example::

   python:context['list']

will access the "list" property, rather than the list method.


Hyperdb item wrapper
::::::::::::::::::::

This is implemented by the ``roundup.cgi.templating.HTMLItem``
class.

This wrapper object provides access to a hyperb item.

We allow access to properties. There will be no "id" property. The value
accessed through the property will be the current value of the same name
from the CGI form.

There are several methods available on these wrapper objects:

=============== ========================================================
Method          Description
=============== ========================================================
submit          generate a submit button (and action hidden element)
journal         return the journal of the current item (**not
                implemented**)
history         render the journal of the current item as HTML
renderQueryForm specific to the "query" class - render the search form
                for the query
hasPermission   specific to the "user" class - determine whether the
                user has a Permission. The signature is::

                    hasPermission(self, permission, [classname=],
                        [property=], [itemid=])

                where the classname defaults to the current context.
hasRole         specific to the "user" class - determine whether the
                user has a Role. The signature is::

                    hasRole(self, rolename)

is_edit_ok      is the user allowed to Edit the current item?
is_view_ok      is the user allowed to View the current item?
is_retired      is the item retired?
download_url    generate a url-quoted link for download of FileClass
                item contents (ie. file<id>/<name>)
copy_url        generate a url-quoted link for creating a copy
                of this item.  By default, the copy will acquire
                all properties of the current item except for
                ``messages`` and ``files``.  This can be overridden
                by passing ``exclude`` argument which contains a list
                (or any iterable) of property names that shall not be
                copied.  Database-driven properties like ``id`` or
                ``activity`` cannot be copied.
=============== ========================================================

Note that if you have a property of the same name as one of the above
methods, you'll need to access it using a python "item access"
expression. For example::

   python:context['journal']

will access the "journal" property, rather than the journal method.


Hyperdb property wrapper
::::::::::::::::::::::::

This is implemented by subclasses of the
``roundup.cgi.templating.HTMLProperty`` class (``HTMLStringProperty``,
``HTMLNumberProperty``, and so on).

This wrapper object provides access to a single property of a class. Its
value may be either:

1. if accessed through a `hyperdb item wrapper`_, then it's a value from
   the hyperdb
2. if access through a `hyperdb class wrapper`_, then it's a value from
   the CGI form


The property wrapper has some useful attributes:

=============== ========================================================
Attribute       Description
=============== ========================================================
_name           the name of the property
_value          the value of the property if any - this is the actual
                value retrieved from the hyperdb for this property
=============== ========================================================

There are several methods available on these wrapper objects:

=========== ================================================================
Method      Description
=========== ================================================================
plain       render a "plain" representation of the property. This method
            may take two arguments:

            escape
             If true, escape the text so it is HTML safe (default: no). The
             reason this defaults to off is that text is usually escaped
             at a later stage by the TAL commands, unless the "structure"
             option is used in the template. The following ``tal:content``
             expressions are all equivalent::
 
              "structure python:msg.content.plain(escape=1)"
              "python:msg.content.plain()"
              "msg/content/plain"
              "msg/content"

             Usually you'll only want to use the escape option in a
             complex expression.

            hyperlink
             If true, turn URLs, email addresses and hyperdb item
             designators in the text into hyperlinks (default: no). Note
             that you'll need to use the "structure" TAL option if you
             want to use this ``tal:content`` expression::
  
              "structure python:msg.content.plain(hyperlink=1)"

             The text is automatically HTML-escaped before the hyperlinking
             transformation done in the plain() method.

hyperlinked The same as msg.content.plain(hyperlink=1), but nicer::

              "structure msg/content/hyperlinked"

field       render an appropriate form edit field for the property - for
            most types this is a text entry box, but for Booleans it's a
            tri-state yes/no/neither selection. This method may take some
            arguments:

            size
              Sets the width in characters of the edit field

            format (Date properties only)
              Sets the format of the date in the field - uses the same
              format string argument as supplied to the ``pretty`` method
              below.

            popcal (Date properties only)
              Include the Javascript-based popup calendar for date
              selection. Defaults to on.

stext       only on String properties - render the value of the property
            as StructuredText (requires the StructureText module to be
            installed separately)
multiline   only on String properties - render a multiline form edit
            field for the property
email       only on String properties - render the value of the property
            as an obscured email address
confirm     only on Password properties - render a second form edit field
            for the property, used for confirmation that the user typed
            the password correctly. Generates a field with name
            "name:confirm".
now         only on Date properties - return the current date as a new
            property
reldate     only on Date properties - render the interval between the date
            and now
local       only on Date properties - return this date as a new property
            with some timezone offset, for example::
            
                python:context.creation.local(10)

            will render the date with a +10 hour offset.
pretty      Date properties - render the date as "dd Mon YYYY" (eg. "19
            Mar 2004"). Takes an optional format argument, for example::

                python:context.activity.pretty('%Y-%m-%d')

            Will format as "2004-03-19" instead.

            Interval properties - render the interval in a pretty
            format (eg. "yesterday"). The format arguments are those used
            in the standard ``strftime`` call (see the `Python Library
            Reference: time module`__)
popcal      Generate a link to a popup calendar which may be used to
            edit the date field, for example::

              <span tal:replace="structure context/due/popcal" />

            you still need to include the ``field`` for the property, so
            typically you'd have::

              <span tal:replace="structure context/due/field" />
              <span tal:replace="structure context/due/popcal" />

menu        only on Link and Multilink properties - render a form select
            list for this property. Takes a number of optional arguments

            size
               is used to limit the length of the list labels
            height
               is used to set the <select> tag's "size" attribute
            showid
               includes the item ids in the list labels
            additional
               lists properties which should be included in the label
            sort_on
                indicates the property to sort the list on as (direction,
                (direction, property) where direction is '+' or '-'. A
                single string with the direction prepended may be used.
                For example: ('-', 'order'), '+name'.
            value
                gives a default value to preselect in the menu

            The remaining keyword arguments are used as conditions for
            filtering the items in the list - they're passed as the
            "filterspec" argument to a Class.filter() call. For example::

             <span tal:replace="structure context/status/menu" />

             <span tal:replace="python:context.status.menu(order='+name",
                                   value='chatting', 
                                   filterspec={'status': '1,2,3,4'}" />

sorted      only on Multilink properties - produce a list of the linked
            items sorted by some property, for example::
            
                python:context.files.sorted('creation')

            Will list the files by upload date.
reverse     only on Multilink properties - produce a list of the linked
            items in reverse order
isset       returns True if the property has been set to a value
=========== ================================================================

__ http://docs.python.org/lib/module-time.html

All of the above functions perform checks for permissions required to
display or edit the data they are manipulating. The simplest case is
editing an issue title. Including the expression::

   context/title/field

Will present the user with an edit field, if they have edit permission. If
not, then they will be presented with a static display if they have view
permission. If they don't even have view permission, then an error message
is raised, preventing the display of the page, indicating that they don't
have permission to view the information.


The request variable
~~~~~~~~~~~~~~~~~~~~

This is implemented by the ``roundup.cgi.templating.HTMLRequest``
class.

The request variable is packed with information about the current
request.

.. taken from ``roundup.cgi.templating.HTMLRequest`` docstring

=========== ============================================================
Variable    Holds
=========== ============================================================
form        the CGI form as a cgi.FieldStorage
env         the CGI environment variables
base        the base URL for this tracker
user        a HTMLUser instance for this user
classname   the current classname (possibly None)
template    the current template (suffix, also possibly None)
form        the current CGI form variables in a FieldStorage
=========== ============================================================

**Index page specific variables (indexing arguments)**

=========== ============================================================
Variable    Holds
=========== ============================================================
columns     dictionary of the columns to display in an index page
show        a convenience access to columns - request/show/colname will
            be true if the columns should be displayed, false otherwise
sort        index sort columns [(direction, column name)]
group       index grouping properties [(direction, column name)]
filter      properties to filter the index on
filterspec  values to filter the index on (property=value, eg
            ``priority=1`` or ``messages.author=42``
search_text text to perform a full-text search on for an index
=========== ============================================================

There are several methods available on the request variable:

=============== ========================================================
Method          Description
=============== ========================================================
description     render a description of the request - handle for the
                page title
indexargs_form  render the current index args as form elements
indexargs_url   render the current index args as a URL
base_javascript render some javascript that is used by other components
                of the templating
batch           run the current index args through a filter and return a
                list of items (see `hyperdb item wrapper`_, and
                `batching`_)
=============== ========================================================

The form variable
:::::::::::::::::

The form variable is a bit special because it's actually a python
FieldStorage object. That means that you have two ways to access its
contents. For example, to look up the CGI form value for the variable
"name", use the path expression::

   request/form/name/value

or the python expression::

   python:request.form['name'].value

Note the "item" access used in the python case, and also note the
explicit "value" attribute we have to access. That's because the form
variables are stored as MiniFieldStorages. If there's more than one
"name" value in the form, then the above will break since
``request/form/name`` is actually a *list* of MiniFieldStorages. So it's
best to know beforehand what you're dealing with.


The db variable
~~~~~~~~~~~~~~~

This is implemented by the ``roundup.cgi.templating.HTMLDatabase``
class.

Allows access to all hyperdb classes as attributes of this variable. If
you want access to the "user" class, for example, you would use::

  db/user
  python:db.user

Also, the current id of the current user is available as
``db.getuid()``. This isn't so useful in templates (where you have
``request/user``), but it can be useful in detectors or interfaces.

The access results in a `hyperdb class wrapper`_.


The templates variable
~~~~~~~~~~~~~~~~~~~~~~

This is implemented by the ``roundup.cgi.templating.Templates``
class.

This variable doesn't have any useful methods defined. It supports being
used in expressions to access the templates, and consequently the
template macros. You may access the templates using the following path
expression::

   templates/name

or the python expression::

   templates[name]

where "name" is the name of the template you wish to access. The
template has one useful attribute, namely "macros". To access a specific
macro (called "macro_name"), use the path expression::

   templates/name/macros/macro_name

or the python expression::

   templates[name].macros[macro_name]

The repeat variable
~~~~~~~~~~~~~~~~~~~

The repeat variable holds an entry for each active iteration. That is, if
you have a ``tal:repeat="user db/users"`` command, then there will be a
repeat variable entry called "user". This may be accessed as either::

    repeat/user
    python:repeat['user']

The "user" entry has a number of methods available for information:

=============== =========================================================
Method          Description
=============== =========================================================
first           True if the current item is the first in the sequence.
last            True if the current item is the last in the sequence.
even            True if the current item is an even item in the sequence.
odd             True if the current item is an odd item in the sequence.
number          Current position in the sequence, starting from 1.
letter          Current position in the sequence as a letter, a through
                z, then aa through zz, and so on.
Letter          Same as letter(), except uppercase.
roman           Current position in the sequence as lowercase roman
                numerals.
Roman           Same as roman(), except uppercase.
=============== =========================================================


The utils variable
~~~~~~~~~~~~~~~~~~

This is implemented by the
``roundup.cgi.templating.TemplatingUtils`` class, but it may be extended
as described below.

=============== ========================================================
Method          Description
=============== ========================================================
Batch           return a batch object using the supplied list
url_quote       quote some text as safe for a URL (ie. space, %, ...)
html_quote      quote some text as safe in HTML (ie. <, >, ...)
html_calendar   renders an HTML calendar used by the
                ``_generic.calendar.html`` template (itself invoked by
                the popupCalendar DateHTMLProperty method
=============== ========================================================

You may add additional utility methods by writing them in your tracker
``extensions`` directory and registering them with the templating system
using ``instance.registerUtil`` (see `adding a time log to your issues`_ for
an example of this).


Batching
::::::::

Use Batch to turn a list of items, or item ids of a given class, into a
series of batches. Its usage is::

    python:utils.Batch(sequence, size, start, end=0, orphan=0,
    overlap=0)

or, to get the current index batch::

    request/batch

The parameters are:

========= ==============================================================
Parameter  Usage
========= ==============================================================
sequence  a list of HTMLItems
size      how big to make the sequence.
start     where to start (0-indexed) in the sequence.
end       where to end (0-indexed) in the sequence.
orphan    if the next batch would contain less items than this value,
          then it is combined with this batch
overlap   the number of items shared between adjacent batches
========= ==============================================================

All of the parameters are assigned as attributes on the batch object. In
addition, it has several more attributes:

=============== ========================================================
Attribute       Description
=============== ========================================================
start           indicates the start index of the batch. *Unlike
                the argument, is a 1-based index (I know, lame)*
first           indicates the start index of the batch *as a 0-based
                index*
length          the actual number of elements in the batch
sequence_length the length of the original, unbatched, sequence.
=============== ========================================================

And several methods:

=============== ========================================================
Method          Description
=============== ========================================================
previous        returns a new Batch with the previous batch settings
next            returns a new Batch with the next batch settings
propchanged     detect if the named property changed on the current item
                when compared to the last item
=============== ========================================================

An example of batching::

 <table class="otherinfo">
  <tr><th colspan="4" class="header">Existing Keywords</th></tr>
  <tr tal:define="keywords db/keyword/list"
      tal:repeat="start python:range(0, len(keywords), 4)">
   <td tal:define="batch python:utils.Batch(keywords, 4, start)"
       tal:repeat="keyword batch" tal:content="keyword/name">
       keyword here</td>
  </tr>
 </table>

... which will produce a table with four columns containing the items of
the "keyword" class (well, their "name" anyway).


Displaying Properties
---------------------

Properties appear in the user interface in three contexts: in indices,
in editors, and as search arguments. For each type of property, there
are several display possibilities. For example, in an index view, a
string property may just be printed as a plain string, but in an editor
view, that property may be displayed in an editable field.


Index Views
-----------

This is one of the class context views. It is also the default view for
classes. The template used is "*classname*.index".


Index View Specifiers
~~~~~~~~~~~~~~~~~~~~~

An index view specifier (URL fragment) looks like this (whitespace has
been added for clarity)::

    /issue?status=unread,in-progress,resolved&
        topic=security,ui&
        @group=priority,-status&
        @sort=-activity&
        @filters=status,topic&
        @columns=title,status,fixer

The index view is determined by two parts of the specifier: the layout
part and the filter part. The layout part consists of the query
parameters that begin with colons, and it determines the way that the
properties of selected items are displayed. The filter part consists of
all the other query parameters, and it determines the criteria by which
items are selected for display. The filter part is interactively
manipulated with the form widgets displayed in the filter section. The
layout part is interactively manipulated by clicking on the column
headings in the table.

The filter part selects the union of the sets of items with values
matching any specified Link properties and the intersection of the sets
of items with values matching any specified Multilink properties.

The example specifies an index of "issue" items. Only items with a
"status" of either "unread" or "in-progress" or "resolved" are
displayed, and only items with "topic" values including both "security"
and "ui" are displayed. The items are grouped by priority arranged in
ascending order and in descending order by status; and within
groups, sorted by activity, arranged in descending order. The filter
section shows filters for the "status" and "topic" properties, and the
table includes columns for the "title", "status", and "fixer"
properties.

============ =============================================================
Argument     Description
============ =============================================================
@sort        sort by prop name, optionally preceeded with '-' to give
             descending or nothing for ascending sorting. Several
             properties can be specified delimited with comma.
             Internally a search-page using several sort properties may
             use @sort0, @sort1 etc. with option @sortdir0, @sortdir1
             etc. for the direction of sorting (a non-empty value of
             sortdir0 specifies reverse order).
@group       group by prop name, optionally preceeded with '-' or to sort
             in descending or nothing for ascending order. Several
             properties can be specified delimited with comma.
             Internally a search-page using several grouping properties may
             use @group0, @group1 etc. with option @groupdir0, @groupdir1
             etc. for the direction of grouping (a non-empty value of
             groupdir0 specifies reverse order).
@columns     selects the columns that should be displayed. Default is
             all.                     
@filter      indicates which properties are being used in filtering.
             Default is none.
propname     selects the values the item properties given by propname must
             have (very basic search/filter).
@search_text if supplied, performs a full-text search (message bodies,
             issue titles, etc)
============ =============================================================


Searching Views
---------------

.. note::
   if you add a new column to the ``@columns`` form variable potentials
   then you will need to add the column to the appropriate `index views`_
   template so that it is actually displayed.

This is one of the class context views. The template used is typically
"*classname*.search". The form on this page should have "search" as its
``@action`` variable. The "search" action:

- sets up additional filtering, as well as performing indexed text
  searching
- sets the ``@filter`` variable correctly
- saves the query off if ``@query_name`` is set.

The search page should lay out any fields that you wish to allow the
user to search on. If your schema contains a large number of properties,
you should be wary of making all of those properties available for
searching, as this can cause confusion. If the additional properties are
Strings, consider having their value indexed, and then they will be
searchable using the full text indexed search. This is both faster, and
more useful for the end user.

If the search view does specify the "search" ``@action``, then it may also
provide an additional argument:

============ =============================================================
Argument     Description
============ =============================================================
@query_name  if supplied, the index parameters (including @search_text)
             will be saved off as a the query item and registered against
             the user's queries property. Note that the *classic* template
             schema has this ability, but the *minimal* template schema
             does not.
============ =============================================================


Item Views
----------

The basic view of a hyperdb item is provided by the "*classname*.item"
template. It generally has three sections; an "editor", a "spool" and a
"history" section.


Editor Section
~~~~~~~~~~~~~~

The editor section is used to manipulate the item - it may be a static
display if the user doesn't have permission to edit the item.

Here's an example of a basic editor template (this is the default
"classic" template issue item edit form - from the "issue.item.html"
template)::

 <table class="form">
 <tr>
  <th>Title</th>
  <td colspan="3" tal:content="structure python:context.title.field(size=60)">title</td>
 </tr>
 
 <tr>
  <th>Priority</th>
  <td tal:content="structure context/priority/menu">priority</td>
  <th>Status</th>
  <td tal:content="structure context/status/menu">status</td>
 </tr>
 
 <tr>
  <th>Superseder</th>
  <td>
   <span tal:replace="structure python:context.superseder.field(showid=1, size=20)" />
   <span tal:replace="structure python:db.issue.classhelp('id,title')" />
   <span tal:condition="context/superseder">
    <br>View: <span tal:replace="structure python:context.superseder.link(showid=1)" />
   </span>
  </td>
  <th>Nosy List</th>
  <td>
   <span tal:replace="structure context/nosy/field" />
   <span tal:replace="structure python:db.user.classhelp('username,realname,address,phone')" />
  </td>
 </tr>
 
 <tr>
  <th>Assigned To</th>
  <td tal:content="structure context/assignedto/menu">
   assignedto menu
  </td>
  <td>&nbsp;</td>
  <td>&nbsp;</td>
 </tr>
 
 <tr>
  <th>Change Note</th>
  <td colspan="3">
   <textarea name=":note" wrap="hard" rows="5" cols="60"></textarea>
  </td>
 </tr>
 
 <tr>
  <th>File</th>
  <td colspan="3"><input type="file" name=":file" size="40"></td>
 </tr>
 
 <tr>
  <td>&nbsp;</td>
  <td colspan="3" tal:content="structure context/submit">
   submit button will go here
  </td>
 </tr>
 </table>


When a change is submitted, the system automatically generates a message
describing the changed properties. As shown in the example, the editor
template can use the ":note" and ":file" fields, which are added to the
standard changenote message generated by Roundup.


Form values
:::::::::::

We have a number of ways to pull properties out of the form in order to
meet the various needs of:

1. editing the current item (perhaps an issue item)
2. editing information related to the current item (eg. messages or
   attached files)
3. creating new information to be linked to the current item (eg. time
   spent on an issue)

In the following, ``<bracketed>`` values are variable, ":" may be one of
":" or "@", and other text ("required") is fixed.

Properties are specified as form variables:

``<propname>``
  property on the current context item

``<designator>:<propname>``
  property on the indicated item (for editing related information)

``<classname>-<N>:<propname>``
  property on the Nth new item of classname (generally for creating new
  items to attach to the current item)

Once we have determined the "propname", we check to see if it is one of
the special form values:

``@required``
  The named property values must be supplied or a ValueError will be
  raised.

``@remove@<propname>=id(s)``
  The ids will be removed from the multilink property.

``:add:<propname>=id(s)``
  The ids will be added to the multilink property.

``:link:<propname>=<designator>``
  Used to add a link to new items created during edit. These are
  collected and returned in ``all_links``. This will result in an
  additional linking operation (either Link set or Multilink append)
  after the edit/create is done using ``all_props`` in ``_editnodes``.
  The <propname> on the current item will be set/appended the id of the
  newly created item of class <designator> (where <designator> must be
  <classname>-<N>).

Any of the form variables may be prefixed with a classname or
designator.

Two special form values are supported for backwards compatibility:

``:note``
  create a message (with content, author and date), linked to the
  context item. This is ALWAYS designated "msg-1".
``:file``
  create a file, attached to the current item and any message created by
  :note. This is ALWAYS designated "file-1".


Spool Section
~~~~~~~~~~~~~

The spool section lists related information like the messages and files
of an issue.

TODO


History Section
~~~~~~~~~~~~~~~

The final section displayed is the history of the item - its database
journal. This is generally generated with the template::

 <tal:block tal:replace="structure context/history" />

*To be done:*

*The actual history entries of the item may be accessed for manual
templating through the "journal" method of the item*::

 <tal:block tal:repeat="entry context/journal">
  a journal entry
 </tal:block>

*where each journal entry is an HTMLJournalEntry.*


Defining new web actions
------------------------

You may define new actions to be triggered by the ``@action`` form variable.
These are added to the tracker ``extensions`` directory and registered
using ``instance.registerAction``.

All the existing Actions are defined in ``roundup.cgi.actions``.

Adding action classes takes three steps; first you `define the new
action class`_, then you `register the action class`_ with the cgi
interface so it may be triggered by the ``@action`` form variable.
Finally you `use the new action`_ in your HTML form.

See "`setting up a "wizard" (or "druid") for controlled adding of
issues`_" for an example.


Define the new action class
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Create a new action class in your tracker's ``extensions`` directory, for
example ``myaction.py``::

 from roundup.cgi.actions import Action

 class MyAction(Action):
     def handle(self):
         ''' Perform some action. No return value is required.
         '''

The *self.client* attribute is an instance of ``roundup.cgi.client.Client``.
See the docstring of that class for details of what it can do.

The method will typically check the ``self.form`` variable's contents.
It may then:

- add information to ``self.client.ok_message`` or ``self.client.error_message``
- change the ``self.client.template`` variable to alter what the user will see
  next
- raise Unauthorised, SendStaticFile, SendFile, NotFound or Redirect
  exceptions (import them from roundup.cgi.exceptions)


Register the action class
~~~~~~~~~~~~~~~~~~~~~~~~~~

The class is now written, but isn't available to the user until you register
it with the following code appended to your ``myaction.py`` file::

    def init(instance):
        instance.registerAction('myaction', myActionClass)

This maps the action name "myaction" to the action class we defined.


Use the new action
~~~~~~~~~~~~~~~~~~

In your HTML form, add a hidden form element like so::

  <input type="hidden" name="@action" value="myaction">

where "myaction" is the name you registered in the previous step.

Actions may return content to the user
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Actions generally perform some database manipulation and then pass control
on to the rendering of a template in the current context (see `Determining
web context`_ for how that works.) Some actions will want to generate the
actual content returned to the user. Action methods may return their own
content string to be displayed to the user, overriding the templating step.
In this situation, we assume that the content is HTML by default. You may
override the content type indicated to the user by calling ``setHeader``::

   self.client.setHeader('Content-Type', 'text/csv')

This example indicates that the value sent back to the user is actually
comma-separated value content (eg. something to be loaded into a
spreadsheet or database).


8-bit character set support in Web interface
--------------------------------------------

The web interface uses UTF-8 default. It may be overridden in both forms
and a browser cookie.

- In forms, use the ``@charset`` variable.
- To use the cookie override, have the ``roundup_charset`` cookie set.

In both cases, the value is a valid charset name (eg. ``utf-8`` or
``kio8-r``).

Inside Roundup, all strings are stored and processed in utf-8.
Unfortunately, some older browsers do not work properly with
utf-8-encoded pages (e.g. Netscape Navigator 4 displays wrong
characters in form fields).  This version allows one to change
the character set for http transfers.  To do so, you may add
the following code to your ``page.html`` template::

 <tal:block define="uri string:${request/base}${request/env/PATH_INFO}">
  <a tal:attributes="href python:request.indexargs_url(uri,
   {'@charset':'utf-8'})">utf-8</a>
  <a tal:attributes="href python:request.indexargs_url(uri,
   {'@charset':'koi8-r'})">koi8-r</a>
 </tal:block>

(substitute ``koi8-r`` with appropriate charset for your language).
Charset preference is kept in the browser cookie ``roundup_charset``.

``meta http-equiv`` lines added to the tracker templates in version 0.6.0
should be changed to include actual character set name::

 <meta http-equiv="Content-Type"
  tal:attributes="content string:text/html;; charset=${request/client/charset}"
 />

The charset is also sent in the http header.


Examples
========

.. contents::
   :local:
   :depth: 2


Changing what's stored in the database
--------------------------------------

The following examples illustrate ways to change the information stored in
the database.


Adding a new field to the classic schema
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

This example shows how to add a simple field (a due date) to the default
classic schema. It does not add any additional behaviour, such as enforcing
the due date, or causing automatic actions to fire if the due date passes.

You add new fields by editing the ``schema.py`` file in you tracker's home.
Schema changes are automatically applied to the database on the next
tracker access (note that roundup-server would need to be restarted as it
caches the schema).

1. modify the ``schema.py``::

    issue = IssueClass(db, "issue", 
                    assignedto=Link("user"), topic=Multilink("keyword"),
                    priority=Link("priority"), status=Link("status"),
                    due_date=Date())

2. add an edit field to the ``issue.item.html`` template::

    <tr> 
     <th>Due Date</th> 
     <td tal:content="structure context/due_date/field" /> 
    </tr> 

3. add the property to the ``issue.index.html`` page::

    (in the heading row)
      <th tal:condition="request/show/due_date">Due Date</th>
    (in the data row)
      <td tal:condition="request/show/due_date" tal:content="i/due_date" />

4. add the property to the ``issue.search.html`` page::

     <tr tal:define="name string:due_date">
       <th i18n:translate="">Due Date:</th>
       <td metal:use-macro="search_input"></td>
       <td metal:use-macro="column_input"></td>
       <td metal:use-macro="sort_input"></td>
       <td metal:use-macro="group_input"></td>
     </tr>

5. if you wish for the due date to appear in the standard views listed
    in the sidebar of the web interface then you'll need to add "due_date"
    to the list of @columns in the links in the sidebar section of
    ``page.html``.


Adding a new constrained field to the classic schema
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

This example shows how to add a new constrained property (i.e. a
selection of distinct values) to your tracker.


Introduction
::::::::::::

To make the classic schema of Roundup useful as a TODO tracking system
for a group of systems administrators, it needs an extra data field per
issue: a category.

This would let sysadmins quickly list all TODOs in their particular area
of interest without having to do complex queries, and without relying on
the spelling capabilities of other sysadmins (a losing proposition at
best).


Adding a field to the database
::::::::::::::::::::::::::::::

This is the easiest part of the change. The category would just be a
plain string, nothing fancy. To change what is in the database you need
to add some lines to the ``schema.py`` file of your tracker instance.
Under the comment::

    # add any additional database schema configuration here

add::

    category = Class(db, "category", name=String())
    category.setkey("name")

Here we are setting up a chunk of the database which we are calling
"category". It contains a string, which we are refering to as "name" for
lack of a more imaginative title. (Since "name" is one of the properties
that Roundup looks for on items if you do not set a key for them, it's
probably a good idea to stick with it for new classes if at all
appropriate.) Then we are setting the key of this chunk of the database
to be that "name". This is equivalent to an index for database types.
This also means that there can only be one category with a given name.

Adding the above lines allows us to create categories, but they're not
tied to the issues that we are going to be creating. It's just a list of
categories off on its own, which isn't much use. We need to link it in
with the issues. To do that, find the lines 
in ``schema.py`` which set up the "issue" class, and then add a link to
the category::

    issue = IssueClass(db, "issue", ... ,
        category=Multilink("category"), ... )

The ``Multilink()`` means that each issue can have many categories. If
you were adding something with a one-to-one relationship to issues (such
as the "assignedto" property), use ``Link()`` instead.

That is all you need to do to change the schema. The rest of the effort
is fiddling around so you can actually use the new category.


Populating the new category class
:::::::::::::::::::::::::::::::::

If you haven't initialised the database with the ``roundup-admin``
"initialise" command, then you can add the following to the tracker
``initial_data.py`` under the comment::

    # add any additional database creation steps here - but only if you
    # haven't initialised the database with the admin "initialise" command

Add::

     category = db.getclass('category')
     category.create(name="scipy")
     category.create(name="chaco")
     category.create(name="weave")

If the database has already been initalised, then you need to use the
``roundup-admin`` tool::

     % roundup-admin -i <tracker home>
     Roundup <version> ready for input.
     Type "help" for help.
     roundup> create category name=scipy
     1
     roundup> create category name=chaco
     2
     roundup> create category name=weave
     3
     roundup> exit...
     There are unsaved changes. Commit them (y/N)? y


Setting up security on the new objects
::::::::::::::::::::::::::::::::::::::

By default only the admin user can look at and change objects. This
doesn't suit us, as we want any user to be able to create new categories
as required, and obviously everyone needs to be able to view the
categories of issues for it to be useful.

We therefore need to change the security of the category objects. This
is also done in ``schema.py``.

There are currently two loops which set up permissions and then assign
them to various roles. Simply add the new "category" to both lists::

    # Assign the access and edit permissions for issue, file and message
    # to regular users now
    for cl in 'issue', 'file', 'msg', 'category':
        p = db.security.getPermission('View', cl)
        db.security.addPermissionToRole('User', 'View', cl)
        db.security.addPermissionToRole('User', 'Edit', cl)
        db.security.addPermissionToRole('User', 'Create', cl)

These lines assign the "View" and "Edit" Permissions to the "User" role,
so that normal users can view and edit "category" objects.

This is all the work that needs to be done for the database. It will
store categories, and let users view and edit them. Now on to the
interface stuff.


Changing the web left hand frame
::::::::::::::::::::::::::::::::

We need to give the users the ability to create new categories, and the
place to put the link to this functionality is in the left hand function
bar, under the "Issues" area. The file that defines how this area looks
is ``html/page.html``, which is what we are going to be editing next.

If you look at this file you can see that it contains a lot of
"classblock" sections which are chunks of HTML that will be included or
excluded in the output depending on whether the condition in the
classblock is met. We are going to add the category code at the end of
the classblock for the *issue* class::

  <p class="classblock"
     tal:condition="python:request.user.hasPermission('View', 'category')">
   <b>Categories</b><br>
   <a tal:condition="python:request.user.hasPermission('Edit', 'category')"
      href="category?@template=item">New Category<br></a>
  </p>

The first two lines is the classblock definition, which sets up a
condition that only users who have "View" permission for the "category"
object will have this section included in their output. Next comes a
plain "Categories" header in bold. Everyone who can view categories will
get that.

Next comes the link to the editing area of categories. This link will
only appear if the condition - that the user has "Edit" permissions for
the "category" objects - is matched. If they do have permission then
they will get a link to another page which will let the user add new
categories.

Note that if you have permission to *view* but not to *edit* categories,
then all you will see is a "Categories" header with nothing underneath
it. This is obviously not very good interface design, but will do for
now. I just claim that it is so I can add more links in this section
later on. However, to fix the problem you could change the condition in
the classblock statement, so that only users with "Edit" permission
would see the "Categories" stuff.


Setting up a page to edit categories
::::::::::::::::::::::::::::::::::::

We defined code in the previous section which let users with the
appropriate permissions see a link to a page which would let them edit
conditions. Now we have to write that page.

The link was for the *item* template of the *category* object. This
translates into Roundup looking for a file called ``category.item.html``
in the ``html`` tracker directory. This is the file that we are going to
write now.

First, we add an info tag in a comment which doesn't affect the outcome
of the code at all, but is useful for debugging. If you load a page in a
browser and look at the page source, you can see which sections come
from which files by looking for these comments::

    <!-- category.item -->

Next we need to add in the METAL macro stuff so we get the normal page
trappings::

 <tal:block metal:use-macro="templates/page/macros/icing">
  <title metal:fill-slot="head_title">Category editing</title>
  <td class="page-header-top" metal:fill-slot="body_title">
   <h2>Category editing</h2>
  </td>
  <td class="content" metal:fill-slot="content">

Next we need to setup up a standard HTML form, which is the whole
purpose of this file. We link to some handy javascript which sends the
form through only once. This is to stop users hitting the send button
multiple times when they are impatient and thus having the form sent
multiple times::

    <form method="POST" onSubmit="return submit_once()"
          enctype="multipart/form-data">

Next we define some code which sets up the minimum list of fields that
we require the user to enter. There will be only one field - "name" - so
they better put something in it, otherwise the whole form is pointless::

    <input type="hidden" name="@required" value="name">

To get everything to line up properly we will put everything in a table,
and put a nice big header on it so the user has an idea what is
happening::

    <table class="form">
     <tr><th class="header" colspan="2">Category</th></tr>

Next, we need the field into which the user is going to enter the new
category. The ``context.name.field(size=60)`` bit tells Roundup to
generate a normal HTML field of size 60, and the contents of that field
will be the "name" variable of the current context (namely "category").
The upshot of this is that when the user types something in
to the form, a new category will be created with that name::

    <tr>
     <th>Name</th>
     <td tal:content="structure python:context.name.field(size=60)">
     name</td>
    </tr>

Then a submit button so that the user can submit the new category::

    <tr>
     <td>&nbsp;</td>
     <td colspan="3" tal:content="structure context/submit">
      submit button will go here
     </td>
    </tr>

Finally we finish off the tags we used at the start to do the METAL
stuff::

  </td>
 </tal:block>

So putting it all together, and closing the table and form we get::

 <!-- category.item -->
 <tal:block metal:use-macro="templates/page/macros/icing">
  <title metal:fill-slot="head_title">Category editing</title>
  <td class="page-header-top" metal:fill-slot="body_title">
   <h2>Category editing</h2>
  </td>
  <td class="content" metal:fill-slot="content">
   <form method="POST" onSubmit="return submit_once()"
         enctype="multipart/form-data">

    <table class="form">
     <tr><th class="header" colspan="2">Category</th></tr>

     <tr>
      <th>Name</th>
      <td tal:content="structure python:context.name.field(size=60)">
      name</td>
     </tr>

     <tr>
      <td>
        &nbsp;
        <input type="hidden" name="@required" value="name"> 
      </td>
      <td colspan="3" tal:content="structure context/submit">
       submit button will go here
      </td>
     </tr>
    </table>
   </form>
  </td>
 </tal:block>

This is quite a lot to just ask the user one simple question, but there
is a lot of setup for basically one line (the form line) to do its work.
To add another field to "category" would involve one more line (well,
maybe a few extra to get the formatting correct).


Adding the category to the issue
::::::::::::::::::::::::::::::::

We now have the ability to create issues to our heart's content, but
that is pointless unless we can assign categories to issues.  Just like
the ``html/category.item.html`` file was used to define how to add a new
category, the ``html/issue.item.html`` is used to define how a new issue
is created.

Just like ``category.issue.html``, this file defines a form which has a
table to lay things out. It doesn't matter where in the table we add new
stuff, it is entirely up to your sense of aesthetics::

   <th>Category</th>
   <td>
    <span tal:replace="structure context/category/field" />
    <span tal:replace="structure python:db.category.classhelp('name',
                property='category', width='200')" />
   </td>

First, we define a nice header so that the user knows what the next
section is, then the middle line does what we are most interested in.
This ``context/category/field`` gets replaced by a field which contains
the category in the current context (the current context being the new
issue).

The classhelp lines generate a link (labelled "list") to a popup window
which contains the list of currently known categories.


Searching on categories
:::::::::::::::::::::::

Now we can add categories, and create issues with categories. The next
obvious thing that we would like to be able to do, would be to search
for issues based on their category, so that, for example, anyone working
on the web server could look at all issues in the category "Web".

If you look for "Search Issues" in the ``html/page.html`` file, you will
find that it looks something like 
``<a href="issue?@template=search">Search Issues</a>``. This shows us
that when you click on "Search Issues" it will be looking for a
``issue.search.html`` file to display. So that is the file that we will
change.

If you look at this file it should begin to seem familiar, although it
does use some new macros. You can add the new category search code anywhere you
like within that form::

  <tr tal:define="name string:category;
                  db_klass string:category;
                  db_content string:name;">
    <th>Priority:</th>
    <td metal:use-macro="search_select"></td>
    <td metal:use-macro="column_input"></td>
    <td metal:use-macro="sort_input"></td>
    <td metal:use-macro="group_input"></td>
  </tr>

The definitions in the ``<tr>`` opening tag are used by the macros:

- ``search_select`` expands to a drop-down box with all categories using
  ``db_klass`` and ``db_content``.
- ``column_input`` expands to a checkbox for selecting what columns
  should be displayed.
- ``sort_input`` expands to a radio button for selecting what property
  should be sorted on.
- ``group_input`` expands to a radio button for selecting what property
  should be grouped on.

The category search code above would expand to the following::

  <tr>
    <th>Category:</th>
    <td>
      <select name="category">
        <option value="">don't care</option>
        <option value="">------------</option>      
        <option value="1">scipy</option>
        <option value="2">chaco</option>
        <option value="3">weave</option>
      </select>
    </td>
    <td><input type="checkbox" name=":columns" value="category"></td>
    <td><input type="radio" name=":sort0" value="category"></td>
    <td><input type="radio" name=":group0" value="category"></td>
  </tr>

Adding category to the default view
:::::::::::::::::::::::::::::::::::

We can now add categories, add issues with categories, and search for
issues based on categories. This is everything that we need to do;
however, there is some more icing that we would like. I think the
category of an issue is important enough that it should be displayed by
default when listing all the issues.

Unfortunately, this is a bit less obvious than the previous steps. The
code defining how the issues look is in ``html/issue.index.html``. This
is a large table with a form down at the bottom for redisplaying and so
forth. 

Firstly we need to add an appropriate header to the start of the table::

    <th tal:condition="request/show/category">Category</th>

The *condition* part of this statement is to avoid displaying the
Category column if the user has selected not to see it.

The rest of the table is a loop which will go through every issue that
matches the display criteria. The loop variable is "i" - which means
that every issue gets assigned to "i" in turn.

The new part of code to display the category will look like this::

    <td tal:condition="request/show/category"
        tal:content="i/category"></td>

The condition is the same as above: only display the condition when the
user hasn't asked for it to be hidden. The next part is to set the
content of the cell to be the category part of "i" - the current issue.

Finally we have to edit ``html/page.html`` again. This time, we need to
tell it that when the user clicks on "Unassigned Issues" or "All Issues",
the category column should be included in the resulting list. If you
scroll down the page file, you can see the links with lots of options.
The option that we are interested in is the ``:columns=`` one which
tells roundup which fields of the issue to display. Simply add
"category" to that list and it all should work.

Adding a time log to your issues
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

We want to log the dates and amount of time spent working on issues, and
be able to give a summary of the total time spent on a particular issue.

1. Add a new class to your tracker ``schema.py``::

    # storage for time logging
    timelog = Class(db, "timelog", period=Interval())

   Note that we automatically get the date of the time log entry
   creation through the standard property "creation".

   You will need to grant "Creation" permission to the users who are
   allowed to add timelog entries. You may do this with::

    db.security.addPermissionToRole('User', 'Create', 'timelog')
    db.security.addPermissionToRole('User', 'View', 'timelog')

   If users are also able to *edit* timelog entries, then also include::

    db.security.addPermissionToRole('User', 'Edit', 'timelog')

2. Link to the new class from your issue class (again, in
   ``schema.py``)::

    issue = IssueClass(db, "issue", 
                    assignedto=Link("user"), topic=Multilink("keyword"),
                    priority=Link("priority"), status=Link("status"),
                    times=Multilink("timelog"))

   the "times" property is the new link to the "timelog" class.

3. We'll need to let people add in times to the issue, so in the web
   interface we'll have a new entry field. This is a special field
   because unlike the other fields in the ``issue.item`` template, it
   affects a different item (a timelog item) and not the template's
   item (an issue). We have a special syntax for form fields that affect
   items other than the template default item (see the cgi 
   documentation on `special form variables`_). In particular, we add a
   field to capture a new timelog item's period::

    <tr> 
     <th>Time Log</th> 
     <td colspan=3><input type="text" name="timelog-1@period" /> 
      <br />(enter as '3y 1m 4d 2:40:02' or parts thereof) 
     </td> 
    </tr> 
         
   and another hidden field that links that new timelog item (new
   because it's marked as having id "-1") to the issue item. It looks
   like this::

     <input type="hidden" name="@link@times" value="timelog-1" />

   On submission, the "-1" timelog item will be created and assigned a
   real item id. The "times" property of the issue will have the new id
   added to it.

4. We want to display a total of the timelog times that have been
   accumulated for an issue. To do this, we'll need to actually write
   some Python code, since it's beyond the scope of PageTemplates to
   perform such calculations. We do this by adding a module ``timespent.py``
   to the ``extensions`` directory in our tracker. The contents of this
   file is as follows::

    from roundup import date

    def totalTimeSpent(times):
        ''' Call me with a list of timelog items (which have an
            Interval "period" property)
        '''
        total = date.Interval('0d')
        for time in times:
            total += time.period._value
        return total

    def init(instance):
        instance.registerUtil('totalTimeSpent', totalTimeSpent)

   We will now be able to access the ``totalTimeSpent`` function via the
   ``utils`` variable in our templates, as shown in the next step.

5. Display the timelog for an issue::

     <table class="otherinfo" tal:condition="context/times">
      <tr><th colspan="3" class="header">Time Log
       <tal:block
            tal:replace="python:utils.totalTimeSpent(context.times)" />
      </th></tr>
      <tr><th>Date</th><th>Period</th><th>Logged By</th></tr>
      <tr tal:repeat="time context/times">
       <td tal:content="time/creation"></td>
       <td tal:content="time/period"></td>
       <td tal:content="time/creator"></td>
      </tr>
     </table>

   I put this just above the Messages log in my issue display. Note our
   use of the ``totalTimeSpent`` method which will total up the times
   for the issue and return a new Interval. That will be automatically
   displayed in the template as text like "+ 1y 2:40" (1 year, 2 hours
   and 40 minutes).

8. If you're using a persistent web server - ``roundup-server`` or
   ``mod_python`` for example - then you'll need to restart that to pick up
   the code changes. When that's done, you'll be able to use the new
   time logging interface.

An extension of this modification attaches the timelog entries to any
change message entered at the time of the timelog entry:

1. Add a link to the timelog to the msg class:

    msg = FileClass(db, "msg",
                    author=Link("user", do_journal='no'),
                    recipients=Multilink("user", do_journal='no'),
                    date=Date(),
                    summary=String(),
                    files=Multilink("file"),
                    messageid=String(),
                    inreplyto=String()
                    times=Multilink("timelog"))

2. Add a new hidden field that links that new timelog item (new
   because it's marked as having id "-1") to the new message.
   It looks like this::
 
      <input type="hidden" name="msg-1@link@times" value="timelog-1" />
 
   The "times" property of the message will have the new id added to it.

3. Add the timelog listing from step 5. to the ``msg.item.html`` template
   so that the timelog entry appears on the message view page.


Tracking different types of issues
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Sometimes you will want to track different types of issues - developer,
customer support, systems, sales leads, etc. A single Roundup tracker is
able to support multiple types of issues. This example demonstrates adding
a system support issue class to a tracker.

1. Figure out what information you're going to want to capture. OK, so
   this is obvious, but sometimes it's better to actually sit down for a
   while and think about the schema you're going to implement.

2. Add the new issue class to your tracker's ``schema.py``. Just after the
   "issue" class definition, add::

    # list our systems
    system = Class(db, "system", name=String(), order=Number())
    system.setkey("name")

    # store issues related to those systems
    support = IssueClass(db, "support", 
                    assignedto=Link("user"), topic=Multilink("keyword"),
                    status=Link("status"), deadline=Date(),
                    affects=Multilink("system"))

3. Copy the existing ``issue.*`` (item, search and index) templates in the
   tracker's ``html`` to ``support.*``. Edit them so they use the properties
   defined in the ``support`` class. Be sure to check for hidden form
   variables like "required" to make sure they have the correct set of
   required properties.

4. Edit the modules in the ``detectors``, adding lines to their ``init``
   functions where appropriate. Look for ``audit`` and ``react`` registrations
   on the ``issue`` class, and duplicate them for ``support``.

5. Create a new sidebar box for the new support class. Duplicate the
   existing issues one, changing the ``issue`` class name to ``support``.

6. Re-start your tracker and start using the new ``support`` class.


Optionally, you might want to restrict the users able to access this new
class to just the users with a new "SysAdmin" Role. To do this, we add
some security declarations::

    db.security.addPermissionToRole('SysAdmin', 'View', 'support')
    db.security.addPermissionToRole('SysAdmin', 'Create', 'support')
    db.security.addPermissionToRole('SysAdmin', 'Edit', 'support')

You would then (as an "admin" user) edit the details of the appropriate
users, and add "SysAdmin" to their Roles list.

Alternatively, you might want to change the Edit/View permissions granted
for the ``issue`` class so that it's only available to users with the "System"
or "Developer" Role, and then the new class you're adding is available to
all with the "User" Role.


Using External User Databases
-----------------------------

Using an external password validation source
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. note:: You will need to either have an "admin" user in your external
          password source *or* have one of your regular users have
          the Admin Role assigned. If you need to assign the Role *after*
          making the changes below, you may use the ``roundup-admin``
          program to edit a user's details.

We have a centrally-managed password changing system for our users. This
results in a UN*X passwd-style file that we use for verification of
users. Entries in the file consist of ``name:password`` where the
password is encrypted using the standard UN*X ``crypt()`` function (see
the ``crypt`` module in your Python distribution). An example entry
would be::

    admin:aamrgyQfDFSHw

Each user of Roundup must still have their information stored in the Roundup
database - we just use the passwd file to check their password. To do this, we
need to override the standard ``verifyPassword`` method defined in
``roundup.cgi.actions.LoginAction`` and register the new class. The
following is added as ``externalpassword.py`` in the tracker ``extensions``
directory::

    import os, crypt
    from roundup.cgi.actions import LoginAction    

    class ExternalPasswordLoginAction(LoginAction):
        def verifyPassword(self, userid, password):
            '''Look through the file, line by line, looking for a
            name that matches.
            '''
            # get the user's username
            username = self.db.user.get(userid, 'username')

            # the passwords are stored in the "passwd.txt" file in the
            # tracker home
            file = os.path.join(self.db.config.TRACKER_HOME, 'passwd.txt')

            # see if we can find a match
            for ent in [line.strip().split(':') for line in
                                                open(file).readlines()]:
                if ent[0] == username:
                    return crypt.crypt(password, ent[1][:2]) == ent[1]

            # user doesn't exist in the file
            return 0

    def init(instance):
        instance.registerAction('login', ExternalPasswordLoginAction)

You should also remove the redundant password fields from the ``user.item``
template.


Using a UN*X passwd file as the user database
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

On some systems the primary store of users is the UN*X passwd file. It
holds information on users such as their username, real name, password
and primary user group.

Roundup can use this store as its primary source of user information,
but it needs additional information too - email address(es), roundup
Roles, vacation flags, roundup hyperdb item ids, etc. Also, "retired"
users must still exist in the user database, unlike some passwd files in
which the users are removed when they no longer have access to a system.

To make use of the passwd file, we therefore synchronise between the two
user stores. We also use the passwd file to validate the user logins, as
described in the previous example, `using an external password
validation source`_. We keep the user lists in sync using a fairly
simple script that runs once a day, or several times an hour if more
immediate access is needed. In short, it:

1. parses the passwd file, finding usernames, passwords and real names,
2. compares that list to the current roundup user list:

   a. entries no longer in the passwd file are *retired*
   b. entries with mismatching real names are *updated*
   c. entries only exist in the passwd file are *created*

3. send an email to administrators to let them know what's been done.

The retiring and updating are simple operations, requiring only a call
to ``retire()`` or ``set()``. The creation operation requires more
information though - the user's email address and their Roundup Roles.
We're going to assume that the user's email address is the same as their
login name, so we just append the domain name to that. The Roles are
determined using the passwd group identifier - mapping their UN*X group
to an appropriate set of Roles.

The script to perform all this, broken up into its main components, is
as follows. Firstly, we import the necessary modules and open the
tracker we're to work on::

    import sys, os, smtplib
    from roundup import instance, date

    # open the tracker
    tracker_home = sys.argv[1]
    tracker = instance.open(tracker_home)

Next we read in the *passwd* file from the tracker home::

    # read in the users from the "passwd.txt" file
    file = os.path.join(tracker_home, 'passwd.txt')
    users = [x.strip().split(':') for x in open(file).readlines()]

Handle special users (those to ignore in the file, and those who don't
appear in the file)::

    # users to not keep ever, pre-load with the users I know aren't
    # "real" users
    ignore = ['ekmmon', 'bfast', 'csrmail']

    # users to keep - pre-load with the roundup-specific users
    keep = ['comment_pool', 'network_pool', 'admin', 'dev-team',
            'cs_pool', 'anonymous', 'system_pool', 'automated']

Now we map the UN*X group numbers to the Roles that users should have::

    roles = {
     '501': 'User,Tech',  # tech
     '502': 'User',       # finance
     '503': 'User,CSR',   # customer service reps
     '504': 'User',       # sales
     '505': 'User',       # marketing
    }

Now we do all the work. Note that the body of the script (where we have
the tracker database open) is wrapped in a ``try`` / ``finally`` clause,
so that we always close the database cleanly when we're finished. So, we
now do all the work::

    # open the database
    db = tracker.open('admin')
    try:
        # store away messages to send to the tracker admins
        msg = []

        # loop over the users list read in from the passwd file
        for user,passw,uid,gid,real,home,shell in users:
            if user in ignore:
                # this user shouldn't appear in our tracker
                continue
            keep.append(user)
            try:
                # see if the user exists in the tracker
                uid = db.user.lookup(user)

                # yes, they do - now check the real name for correctness
                if real != db.user.get(uid, 'realname'):
                    db.user.set(uid, realname=real)
                    msg.append('FIX %s - %s'%(user, real))
            except KeyError:
                # nope, the user doesn't exist
                db.user.create(username=user, realname=real,
                    address='%s@ekit-inc.com'%user, roles=roles[gid])
                msg.append('ADD %s - %s (%s)'%(user, real, roles[gid]))

        # now check that all the users in the tracker are also in our
        # "keep" list - retire those who aren't
        for uid in db.user.list():
            user = db.user.get(uid, 'username')
            if user not in keep:
                db.user.retire(uid)
                msg.append('RET %s'%user)

        # if we did work, then send email to the tracker admins
        if msg:
            # create the email
            msg = '''Subject: %s user database maintenance

            %s
            '''%(db.config.TRACKER_NAME, '\n'.join(msg))

            # send the email
            smtp = smtplib.SMTP(db.config.MAILHOST)
            addr = db.config.ADMIN_EMAIL
            smtp.sendmail(addr, addr, msg)

        # now we're done - commit the changes
        db.commit()
    finally:
        # always close the database cleanly
        db.close()

And that's it!


Using an LDAP database for user information
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

A script that reads users from an LDAP store using
http://python-ldap.sf.net/ and then compares the list to the users in the
roundup user database would be pretty easy to write. You'd then have it run
once an hour / day (or on demand if you can work that into your LDAP store
workflow). See the example `Using a UN*X passwd file as the user database`_
for more information about doing this.

To authenticate off the LDAP store (rather than using the passwords in the
Roundup user database) you'd use the same python-ldap module inside an
extension to the cgi interface. You'd do this by overriding the method called
``verifyPassword`` on the ``LoginAction`` class in your tracker's
``extensions`` directory (see `using an external password validation
source`_). The method is implemented by default as::

    def verifyPassword(self, userid, password):
        ''' Verify the password that the user has supplied
        '''
        stored = self.db.user.get(self.userid, 'password')
        if password == stored:
            return 1
        if not password and not stored:
            return 1
        return 0

So you could reimplement this as something like::

    def verifyPassword(self, userid, password):
        ''' Verify the password that the user has supplied
        '''
        # look up some unique LDAP information about the user
        username = self.db.user.get(self.userid, 'username')
        # now verify the password supplied against the LDAP store


Changes to Tracker Behaviour
----------------------------

Stop "nosy" messages going to people on vacation
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

When users go on vacation and set up vacation email bouncing, you'll
start to see a lot of messages come back through Roundup "Fred is on
vacation". Not very useful, and relatively easy to stop.

1. add a "vacation" flag to your users::

         user = Class(db, "user",
                    username=String(),   password=Password(),
                    address=String(),    realname=String(),
                    phone=String(),      organisation=String(),
                    alternate_addresses=String(),
                    roles=String(), queries=Multilink("query"),
                    vacation=Boolean())

2. So that users may edit the vacation flags, add something like the
   following to your ``user.item`` template::

     <tr>
      <th>On Vacation</th> 
      <td tal:content="structure context/vacation/field">vacation</td> 
     </tr> 

3. edit your detector ``nosyreactor.py`` so that the ``nosyreaction()``
   consists of::

    def nosyreaction(db, cl, nodeid, oldvalues):
        users = db.user
        messages = db.msg
        # send a copy of all new messages to the nosy list
        for msgid in determineNewMessages(cl, nodeid, oldvalues):
            try:
                # figure the recipient ids
                sendto = []
                seen_message = {}
                recipients = messages.get(msgid, 'recipients')
                for recipid in messages.get(msgid, 'recipients'):
                    seen_message[recipid] = 1

                # figure the author's id, and indicate they've received
                # the message
                authid = messages.get(msgid, 'author')

                # possibly send the message to the author, as long as
                # they aren't anonymous
                if (db.config.MESSAGES_TO_AUTHOR == 'yes' and
                        users.get(authid, 'username') != 'anonymous'):
                    sendto.append(authid)
                seen_message[authid] = 1

                # now figure the nosy people who weren't recipients
                nosy = cl.get(nodeid, 'nosy')
                for nosyid in nosy:
                    # Don't send nosy mail to the anonymous user (that
                    # user shouldn't appear in the nosy list, but just
                    # in case they do...)
                    if users.get(nosyid, 'username') == 'anonymous':
                        continue
                    # make sure they haven't seen the message already
                    if not seen_message.has_key(nosyid):
                        # send it to them
                        sendto.append(nosyid)
                        recipients.append(nosyid)

                # generate a change note
                if oldvalues:
                    note = cl.generateChangeNote(nodeid, oldvalues)
                else:
                    note = cl.generateCreateNote(nodeid)

                # we have new recipients
                if sendto:
                    # filter out the people on vacation
                    sendto = [i for i in sendto 
                              if not users.get(i, 'vacation', 0)]

                    # map userids to addresses
                    sendto = [users.get(i, 'address') for i in sendto]

                    # update the message's recipients list
                    messages.set(msgid, recipients=recipients)

                    # send the message
                    cl.send_message(nodeid, msgid, note, sendto)
            except roundupdb.MessageSendError, message:
                raise roundupdb.DetectorError, message

   Note that this is the standard nosy reaction code, with the small
   addition of::

    # filter out the people on vacation
    sendto = [i for i in sendto if not users.get(i, 'vacation', 0)]

   which filters out the users that have the vacation flag set to true.

Adding in state transition control
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Sometimes tracker admins want to control the states to which users may
move issues. You can do this by following these steps:

1. make "status" a required variable. This is achieved by adding the
   following to the top of the form in the ``issue.item.html``
   template::

     <input type="hidden" name="@required" value="status">

   This will force users to select a status.

2. add a Multilink property to the status class::

     stat = Class(db, "status", ... , transitions=Multilink('status'),
                  ...)

   and then edit the statuses already created, either:

   a. through the web using the class list -> status class editor, or
   b. using the ``roundup-admin`` "set" command.

3. add an auditor module ``checktransition.py`` in your tracker's
   ``detectors`` directory, for example::

     def checktransition(db, cl, nodeid, newvalues):
         ''' Check that the desired transition is valid for the "status"
             property.
         '''
         if not newvalues.has_key('status'):
             return
         current = cl.get(nodeid, 'status')
         new = newvalues['status']
         if new == current:
             return
         ok = db.status.get(current, 'transitions')
         if new not in ok:
             raise ValueError, 'Status not allowed to move from "%s" to "%s"'%(
                 db.status.get(current, 'name'), db.status.get(new, 'name'))

     def init(db):
         db.issue.audit('set', checktransition)

4. in the ``issue.item.html`` template, change the status editing bit
   from::

    <th>Status</th>
    <td tal:content="structure context/status/menu">status</td>

   to::

    <th>Status</th>
    <td>
     <select tal:condition="context/id" name="status">
      <tal:block tal:define="ok context/status/transitions"
                 tal:repeat="state db/status/list">
       <option tal:condition="python:state.id in ok"
               tal:attributes="
                    value state/id;
                    selected python:state.id == context.status.id"
               tal:content="state/name"></option>
      </tal:block>
     </select>
     <tal:block tal:condition="not:context/id"
                tal:replace="structure context/status/menu" />
    </td>

   which displays only the allowed status to transition to.


Blocking issues that depend on other issues
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

We needed the ability to mark certain issues as "blockers" - that is,
they can't be resolved until another issue (the blocker) they rely on is
resolved. To achieve this:

1. Create a new property on the ``issue`` class:
   ``blockers=Multilink("issue")``. To do this, edit the definition of
   this class in your tracker's ``schema.py`` file. Change this::

    issue = IssueClass(db, "issue", 
                    assignedto=Link("user"), topic=Multilink("keyword"),
                    priority=Link("priority"), status=Link("status"))

   to this, adding the blockers entry::

    issue = IssueClass(db, "issue", 
                    blockers=Multilink("issue"),
                    assignedto=Link("user"), topic=Multilink("keyword"),
                    priority=Link("priority"), status=Link("status"))

2. Add the new ``blockers`` property to the ``issue.item.html`` edit
   page, using something like::

    <th>Waiting On</th>
    <td>
     <span tal:replace="structure python:context.blockers.field(showid=1,
                                  size=20)" />
     <span tal:replace="structure python:db.issue.classhelp('id,title')" />
     <span tal:condition="context/blockers"
           tal:repeat="blk context/blockers">
      <br>View: <a tal:attributes="href string:issue${blk/id}"
                   tal:content="blk/id"></a>
     </span>

   You'll need to fiddle with your item page layout to find an
   appropriate place to put it - I'll leave that fun part up to you.
   Just make sure it appears in the first table, possibly somewhere near
   the "superseders" field.

3. Create a new detector module (see below) which enforces the rules:

   - issues may not be resolved if they have blockers
   - when a blocker is resolved, it's removed from issues it blocks

   The contents of the detector should be something like this::


    def blockresolution(db, cl, nodeid, newvalues):
        ''' If the issue has blockers, don't allow it to be resolved.
        '''
        if nodeid is None:
            blockers = []
        else:
            blockers = cl.get(nodeid, 'blockers')
        blockers = newvalues.get('blockers', blockers)

        # don't do anything if there's no blockers or the status hasn't
        # changed
        if not blockers or not newvalues.has_key('status'):
            return

        # get the resolved state ID
        resolved_id = db.status.lookup('resolved')

        # format the info
        u = db.config.TRACKER_WEB
        s = ', '.join(['<a href="%sissue%s">%s</a>'%(
                        u,id,id) for id in blockers])
        if len(blockers) == 1:
            s = 'issue %s is'%s
        else:
            s = 'issues %s are'%s

        # ok, see if we're trying to resolve
        if newvalues['status'] == resolved_id:
            raise ValueError, "This issue can't be resolved until %s resolved."%s


    def resolveblockers(db, cl, nodeid, oldvalues):
        ''' When we resolve an issue that's a blocker, remove it from the
            blockers list of the issue(s) it blocks.
        '''
        newstatus = cl.get(nodeid,'status')

        # no change?
        if oldvalues.get('status', None) == newstatus:
            return

        resolved_id = db.status.lookup('resolved')

        # interesting?
        if newstatus != resolved_id:
            return

        # yes - find all the blocked issues, if any, and remove me from
        # their blockers list
        issues = cl.find(blockers=nodeid)
        for issueid in issues:
            blockers = cl.get(issueid, 'blockers')
            if nodeid in blockers:
                blockers.remove(nodeid)
                cl.set(issueid, blockers=blockers)

    def init(db):
        # might, in an obscure situation, happen in a create
        db.issue.audit('create', blockresolution)
        db.issue.audit('set', blockresolution)

        # can only happen on a set
        db.issue.react('set', resolveblockers)

   Put the above code in a file called "blockers.py" in your tracker's
   "detectors" directory.

4. Finally, and this is an optional step, modify the tracker web page
   URLs so they filter out issues with any blockers. You do this by
   adding an additional filter on "blockers" for the value "-1". For
   example, the existing "Show All" link in the "page" template (in the
   tracker's "html" directory) looks like this::

     <a href="issue?@sort=-activity&@group=priority&@filter=status&
        @columns=id,activity,title,creator,assignedto,status&
        status=-1,1,2,3,4,5,6,7">Show All</a><br>

   modify it to add the "blockers" info to the URL (note, both the
   "@filter" *and* "blockers" values must be specified)::

     <a href="issue?@sort=-activity&@group=priority&@filter=status,blockers&
        blockers=-1&@columns=id,activity,title,creator,assignedto,status&
        status=-1,1,2,3,4,5,6,7">Show All</a><br>

   The above examples are line-wrapped on the trailing & and should
   be unwrapped.

That's it. You should now be able to set blockers on your issues. Note
that if you want to know whether an issue has any other issues dependent
on it (i.e. it's in their blockers list) you can look at the journal
history at the bottom of the issue page - look for a "link" event to
another issue's "blockers" property.

Add users to the nosy list based on the topic
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Let's say we need the ability to automatically add users to the nosy
list based
on the occurance of a topic. Every user should be allowed to edit their
own list of topics for which they want to be added to the nosy list.

Below, we'll show that this change can be done with minimal
understanding of the Roundup system, using only copy and paste.

This requires three changes to the tracker: a change in the database to
allow per-user recording of the lists of topics for which he wants to
be put on the nosy list, a change in the user view allowing them to edit
this list of topics, and addition of an auditor which updates the nosy
list when a topic is set.

Adding the nosy topic list
::::::::::::::::::::::::::

The change to make in the database, is that for any user there should be
a list of topics for which he wants to be put on the nosy list. Adding
a ``Multilink`` of ``keyword`` seems to fullfill this (note that within
the code, topics are called ``keywords``.) As such, all that has to be
done is to add a new field to the definition of ``user`` within the
file ``schema.py``.  We will call this new field ``nosy_keywords``, and
the updated definition of user will be::

    user = Class(db, "user", 
                    username=String(),   password=Password(),
                    address=String(),    realname=String(), 
                    phone=String(),      organisation=String(),
                    alternate_addresses=String(),
                    queries=Multilink('query'), roles=String(),
                    timezone=String(),
                    nosy_keywords=Multilink('keyword'))
 
Changing the user view to allow changing the nosy topic list
::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::

We want any user to be able to change the list of topics for which
he will by default be added to the nosy list. We choose to add this
to the user view, as is generated by the file ``html/user.item.html``.
We can easily 
see that the topic field in the issue view has very similar editing
requirements as our nosy topics, both being lists of topics. As
such, we look for Topics in ``issue.item.html``, and extract the
associated parts from there. We add this to ``user.item.html`` at the 
bottom of the list of viewed items (i.e. just below the 'Alternate
E-mail addresses' in the classic template)::

 <tr>
  <th>Nosy Topics</th>
  <td>
  <span tal:replace="structure context/nosy_keywords/field" />
  <span tal:replace="structure python:db.keyword.classhelp(property='nosy_keywords')" />
  </td>
 </tr>
  

Addition of an auditor to update the nosy list
::::::::::::::::::::::::::::::::::::::::::::::

The more difficult part is the logic to add
the users to the nosy list when required. 
We choose to perform this action whenever the topics on an
item are set (this includes the creation of items).
Here we choose to start out with a copy of the 
``detectors/nosyreaction.py`` detector, which we copy to the file
``detectors/nosy_keyword_reaction.py``. 
This looks like a good start as it also adds users
to the nosy list. A look through the code reveals that the
``nosyreaction`` function actually sends the e-mail. 
We don't need this. Therefore, we can change the ``init`` function to::

    def init(db):
        db.issue.audit('create', update_kw_nosy)
        db.issue.audit('set', update_kw_nosy)

After that, we rename the ``updatenosy`` function to ``update_kw_nosy``.
The first two blocks of code in that function relate to setting
``current`` to a combination of the old and new nosy lists. This
functionality is left in the new auditor. The following block of
code, which handled adding the assignedto user(s) to the nosy list in
``updatenosy``, should be replaced by a block of code to add the
interested users to the nosy list. We choose here to loop over all
new topics, than looping over all users,
and assign the user to the nosy list when the topic occurs in the user's
``nosy_keywords``. The next part in ``updatenosy`` -- adding the author
and/or recipients of a message to the nosy list -- is obviously not
relevant here and is thus deleted from the new auditor. The last
part, copying the new nosy list to ``newvalues``, can stay as is.
This results in the following function::

    def update_kw_nosy(db, cl, nodeid, newvalues):
        '''Update the nosy list for changes to the topics
        '''
        # nodeid will be None if this is a new node
        current = {}
        if nodeid is None:
            ok = ('new', 'yes')
        else:
            ok = ('yes',)
            # old node, get the current values from the node if they haven't
            # changed
            if not newvalues.has_key('nosy'):
                nosy = cl.get(nodeid, 'nosy')
                for value in nosy:
                    if not current.has_key(value):
                        current[value] = 1

        # if the nosy list changed in this transaction, init from the new value
        if newvalues.has_key('nosy'):
            nosy = newvalues.get('nosy', [])
            for value in nosy:
                if not db.hasnode('user', value):
                    continue
                if not current.has_key(value):
                    current[value] = 1

        # add users with topic in nosy_keywords to the nosy list
        if newvalues.has_key('topic') and newvalues['topic'] is not None:
            topic_ids = newvalues['topic']
            for topic in topic_ids:
                # loop over all users,
                # and assign user to nosy when topic in nosy_keywords
                for user_id in db.user.list():
                    nosy_kw = db.user.get(user_id, "nosy_keywords")
                    found = 0
                    for kw in nosy_kw:
                        if kw == topic:
                            found = 1
                    if found:
                        current[user_id] = 1

        # that's it, save off the new nosy list
        newvalues['nosy'] = current.keys()

These two function are the only ones needed in the file.

TODO: update this example to use the ``find()`` Class method.

Caveats
:::::::

A few problems with the design here can be noted:

Multiple additions
    When a user, after automatic selection, is manually removed
    from the nosy list, he is added to the nosy list again when the
    topic list of the issue is updated. A better design might be
    to only check which topics are new compared to the old list
    of topics, and only add users when they have indicated
    interest on a new topic.

    The code could also be changed to only trigger on the ``create()``
    event, rather than also on the ``set()`` event, thus only setting
    the nosy list when the issue is created.

Scalability
    In the auditor, there is a loop over all users. For a site with
    only few users this will pose no serious problem; however, with
    many users this will be a serious performance bottleneck.
    A way out would be to link from the topics to the users who
    selected these topics as nosy topics. This will eliminate the
    loop over all users.

Changes to Security and Permissions
-----------------------------------

Restricting the list of users that are assignable to a task
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

1. In your tracker's ``schema.py``, create a new Role, say "Developer"::

     db.security.addRole(name='Developer', description='A developer')

2. Just after that, create a new Permission, say "Fixer", specific to
   "issue"::

     p = db.security.addPermission(name='Fixer', klass='issue',
         description='User is allowed to be assigned to fix issues')

3. Then assign the new Permission to your "Developer" Role::

     db.security.addPermissionToRole('Developer', p)

4. In the issue item edit page (``html/issue.item.html`` in your tracker
   directory), use the new Permission in restricting the "assignedto"
   list::

    <select name="assignedto">
     <option value="-1">- no selection -</option>
     <tal:block tal:repeat="user db/user/list">
     <option tal:condition="python:user.hasPermission(
                                'Fixer', context._classname)"
             tal:attributes="
                value user/id;
                selected python:user.id == context.assignedto"
             tal:content="user/realname"></option>
     </tal:block>
    </select>

For extra security, you may wish to setup an auditor to enforce the
Permission requirement (install this as ``assignedtoFixer.py`` in your
tracker ``detectors`` directory)::

  def assignedtoMustBeFixer(db, cl, nodeid, newvalues):
      ''' Ensure the assignedto value in newvalues is used with the
          Fixer Permission
      '''
      if not newvalues.has_key('assignedto'):
          # don't care
          return
  
      # get the userid
      userid = newvalues['assignedto']
      if not db.security.hasPermission('Fixer', userid, cl.classname):
          raise ValueError, 'You do not have permission to edit %s'%cl.classname

  def init(db):
      db.issue.audit('set', assignedtoMustBeFixer)
      db.issue.audit('create', assignedtoMustBeFixer)

So now, if an edit action attempts to set "assignedto" to a user that
doesn't have the "Fixer" Permission, the error will be raised.


Users may only edit their issues
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

In this case, users registering themselves are granted Provisional
access, meaning they
have access to edit the issues they submit, but not others. We create a new
Role called "Provisional User" which is granted to newly-registered users,
and has limited access. One of the Permissions they have is the new "Edit
Own" on issues (regular users have "Edit".)

First up, we create the new Role and Permission structure in
``schema.py``::

    #
    # New users not approved by the admin
    #
    db.security.addRole(name='Provisional User',
        description='New user registered via web or email')

    # These users need to be able to view and create issues but only edit
    # and view their own
    db.security.addPermissionToRole('Provisional User', 'Create', 'issue')
    def own_issue(db, userid, itemid):
        '''Determine whether the userid matches the creator of the issue.'''
        return userid == db.issue.get(itemid, 'creator')
    p = db.security.addPermission(name='Edit', klass='issue',
        check=own_issue, description='Can only edit own issues')
    db.security.addPermissionToRole('Provisional User', p)
    p = db.security.addPermission(name='View', klass='issue',
        check=own_issue, description='Can only view own issues')
    db.security.addPermissionToRole('Provisional User', p)

    # Assign the Permissions for issue-related classes
    for cl in 'file', 'msg', 'query', 'keyword':
        db.security.addPermissionToRole('Provisional User', 'View', cl)
        db.security.addPermissionToRole('Provisional User', 'Edit', cl)
        db.security.addPermissionToRole('Provisional User', 'Create', cl)
    for cl in 'priority', 'status':
        db.security.addPermissionToRole('Provisional User', 'View', cl)

    # and give the new users access to the web and email interface
    db.security.addPermissionToRole('Provisional User', 'Web Access')
    db.security.addPermissionToRole('Provisional User', 'Email Access')

    # make sure they can view & edit their own user record
    def own_record(db, userid, itemid):
        '''Determine whether the userid matches the item being accessed.'''
        return userid == itemid
    p = db.security.addPermission(name='View', klass='user', check=own_record,
        description="User is allowed to view their own user details")
    db.security.addPermissionToRole('Provisional User', p)
    p = db.security.addPermission(name='Edit', klass='user', check=own_record,
        description="User is allowed to edit their own user details")
    db.security.addPermissionToRole('Provisional User', p)

Then, in ``config.ini``, we change the Role assigned to newly-registered
users, replacing the existing ``'User'`` values::

    [main]
    ...
    new_web_user_roles = 'Provisional User'
    new_email_user_roles = 'Provisional User'


All users may only view and edit issues, files and messages they create
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Replace the standard "classic" tracker View and Edit Permission assignments
for the "issue", "file" and "msg" classes with the following::

    def checker(klass):
        def check(db, userid, itemid, klass=klass):
            return db.getclass(klass).get(itemid, 'creator') == userid
        return check
    for cl in 'issue', 'file', 'msg':
        p = db.security.addPermission(name='View', klass=cl,
            check=checker(cl))
        db.security.addPermissionToRole('User', p)
        p = db.security.addPermission(name='Edit', klass=cl,
            check=checker(cl))
        db.security.addPermissionToRole('User', p)
        db.security.addPermissionToRole('User', 'Create', cl)



Changes to the Web User Interface
---------------------------------

Adding action links to the index page
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Add a column to the ``item.index.html`` template.

Resolving the issue::

  <a tal:attributes="href
     string:issue${i/id}?:status=resolved&:action=edit">resolve</a>

"Take" the issue::

  <a tal:attributes="href
     string:issue${i/id}?:assignedto=${request/user/id}&:action=edit">take</a>

... and so on.

Colouring the rows in the issue index according to priority
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

A simple ``tal:attributes`` statement will do the bulk of the work here. In
the ``issue.index.html`` template, add this to the ``<tr>`` that
displays the rows of data::

   <tr tal:attributes="class string:priority-${i/priority/plain}">

and then in your stylesheet (``style.css``) specify the colouring for the
different priorities, as follows::

   tr.priority-critical td {
       background-color: red;
   }

   tr.priority-urgent td {
       background-color: orange;
   }

and so on, with far less offensive colours :)

Editing multiple items in an index view
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

To edit the status of all items in the item index view, edit the
``issue.item.html``:

1. add a form around the listing table (separate from the existing
   index-page form), so at the top it reads::

    <form method="POST" tal:attributes="action request/classname">
     <table class="list">

   and at the bottom of that table::

     </table>
    </form

   making sure you match the ``</table>`` from the list table, not the
   navigation table or the subsequent form table.

2. in the display for the issue property, change::

    <td tal:condition="request/show/status"
        tal:content="python:i.status.plain() or default">&nbsp;</td>

   to::

    <td tal:condition="request/show/status"
        tal:content="structure i/status/field">&nbsp;</td>

   this will result in an edit field for the status property.

3. after the ``tal:block`` which lists the index items (marked by
   ``tal:repeat="i batch"``) add a new table row::

    <tr>
     <td tal:attributes="colspan python:len(request.columns)">
      <input type="submit" value=" Save Changes ">
      <input type="hidden" name="@action" value="edit">
      <tal:block replace="structure request/indexargs_form" />
     </td>
    </tr>

   which gives us a submit button, indicates that we are performing an edit
   on any changed statuses. The final ``tal:block`` will make sure that the
   current index view parameters (filtering, columns, etc) will be used in 
   rendering the next page (the results of the editing).


Displaying only message summaries in the issue display
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Alter the ``issue.item`` template section for messages to::

 <table class="messages" tal:condition="context/messages">
  <tr><th colspan="5" class="header">Messages</th></tr>
  <tr tal:repeat="msg context/messages">
   <td><a tal:attributes="href string:msg${msg/id}"
          tal:content="string:msg${msg/id}"></a></td>
   <td tal:content="msg/author">author</td>
   <td class="date" tal:content="msg/date/pretty">date</td>
   <td tal:content="msg/summary">summary</td>
   <td>
    <a tal:attributes="href string:?@remove@messages=${msg/id}&@action=edit">
    remove</a>
   </td>
  </tr>
 </table>


Enabling display of either message summaries or the entire messages
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

This is pretty simple - all we need to do is copy the code from the
example `displaying only message summaries in the issue display`_ into
our template alongside the summary display, and then introduce a switch
that shows either the one or the other. We'll use a new form variable,
``@whole_messages`` to achieve this::

 <table class="messages" tal:condition="context/messages">
  <tal:block tal:condition="not:request/form/@whole_messages/value | python:0">
   <tr><th colspan="3" class="header">Messages</th>
       <th colspan="2" class="header">
         <a href="?@whole_messages=yes">show entire messages</a>
       </th>
   </tr>
   <tr tal:repeat="msg context/messages">
    <td><a tal:attributes="href string:msg${msg/id}"
           tal:content="string:msg${msg/id}"></a></td>
    <td tal:content="msg/author">author</td>
    <td class="date" tal:content="msg/date/pretty">date</td>
    <td tal:content="msg/summary">summary</td>
    <td>
     <a tal:attributes="href string:?@remove@messages=${msg/id}&@action=edit">remove</a>
    </td>
   </tr>
  </tal:block>

  <tal:block tal:condition="request/form/@whole_messages/value | python:0">
   <tr><th colspan="2" class="header">Messages</th>
       <th class="header">
         <a href="?@whole_messages=">show only summaries</a>
       </th>
   </tr>
   <tal:block tal:repeat="msg context/messages">
    <tr>
     <th tal:content="msg/author">author</th>
     <th class="date" tal:content="msg/date/pretty">date</th>
     <th style="text-align: right">
      (<a tal:attributes="href string:?@remove@messages=${msg/id}&@action=edit">remove</a>)
     </th>
    </tr>
    <tr><td colspan="3" tal:content="msg/content"></td></tr>
   </tal:block>
  </tal:block>
 </table>


Setting up a "wizard" (or "druid") for controlled adding of issues
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

1. Set up the page templates you wish to use for data input. My wizard
   is going to be a two-step process: first figuring out what category
   of issue the user is submitting, and then getting details specific to
   that category. The first page includes a table of help, explaining
   what the category names mean, and then the core of the form::

    <form method="POST" onSubmit="return submit_once()"
          enctype="multipart/form-data">
      <input type="hidden" name="@template" value="add_page1">
      <input type="hidden" name="@action" value="page1_submit">

      <strong>Category:</strong>
      <tal:block tal:replace="structure context/category/menu" />
      <input type="submit" value="Continue">
    </form>

   The next page has the usual issue entry information, with the
   addition of the following form fragments::

    <form method="POST" onSubmit="return submit_once()"
          enctype="multipart/form-data"
          tal:condition="context/is_edit_ok"
          tal:define="cat request/form/category/value">

      <input type="hidden" name="@template" value="add_page2">
      <input type="hidden" name="@required" value="title">
      <input type="hidden" name="category" tal:attributes="value cat">
       .
       .
       .
    </form>

   Note that later in the form, I use the value of "cat" to decide which
   form elements should be displayed. For example::

    <tal:block tal:condition="python:cat in '6 10 13 14 15 16 17'.split()">
     <tr>
      <th>Operating System</th>
      <td tal:content="structure context/os/field"></td>
     </tr>
     <tr>
      <th>Web Browser</th>
      <td tal:content="structure context/browser/field"></td>
     </tr>
    </tal:block>

   ... the above section will only be displayed if the category is one
   of 6, 10, 13, 14, 15, 16 or 17.

3. Determine what actions need to be taken between the pages - these are
   usually to validate user choices and determine what page is next. Now encode
   those actions in a new ``Action`` class (see `defining new web actions`_)::

    from roundup.cgi.actions import Action

    class Page1SubmitAction(Action):
        def handle(self):
            ''' Verify that the user has selected a category, and then move
                on to page 2.
            '''
            category = self.form['category'].value
            if category == '-1':
                self.error_message.append('You must select a category of report')
                return
            # everything's ok, move on to the next page
            self.template = 'add_page2'

    def init(instance):
        instance.registerAction('page1_submit', Page1SubmitAction)

4. Use the usual "new" action as the ``@action`` on the final page, and
   you're done (the standard context/submit method can do this for you).


Debugging Trackers
==================

There are three switches in tracker configs that turn on debugging in
Roundup:

1. web :: debug
2. mail :: debug
3. logging :: level

See the config.ini file or the `tracker configuration`_ section above for
more information.

Additionally, the ``roundup-server.py`` script has its own debugging mode
in which it reloads edited templates immediately when they are changed,
rather than requiring a web server restart.


-------------------

Back to `Table of Contents`_

.. _`Table of Contents`: index.html
.. _`design documentation`: design.html
.. _`admin guide`: admin_guide.html