$Cambridge: hermes/src/prayer/docs/DONE-OLD,v 1.2 2008/09/16 09:59:56 dpc22 Exp $

10/01/2002
==========

Lots of undocumented work cleaning up the user interface. This file
probably now redundant!


18/11/2001
==========

Folder delete with confirmation disabled --> bang

prayer process could end up hanging if far end gets stuck:
  added (long) timeout as countermeasure.

<h3> --> <strong> on login and logout page until we can come up with
something better. (and later fixed stray </h3>: sigh).

Added (sometimes rather large) timeouts to all iostreams.

Search for closest map in zoommap.

Fixed logout screen

Added html_session_message to help screen.

Why does postmaster --> postmaster@hermes expansion work correctly even
though no entry in map? Suspect some help from c-client here...
  - postmaster alias defined: need to include in map!

list/unmark finds "closest" message before unmarking
  => no silly jumping around.

Add "Sender:" to messages if/when From address changed.

QP decoding on messages when Reply/ Forward: check closely!

16/11/2001
==========

Post message with invalid address --> boom.
  - Was failing to catch NIL response.

Run through validator.
  - Done (check on live system tomorrow!).

config file for accountd? Need to lift out stuff e.g: password strings.
  - First pass achieved

Run through filter file test.

Test QP engine again!

Silly little bugs:
  Added "cc" to short header list for compose and display. Doh!

  rfc1522_decode stuff not working correctly in msg list screen?

  Refresh no longer pushs to last message in folder if new mail.

  Cc not copied when forwarding messages.

  "Compose new message" on abook_lookup screen not getting on well with
  roles.

15/11/2001
==========

Reply -> Linewrap -> send causes reproducible crash.
  - Fixed: rfc1522_encode being painful..

Restart from list with substitution: garbage in output
  - response_redirect has to discard any accumulated body data.

Bugs:
  Sucessful passwd change crashes engine compare and contrast.
  Need to spot empty .MSforward upload
    - suggest read into array and then scan before writing file!

Reverse arrival sort was broken.

14/11/2001
==========

Negotiate filter_restrict on login 

13/11/2001
==========

Send binary downloads as HTTP/1.0 with use_persist disabled.
  - should work will all browsers this way.

Need background colour set on some frontend pages:
  set of utility routines?

Work out why icons pulled _every_ time that someone logs in from new session.
  - No idea: ask Jon

Add foreground, link colours to documents

Switched back to using text links rather than form entries on various
folder screens: appears that forms within tables are illegal in HTML 4.01!
  - See Interface Changes

Passed all screens through HTML validator looking for HTML markup bugs.

12/11/2001
==========

"-- " optimisations.

IP address lock now selective.

06/11/2001
==========

Check/rationalise ping events:
  SSL Key generate. Log cycling. CDB reopen.

Check error handling esp. logging of invalid input

Fixed abook import stuff

Save options more:
 After initial user interface level selection. Whenever abook updated.
 Done: may wish to only save when exiting abook_list screen?

02/11/2001
==========

BUG: Crash if sorting switched on in empty mailbox
  mail_sort doesn't like being passed an empty list? Fixed with special case.

prayer-session crashes if ispell not present
  - catch error condition.

Lost core dumps?
  - "daemon" option in Redhat init.d scripts disables this
  - Solution: just run prayer by hand


01/11/2001
==========

Fix rename -> Cancel bug

__ICONS__ == /icons/ 

Need log rotation script

User Preferences: Should have use_welcome and confirm logout on novice mode
  - help text correct?

30/10/2001
==========

Fix Date headers

Fix check_cookie -> fail case. Simple redirection

Lost core dumps?
  - No: just need to set ulimit -c unlimited
      - How do this portably?

Help text on preferences screen should update immediately if user-level
changed.

26/10/2001
==========

Sort out logging code (again):
  Either reopen log file for every entry or record inode and check it
  every couple of minutes
   ("struct log" as wrapper might help).

25/10/2001
==========

Fixed (Un)Delete and (Un)Mark bugs on display screen when sorting !=
arrival (didn't use zoommap).

23/10/2001
==========

Fix log/CDB file rotation.
  - DONE

Make sure that invalid page requests return 404.

robots.txt

22/10/2001
==========

Test concurrent access to mailfolder code with MBX folders

Forward messages needs to track zoommap for sorted message

Fixed Text/HTML stuff

19/10/2001
==========

Compose: Send, Cancel, Postpone should all return to correct parent
         screen. Does compose parent exist?

Increased width of abook_xfer field

18/10/2001
==========

Folder transfer: replaced Buttons with links

Extend compose lines

Expanded welcome and user_level screen

13/10/2001
==========

User Preferences for sent-mail and postponed-msgs folders

Mailbox --> Folder on Preferences screen

14/10/2001
==========

Change preferences code to only save stuff which is != default?
  means that we can quitely change defaults if problems emerge.

  Need to extend config to add missing option!

13/10/2001
==========

Completed code documentation after 4 days. Yuck!

Need config options for sent-mail and postponed-msgs.

Mailbox --> Folder on Preferences screen

09/10/2001
==========

account.c:
  Need to fix transparent reconnection + timeouts properly.

accountd:
  Conver int -> BOOL

accountd_server: need to support "red-intramail, yellow-intramail, ..."

08/10/2001
==========

Replace config_mailbox and config_maildir with session_XXX equivalents
  -> can apply more intelligence when building server locations.

config:
  Add cdb maps for imapd, accountd server
    - DONE: Need to test, use in config_mailbox and config_dir
  Add support for separate accountd_nis server.

Accountd:
  Use prayer style logging.

config:
  Need support for
    imapd_server = "yellow-intramail : red-intramail : ..."

config:
  Improve Documentation. Rename a few options?

config:
  Parser should allocate into one pool, then copy into second to minimise
  overhead. Decided not (at least for the time being: quite a lot of work).
  Changed my mind! DONE.

Check install procedure
  - Fix silly permissions. Generate self signed certificate as part of
    install (see stunnel install scripts).

08/10/2001
==========

Document command line options prayer+prayer_session

Separate SSL startup from iostream_create: need two separate routines for
client and server size startup.

Add SSL to (both ends of) the accountd stuff
  - DONE: Need to fix + test

Fix SSL accountd support.

Accountd:
  Need config option plus accountd command line options to work around
  mail/ restriction!

06/10/2001
==========

Replaced MSconvert.pl with equivalent C. Need to test fully

03/10/2001
==========

Add config option to determine prayer_background

Prayer session needs to support all options used by prayer frontend.

Prayer -> prayer-session startup
  Close down unused sockets 
  Need config option for min servers, max servers

Session Idle mode: shut down IMAP connections after few minutes idle time.

28/09/2001
==========

Second go at install script:
  make config, edit ./Config, make, make install
   + make install-config + make install
   need proper redhat start and stop scripts.

"make clean" should clear out Emacs backup files in docs directory.

Rename prayer-backend ---> prayer-session

Document properly (user + technical documentation).
  - docs/ mostly done now (will need to review!)

25/09/2001
==========

prayer-frontend --> prayer, should fork off subprocess

First go an install scripts

25/09/2001
==========

Need correct MIME encoding when sending messages containing 8 bit chars
  - Done.

Fixed attachment upload. Still some kind of problem involving download...

Added directory cache upload button for cmd_folders and cmd_transfer.
Need on other mailbox manipulation screens.

19/09/2001
==========

Short URLs links e.g:
  "AAAD@list@34@2"  --> .../session/dpc22//AAAD@list@34@2
  "/icons/fred.gif" --> .../icons:fred.gif

Port to FreeBSD and Solaris. 
  - Factor out OS dependant code into os.c: STARTED
  - Solaris needs EDG support, maybe fake urandom daemon using pwd hashes

12/09/2001
==========

Problem with SSL and Mac Netscape 4.7X: double clicks for many operations
 - Suggest: Use old/new versions of stunnel, see if any difference.
 - Answer Mac Netscape 4.7X appears to be broken.
    - Disable user_agent use_persist for Mac Netcape 4, come back and
      be more efficient later.

10/09/2001
==========

User Interface: Back --> Mailbox on some screens. Mailboxes--> folders.
  - DONE: may need to rename list --> mailbox (tedious!).

Correct MIME encoding for ISO-latin-1 in headers please!
  - DONE: Test please!

10/09/2001
==========

string_printf() --> pool_printf().

Fix Makefile.

Fix compilation with -Wall -W?
  - No: Lots of cases involving session where we might actually want
  - to use session logs etc at some later point

05/09/2001
==========

Why no coredump when running as root?
  - Answer: programs which start out running as root are paranoid about
    leaking state on at least some operating systems . Answer would be to
    bind ports, exec subprocess as unpriveldged user.

Links should be buttons on mailboxes etc screen
  - Try one and see what it looks like.
  - Answer: looks good. Have done this everywhere.

Add "Compose New message" to abook listing screen.

Should backend_session_inet use asynchronous send as well as receive for
normal requests? Appears to work just fine with aync input, sync output.

One last go at "Undo"?
  - Undo is working correct, just not entirely intuitively.

03/09/2001
==========

Shortcuts in novice mode.

Fix line wrap algorithmn: "I have"

"Save message" needs to do a ml_check if save to open stream.

From address on list screen should be link to display, not reply

Address book import should translate: "Surname, Firstname".

Small problem with compose and postponed messages?
  - tripping over own feet.

Colour clues for Compose screen?

30/08/2001
==========

Problem with empty mail folders & compose?
  - can't duplicate!

Login screen:
  Emphasis SSL Warning.
  Enter on Login screen should be login:
    NO: Don't think that you can do this w/o Javascript?

Warning to user page: cmd_welcome
  Emphasise fact that people should log out!
  Tell them to use navigation icons on page rather than browser.

Add abook alias: should be form button. Others?

Add new addressbook entry should warn if overwriting existing alias

cmd_seach: Year as input field rather than select.

Single name rather than First Name/Last Name in addressbook

28/08/2001
==========

MIME torture test:
  Passed first time (thanks to c-client!). 
  Mozilla however falls over in a heap. <Giggles>.

See Interface Changes file.

28/08/2001
==========

Moved Favourite manipulation to subscreen of mailboxes.

Replaced Logout icon

Added "Check" icon.

Added banner to bottom of display screen. Others too?
  - New: use_tail_banner option

Switched icons on list screen around to match display screen. This might
take several attempts to get it right.

Add "check for new mail" button to list screen. Rearrange icons so that
list and display correspond better? DONE

Added "(Hermes) Webmail service" to title bar of all screens.

Paperclip on cmd_display screen

All:
  Put "Hermes Email Service: xxx" or similiar in title bar

Initial User level:
  Indicate "_Webmail_ User Interface Level".
  Logout button.

Logout screen should say logout from Hermes.

26/08/2001
==========

Improve directory cache:
  Shouldn't need to invalidate parts of cache automatically
  Need invalidate link!

Factor out os dependant code into os.c
  - Added IP address handing code

Preferences file: Need some useful way to wrap lines.
  - Can't think of terribly sensible way to do this without two stage
    parsing: string_get_lws_line followed by decode, _then_ split into tokens.
  - Lets stick with the current approach for the time being!

frontend/session should bail out on illegal option

25/08/2001
==========

Folder transfer slow:
  Use multiappend for upload. Any way to improve download speeds?
  Done. Howeer, no obvious imporvement. Sigh!

Caching problems with Mozilla, Netscape. Look at what hotmail do.
  - Answer: use Javascript to clear out history and Cache. Sigh :(.
  - Leave as is for the time being.

Hide history list: Look at Javascript?
  - Leave as is for the time being

24/08/2001
==========

Better role system: NO

Other languages: Later

Fixed ghastly SSL bug:
  SSL * can mention own local read buffer. Need to check this
  before we select().

(use_persist == NIL) broken with direct connection. Almost certainly a
problem with SSL * buffering? No: a completely separate bug. Hmmm!

Mozilla caching far too much (Expires: ignored).
  - Yep: Doesn't seem any way to stop it. Sigh!
  - Check bug database. Warning message? Seperate window?
  - Seems to be true of all browsers. Check what Hotmail do.

23/08/2001
==========

Added border to abook_list

Save deletes message

Icons:
  Prev  --> Previous on list and display pages
  Abook --> Addressbook on list and display pages

Moved Back icon on display screen to the canonical place

Added banner to reply page

Lookup --> Expand on Directory page. Need Abook search to?

Save message => mark as deleted. Should be option?

23/08/2001
==========

Abook lookups: case insensitive lookup done right at the assoc_lookup level...

Save icon replaced (need a small version).

Address Books -> Addressbooks singulat.

Save Msgs: mark messages as deleted.

Emphasis status line. (And hide if no status)

Abook --> Addressbook, Prebv --> Previous etc

22/08/2001
==========

TUS meeting.

21/08/2001
==========

  Warning on HTTP screen (infer default HTTPS channel!)
  Accountd needs to have new canon encoding

iostream.c:
  SSL_accept should not be fatal error
  Translate to be end of file

Attachment download:
  Need to translate ' ' -> '_' in URL for attachment name somehow


17/08/2001
==========

Upload/Download mail folder broken?
  - Fixed. Still some speed issues here.

Verify configuration properly!
  - Done. May need to test fully

abook.c: Use hash for lookup

Lost favourites list?

Clean up management screens when accountd disabled.
  - No: they are fine the way that they are I think

Added icons to logout and rm screens to keep vrw10 happy...

r -> request in request.c and response.c

Fixed string_get_lws and use for e.g prefences.

Added whole bunch of stuff to config file

Need length limits for method and headers as well as body
  - DONE. Also fixed the response_error stuff (again)

16/08/2001
==========

Sanity check prefs esp: From, Reply-To and alt-addresses

Session preferences handled correctly? Automatic update should take effect

Add user_agent debug page

SSL: Do we need to support DH? Yes What is DH? Still no idea!
     Separate out as module? : NO

Check Quota error handling
  - First part complete: on the right track!
  - Test things more extensively as we go on

Verify Reply-To address in role system!
 - DONE. Also fixed silly bug with empty alias!

15/08/2001
==========

Better logging:
  More session events. Log HTTP level + response code sent to broswer

09/08/2001
==========

string_canon_encode() && string_canon_decode()
  - appears to be some kind of obscure bug with '@' in URLs and
    Mozilla 0.9.2? Appeared to be specific to cmd_display and
    cmd_rawdisplay. Replace with url_encode/url_decode throughout?
  - Need to investigate some.
 - Switched string_canon_encode to use '%' rather than '@'
 - lets see if this has any unpleasant side effects!

Bugs: Folder download broken? Long delay: 
  - Downloading large folder from IMAP server takes time!
  - Some kind of feedback would be useful!

Attachment Download names silly
  - Fixed (I hope...)

Forward and reply should forward attachments too, (as an option).
  - Already does this!

Put sessions into own process group so that kill signal to server doesn't
kill everything (actually rather useful while developing!)
  - Probably not

Personalisated From as well as default domain
  - DONE

Stuff affecting UI:
  Rich headers?

Dircache invalidate root doesn't work!
  - Now fixed

Fix frontend reuse-connection stuff for ports 9090 etc
  - Need test case to show to Ben!

Check that every string printed as html goes through buffer_puts_quote_html!
  - DONE



08/08/2001
==========

Need override mechanism for session setup:
  user_agent=mozilla, netscape, ie, lynx, other
  debug
  record_telemetry.
  failsafe (equiv: disable = all)
  disable = 1.1, persist,            <-- Only one that frontend interested in
            icons, cookie, frame, 
            html_inline, html_inline_auto, preserve_mimetype
  enable  = (the same)
    - fixed silly bug with user-agent stuff

Abook lookup should convert
  hermes -> hermes.cam.ac.uk etc?
  - NO

Better hash function.
  - Look at Perl algorithm
  - No: Existing one seems fine.

Use sigaction plus robust repear everywhere.

Signal handling code:
  At very least backend master server should remove session sockets.
  - No: current arrangement works rather well thank you!

Netscape has problems if images disabled
  - but only if frame in use => Netscape bug!

Added return_path_domain config option.

06/08/2001
==========

Mike O'Donahue needs quoting.

backend_session_inet can now cope with arbitary number of simultaneous
connections to backend. Question: Do we need asynchronous responses too?

Fix ml stuff: no automatic reopen...  : DONE

Streams:
  Flush .prayer file after each prefs/abook update:
  Would need special stream. :: DONE
  Special stream for uploads/downloaded :: DONE

Crash after timeout (UIDs changed)?
  - Need to check this consistency. Check all session_redirect --> "restart"
  - FIXED: was failure to call zoommap_associate after reopen!

Need to test all ioXXX fns for error codes
   No: io fns all silently discard output on error condition
   Only need to worry about this for fns that generate lots of output.
   - Should be safe

31/07/2001
==========

Pat thinks that "maintainly maintained forward file" stuff not obvious.
  Can't distinguish "no such file" from other NO responses from accountd
  server. Propose either:

    NO [NOEXISTS] ...      or
    OK {0}           

Message sending using c-client SMTP routines
  Use c-client routines to construct message with attachments
    - less likely to break down?
    - Look at Pine code for sending messages
    - NO: own version is fine.
        - apart from CRLF -> LF conversion which is a ghastly hack.

    - Consistent use of c-client message handling code could make our life
      much easier...
  - NO: Exim handling error queue much nicer.

Abook sorting.
  Probably not.

Adopt Exim style syntax for configuration file.

30/07/2001
==========

Possible to use Keep-Alive for permanant sessions, not for icons?
  - Removes overhead associated with SSL channel startup.
  - Part of the way there: HTTP redirects are a pain...
  - DONE: separate timeouts for icons and session connections.

Use short session_prefix if User agent can cope?
  - Netscape + Outlook seem fine. Lynx chokes.
  - 12K -> 10K (cmd_list)
  - 16K -> 12K (cmd_list)
  - 30K -> 24K (cmd_mailboxes)
  - Is a 30% saving really worthwhile? :: NO, probably not!

Repeated headers in HTTP request:
  Currently concatinated,
  Any situation where this is incorrect thing to do?  
  What does Apache do in this situation?
  Check Apache behaviour in general: might learn a few useful lesons.
    - cookies only likely problem case: have list/last split
  - Do nothing, at least for the moment.

Possible to remove intermediate copies of HTTP Requests/Responses
using chunked encoding. Looks like we have to supprt chunked requests
at least properly. Sigh...
  - One case still to be dealt with. Marked in source

Rename: zoommap -> msgmap to reflect additional functionality 
  - NO: Leave it alone!

Capability database for user agents:
  Session Persist       (yes for everything which supports persistent)
  Icons Persist         (no for Netscape, ?Mozilla)
  HTTP Pipeline         (yes if we ever find a broswer which supports it!)
  Use icons             (yes by default, no for Lynx and other text only)
  Use frame             (yes if likely to be useful)
  Use Javascript        (yes if likely to be useful)
  HTTP Redirect Persist (yes if session can persist across HTTP redirect)
  HTTP POST     Persist (yes if session can persist across HTTP POST)
  Send icons over HTTP link

Need consistant way of splitting up User-Agent strings

Clean up .prayer file:
  Version number
  Wrap/unwrap long lines
  true/false rather than 0/1.
  _Some_ level of sanity checking. However current strategy of ignoring
  everything that we don't understand works reasonably well.

Password changes:
  Need to remember two passwords, switch magically
  - DONE: Still need to test properly.

Add confirmation dialogue for delete mail folder
  - DONE

29/07/2001
==========

Remove proxy (as option)
  o could run each backend server on different port,           :: DONE
    with master server a la inetd to catch requests to expired :: DONE
    slots which have not been reused.

24/07/2001
==========

  o Possible to send icons over non-https channel? : DONE as hack
     - Icons channels should only use Keep-Alive if we are
       confident that it is likely to be a benifit e.g: HTTP/1.1 pipeline
     - RAN TEST: Works in IE and Mozilla but not Netscape (sigh)
     - Need some kind of capability table for browsers...

Added extra links on cmd_list screen

Added year to Dates on cmd_list, cmd_reply eytc screens.

Use prefork at frontend

23/07/2001
==========

Need directory permissions in prayer.cf file
  - DONE

Move files to /usr/local/prayer/[version] and /var/spool/prayer/[stuff]
  - DONE

Added check_directory_perms option to check directory permissions on
$var_prefix and directories that appear under it.
  - not 100% there but should be good enough.

Automatically create directory hierachy if doesn't exist.
  - Need some way to define directory permissions in config file.

Better line wrap algorithmn
  - DONE earlier

Split frontend/backend?
  DONE!

20/07/2001
==========

iostream.c -> frontend_iostream.c & backend_iostream.c.
Only compile in SSL support if it is required!

19/07/2001
==========

Add setproctitle with some useful information.

16/07/2001
==========

Add session default preferences to config option e.g: window sizes

13/07/2001
==========

Configuration file tuning.

Address lookups:
  If unqualified address doesn't appear in abook, hermes or cam
  databases then its an invalid address: should report as such
  immediately rather than trying to send mail?

12/07/2001
==========

Configuration file.

10/07/2001
==========

Rename:
  int -> unsigned long or BOOL forall int
  printf routines should support/use lu rather than d
 - DONE. About 3 hours work!

Generic List function.
  - DONE: moved all the obvious things over to use it

Replace ioputs, bgetc, bputs etc with macros

09/07/2001
==========

Further work on Line Wrap algorithmn

Mixed cased aliases in name.

Funny interactions with Mozilla: need extra CRLFs somewhere?
  - Extra CRLF at end of each response appears to fix this
  - read RFCs a bit more closely

Sort out modal help text.

Forward as postmaster, lookup "Pat" crashes?
  - No: large amount of text in Compose window stops User Agent
  = from sending.
  - Look at RFCs, make sure that its not Prayer doing something
  - stupid with byte counts. 

session_check_streams?
  - ml_ping inbox and other if > 20 mins                     : DONE
  - close postponed, transfer and prefs streams if > 20 mins : DONE

abook_entry_to_string:
  code duplicated in one or two places:
    should try and factor out set of routines that are single place
    so that any updates can be made consistantly.

  - abook_entry_to_string() itself
  - abook_list   - should interpret strings better: See "joel" entry
  - add_address  - clone of abooj_entry_to_strin() code is marked XXX
  - DONE


05/07/2001
==========

Need better line wrap algorithmn which wraps in blocks...
  - DONE. Hurrah! That was hard work!

Date search broken
  - Fixed

04/07/2001
==========

Pat:
  has some messages that blow up Prayer
  Fixed: invalid QP messages

Roger managed to blow system up on preferences screen:
  Yes - works for me too now.  The only thing I had done in the
  previous session before attempting to change the reply-to header
  default value was to send a message.  However, doing that now
  still works.

  - Fixed: options->prefs_save = options->prefs without copy.

DPC:
  Small button loses headers. Deja vu...
  - Fixed

DPC:
  Spell check in "Large" mode corrupts headers
  - Fixed

PH:
  Fixed cmd_help() to record last help text displayed for reload.

  redirect file:
    I went to the mail redirection screen. It correctly showed my
    redirection to CUS. However, below the top line of links it said:

      Status: Account Server response: no such file

    which seems a bit odd.

    Believe that he means vacation screen, however need to check this.
    Added a few literal_catch_no
      - may be better to have server generate OK {0} for empty files.

  Added "Send" button on Large screen.

6. I did a speeling check. It picked up the error and said it was
   highlighted, but it wasn't.

   Seems to be there!
   <pre>
   <b>Hllo</b> Wrld
   </pre>

cmd_add_address wasn't dealing with:
  Joel Seymour  "Joel Seymour <addr>" in same way as main
  abook_entry -> string routine. [Have companion routine?]

CJ:
  Wants rawdisplay to rawdisplay: DONE

PH: Would like "Switch to inbox" on list screen
  Not done unless someone else asks for it.

03/07/2001
==========

Use cdb database of CRSid->Full name mapping in both places.
  - Come back and do this later
  - Can also use CRSid database to decide when unqualified filter is safe
     (dpc22 --> dpc22@*cam.ac.uk)
  - Partly (mostly!) done.

Add comment: "including XX attachments" in postpone and forward code.

Addressbook lookup bug?
  Stella -> bang?

Assorted tidying up.

Fixed mailboxes bug

Address parsing 
---------------

Default domain for outgoing messages.

Stella stuff
============

June 25 - Reply all with 1 address in cc: produces a surplus comma
          To: Stella Page <sp253@cam.ac.uk>
          Cc: , spage@kanina.freeserve.co.uk 
  - can't reproduce this

02/07/2001
==========

T/C/B links on abook_list/abook_search pages now To/Cc/Bcc

Extra checks on form input into abook_entry...

Sort out HTTP headers in response
  Current situation is overkill
  - DONE. Mozilla caches regardless :(.

Extra user options:
  Display HTML mail
  Display text/plain that start <HTML> as HTML...
  DONE

cmd_display(), cmd_reply2() and cmd_forward1():
  All call ml_fetchbody(), then immediately set initial_msg[len] = '\0';
    - is this safe?
    - better way to do this?
    - factor out common code
  - Made all consistent. 
  - Sufficient space is allocated, so string[len] = '\0'; can't hurt?

Postpone message with attachments:
  Attachements get lost?
  - can't reproduce

Fix <td nowrap> problems in mailbox etc screens
  - Possible to define single mailbox "class" that calls back defined
    routines to fill in the skeleton.

  - appears to be sorted now.

29/06/2001
==========

Replace list_add, list_remove with:
  list_push, list_pop, list_shift, list_unshift
   - use unshift in filter menu

Fix frame use: init2 shouldn't be visible!

Reverse sort toggle

28/06/2001
==========

cmd_list: Trailer banner not correct if mail folder empty.

Display HTML inline with links
  Need to do this carefully to remove scripts
  Any conventions about correct way to do this?
   - steal code from sqwebmail which appears to have the right ideas.
  - DONE

Need to improve html_secure_strip_all()
  - replace with html_to_text fn that strips all tags,
    and can translate <&amp;> style codes!
  - DONE

27/06/2001
==========

sp253 bug reports:

June 07 - problem with Addressbook Entry 
          regrettably not reliably reproducible
          After a series of actions, e.g. adding and deleting
          entries, there comes a point when an attempt to add
          an entry fails quietly - i.e. after the Add/Update entry
          button is used the user is returned to the Addressbook list
          screen but the new entry has not been added

June 07 - possible enhancement
          don't let the buggers add an address entry without an
          alias part.

June 19 - HTML buglet - Address Search
          two <form> and only one </form>

June 19 - I would like to add some general words of wisdom somewhere
          along the lines of if you don't logout and the session times
          out you'll lose any changes you made to addressbook or options.
          Could be somewhere general or could be on options and
          addressbook entry pages.  Latter easier owing to handy links
          already there.  General location would also be nice - where?

June 21 - bug in novice message list screen - remove "Mark" td in tr

June 21 - with frames off I still get a target=_top assoc with Logout
          Options screen

June 22 - problem with changing password.  After an initial couple
          of deliberate mistakes entered correct current and valid
          new.  After selecting Change Password the session "hung"
          in the sense that the only action accepted was to use
          the "Stop" button in Netscape.  
          An accountd was started on prism at the time at which
          the Change Password button used
          sp999 15271  5250  0 12:27:38 ?        0:00 ./accountd --daemon

June 25 - Message list screen.  If "Mark All" selected, then "Zoom" should
          not appear

June 25 - small buglet in message listing - to do with division of
          number of messages in mailbox by number you want to list?    
          Scenario:-
          Start with listing 20 messages
          Take a mailbox with 20 messages in it.
          Change to listing 10
          Using the Prev/Next/First/Last buttons will only show you
             1-10 and 20 (it is not possible to list 11-19)
          There is a subtly different thing if you change to, say, 9 or the 
          default 12 messages per screen, the movement through the
          mailbox using Prev/Next/First/Last is:-
           - choosing 12 messages lists 1-12 and 12-20 (rather than 13-20)
           - choosing 9 lists 1-9, 9-17, 18-20

June 25 - stray (control) characters in "ACTION=" in forms.  Unfortunately
          I can't trace when/how this started but I am now noticing it
          every time I View Source or save a page, e.g. in
          mailbox list screen 
          (not maybe the most useful bug report :-( )
  FIXED: Actually stray <form> on banner screens after favourite
         folders set up. Missing fl->form = NIL in banner_reset;

June 25 - there are some stray wrapping of text in table cells.  Examples
          include
	   - mailbox list after one or more "favourite" mailboxes selected
  - Need to fix systematically
  - single mailbox listing fn with callbacks would be very useful.

 - Moved to separate TODO list

27/06/2001
==========

Passwords, quota etc screen:
   We are missing a screen for changing passwords, checking quotas and
   setting vacation/forward messages. This is going to require some
   kind of special protocol server at the Hermes end.


21/06/2001
==========

Added:
  Password Change
  Fullname Change
  Check Quota

Added (Un)Mark All to cmd_list screen

Moved search icon from top to bottom of screen

Experimental:
  Removed nowrap from banner lines
  Switch off 'font size="2"' for time being

Added sort on To and Cc

16/06/2001
==========

Add "favorite favourite" folder
  - DONE

Add shortcuts for subscribed folders at bottom of list screen.
  - (Not convinced that we actually want to do this)
  - Save marked messages to nominated folder
  - Change to nominated folder
  - DONE!

Use HTTP/1.1 Pipelining when Mozilla/Outlook support it reliably.
  - Should be able to push lots of icons down a single connection!
  - DONE previously

14/06/2001
==========

Help text layout from Stella:
  I think the following should all be part of one page
    compose/{role_list, normal, large}

    options/{general, folder, compose, advanced}

  I am easy on search - I am quite happy to amalgamate the existing separate
  pages for Text, Date, Status and Size into the main Search page.

  You're letting novices transfer mailboxes but not addressbooks?

Folder sorting
  - new improved zoommap that combines sorting and zoom mode
  - should probably rename as "msgmap".

12/06/2001
==========

Rearranged source tree:
  src and files subdirectories...

Pine addressbook import/export
 - Partly done: have address book export/import
 - Need to switch to using Pine format address books, which appear to
   be a series of TAB separated items:

   Single item:
      alias\tName\taddress\tComment

   Distribution list:
      alias\tName\t(addr1,addr2,addr3)\tFcc\tComment

   In both cases LWS padding in use! 

  - FINISHED modulo testing. Hurrah!

Mail folder import/export (standard Unix format mail folder).
  - DONE mostly: still need to clean up a little bit.


11/06/2001
==========

Use Frame to remove history list
  - DONE

Mail folder import/export (standard Unix format mail folder).
  - DONE (need to clean up a little bit!)

Address book import/export
  - First attempt finished. Need to refine.

Record session_redirects in log!
  - DONE

Improve HTTP error page (have links back to main session?)
  - DONE

07/06/2001
==========

Added HTTP 1.1 Pipeling?
  - Need some way to test this?

Extra options:
  Autozoom   after search
  Autounzoom after aggregate operation
  "save to sent-mail" should be permanently unsettable
  Use frameset to hide history
  Use HTTP redirects
 
Check user interface:
  Add session_message() and session_log calls as appropriate.
  - DONE PARTLY come back later and see what else is possible.

Need extra abstraction level to remove duplicate calls? : NO


Make sure that every ml_ call checked for error conditions
  about 1 days work.

Do a ml_ping if session idle for more than approx 20 minutes.
  - Need separate timestamps for each MAILSTREAM.
  - ml_XXX routines can recover last access on each stream...
  - Actually now do ping and then checkpoint
  - might what to make the checkpoingt optional

Try to reduce nnumber of random cmd_restart calls?
  Possible: Make cmd_XXX fns return (T) or (NIL)
    - NIL ->  automatic redirect to cmd_restart?
  - Tried this: ended up being far more complex that needed
  - Current situation is better arrangement:
    each cmd_XXX routine will always generate a page or redirect
    even if it wasn't the page that you were expecting after a few
    transparent redirects.

Session caching
  Get rid of HTTP redirects: (as option only!)
    This causes session to disconnect.
    Netscape only: IE better behaved (sigh!)
  Better now, however POST still seems to confuse Netscape. <Sigh>.

Record session sequence number and replay last command if sequence number
doesn't match

06/06/2001
==========

Worked out why Mozilla is not using Persistent HTTP connections:
  Mozilla is broken!

Logging:
  Possible to log more information than at the moment?
    - User_Agent and other headers.


05/06/2001
==========

Help text:
  Need stubs for forms in help text. (Must be way to have inactive form!)
  Need __PREFIX__ and __SESSION_PREFIX__ substitution in help text.

11 remaining XXX comments!
  - remove 2->3, include remainder in this file for sorting.
  - DONE

Bugs:
-----

Better handling for undo.
  - Undo completely broken. Remove entirely?
  - _believe_ that this is working now. Let Stella loose on it?

Spell checker needs some work:
  Case problems.
  Why does it convert first word to lower case?
  - Appears to be okay now that we have removed a few lcases!

Remove html_table*, html_form* nonsense with minimal set of functions,
plus some macros to generate form and table elements.
  - No: current naming scheme makes much more sense.
  - Only problem is that some of the names are > 14 characters

zoommap:
  - Use memblock to avoid need to allocate, reallocate block?  : DONE
  - Partial invalidation should now be possible!
      No, typically more work to work out which parts of the zoommap
      are still valid then to you rebuild the entire thing
      (esp now malloc()/free() overhead is removed, MAILSTREAM scan
       is only remaining bottleneck).

zoommap
  - Define stream/zoommap abstraction:
  - mark message
  - unmark message
  - clear all marks
  - set all marks
     => won't need to worry about getting zm_invalidates correct
     => no main program use of elt->spae
     => may be able to optimise certain cases
  - DONE

04/06/2001
==========

Should be possible to eliminate a lot of strip_whitespaces if string
processing routines do this automatically. Does anything actually _need_
leading/trailing whitespace preserved?  Look for strip_whitespace
references.
 - Was done at some point in the past.

request_decode_get() and request_decode_post() do string_url_decode for
us. Check that we aren't duplicating this elsewhere.
 - DONE.

Need maximum Request-Size: (25 Mbytes?)
 - DONE (need to clean up error strings a bit!)

Check Makefile dependancies. makedepend?
 - DONE

cmd_dispatch():
  - Switch to use binary chop

SSL support:
  Need to regenerate RSA temporary key once an hour
  - DONE

zoommap_find(): Switch to use binary chop
  - DONE

Fixed silly bug in mark/unmark code:
  - Forgot to run zoommap invalidate
  - Would be be better to abstract out all changes to the
    zoommap -> know when to invalidate.
  - FIXED

Use STR * instead of char * for strings
  - record size in each string could make a number of string operations
    a lot more efficient
  - Lots of work:
     Think some about this.
  - Thought: I don't think that we need this. Only real use is to make
    strip_whitespace more efficient, and should be possible to do this
    automatically.
  - DISCARD

30/05/2001
==========

Need to get rid of icons on mailboxes etc screen
  - DONE

Add message status icons unless (prefs->use_icons == NIL)
  - DONE. Need combined icons for mark?

Many assorted small cleanups (Reduced XXX comment count 75 --> 12)
  - replace individual mail_flag with sequences
  - replace mail_append with mail_copy (inc sequence) when appropriate
  - ml_append does auto_create on TRYCREATE
  - add &len parameter to draft_make_recipients and draft_make_msg
      (removes silly strlens on msg)
  - paniclog <-> mm_fatal cleanup and rationalisation
    (still need to log errrors).
  - alt_addr stuff in cmd_display lifted one level
  - ml_fill_overview() [still need zoommap version!]
  - html_XXX routines renames html_XXX_table where appropriate

Replace delete/undelete etc aggregate with sequence range stuff from
cmd_list.c
  - DONE

folder object to route mail folder manipulation through one location?
  - NO

Addressbook lookup applies far too many strip_whitespaces.
  - No longer the case!

Session timeouts:
  backend should just shut down after n mins of inactivity (should be easy!)
  - Did this some time ago!

29/05/2001
==========

Added address and subject length cap (using string_prune) on cmd_compose
(postponed message) and cmd_list screens.

Replace all UID tests in
  cmd_delete etc with single call.


25/05/2001
==========

May 23 - HTML buglet - Options form
         -  <H3>General Options:</H3>
            <td><input type="hidden" name="type" value="general"></td>
            The <TD></TD> is not inside a table
         - No <P></P> round "User Interface Level:" although I agree
           this buggers the spacing.  Works with another TABLE

May 23 - HTML buglet - Options -> Roles
           Multiple occurrences of 
		<HTML>
		<HEAD><TITLE>Roles</TITLE></HEAD>
           etc.
         No <P></P> round "Options:"

May 24 - Did you say you were going to add a "Back" to the main options
         screen?

May 22 - Options form
         - What's on each sub-form in the way of "Help", "Back" etc is
           different.  There's nothing under "General", "Folder" or
           "Compose".  There's 3 GIFS under "Roles" and a "Back" input
           choice under "Personal Dictionary".
         - there doesn't seem to be an "action" defined for the POST
           at the top of the form?

May 24 - two probably thick queries about the Dictionary screen.  
         - If they can delete worms by clicking on them do they really 
           need a "Remove" field as well?
         - Won't it confuse the little dears if they enter a Worm with
           a capital and it appears as a worm without a capital and in 
           front of the all-lower-case worms?

May 22 - odd needs

- Forms in help text, if help is HTML rather than GIF based.  
  - We need a null stub for POST and GET. One to cover both that just 
    returns a "Nope, not active" type message? Named something like
       action="https://magenta.csi.cam.ac.uk:8080/actionstub"
    maybe?
  - I imagine 
       input type="submit"
    things would be dealt with as-if-by-magic if the action changed?

-------------------------------------------

May 24 - trouble with Spell Check.  My original message was:-
    
	  Morandacat and I would like to invite you to join us, that is
	  me and mirandacat at home at CB1 3LL or rather CB1 3LD.  That
	  should trigger something in the spell checker!

         By the time I'd spell checked it (a) it didn't seem to pick
         up my all my Personal Dictionary words - these contain CB1 (which
         it did accept as OK), mirandacat and 3LD (which it didn't), and 
         it changed everything to lower case thus

	 morandacat and I would like to invite you to join us, that is
	 me and mirandacat at home at CB1 3ll or rather CB1 3ld.  That 
         should trigger something in the spell checker!  

         This appears to be related to the fact that the Personal
         Dictionary screen seems to auto-convert everything to lower
         case - see point above this?


22/05/2001
==========

Add lots of help text icons, just to cheer up Stella

Delete folder fails to remove from dircache?

Message status for messages > 100 in folder not displayed correctly!!!
  - DONE (silly string -> number conversion bug !)

Fixed running against Netscape 4.08 (ghastly hack stolen from old version
of Stunnel is not required with modern versions of openssl library:
  #define SSL_CTX_set_tmp_rsa_callback(ctx,cb) \
     SSL_CTX_ctrl(ctx,SSL_CTRL_SET_TMP_RSA_CB,0,(char *)cb)

Small spelling buglets 

- Compose screen reached in expert mode after "Forward marked" selected.  
  Subject:  
  ^seperate^separate

Small HTML buglets and queries

- change <img src="/icons/foo.gif etc etc> to
  <img src="https://magenta.csi.cam.ac.uk:8080/icons/foo.gif etc etc>
      fairly generic, seen in
         message list screen, mailboxes screen

- delete the </h4> after the CRSid of the logged-in-user
    screens so far - message list, mailboxes, addressbook

- in the message list screen - the first <table width=100%>
  add " " <table width="100%">
   so far seen in - message list screen

- can the blank ALT tags be filled in with something?  

- tables - 
  <tr><font size="2">
  delete the <font size> and matching </font> just before </tr> 
  bit as it's in/applicable to TDs
      screens so far - message list, mailboxes, addressbook

Stella
======

Date visible in message srceen listing changes when saving a message:
Message sent on 17th May shows as that in inbox
   20.  No   May 17   Pat Stewart   1K   An FAQ?    D S R F 
Save message to folder
Date visible in message screen listing is date of save 
  157.  No   May 18   Pat Stewart   1K   An FAQ?    D S R F 

=============================================================

Random small things

- Sometimes they're mailboxes and sometimes mail folders, e.g.
  see the "Mailboxes" screen with Subscribed "mail folders" and
  creating "mailbox".  Is there any point in standardising on one name?

21/05/2001
==========

Check running against Mozilla
  - Hangs: expecting more input that it is getting on each transaction?
    Extraneous CRLF in send_html interaction.

Something funny going on with Back button in Roles screen:

  HTTP/1.0 200 OK Date: Tue, 08 May 2001 12:18:07 GMT Server: Prayer/0.0
  MIME-Version: 1.0 Allow: GET, HEAD, POST
  Connection: Keep-Alive Content-Type: text/html; charset=iso-8859-1 Pragma:
  no-cache Cache-control: no-cache Expires:
  Tue, 08 May 2001 12:18:07 GMT Content-Length: 1589 

  blatted across the top of the screen, above the otherwise normal
  remainder of the form. A long pause showing the watch cursor ensues.
  I was going "back" from "roles" to "options". In fact it seems to be
  repeatable.

  FIXED:
    either CRLF bug or string_itoa_tmp bug

17/05/2001
==========

IMAP connection dropped after 30 minutes idle time.
  - Need automatic/transparent reconnect:: DONE

16/05/2001
==========

Transparently create sent-mail folder just like the others.
  - Should work. TRYCREATE stuff broken?
  - TRYCREATE stuff fixed, hopefully consistent.

Putting an entry such as cs-info in the addressbook shows us as cs%2Dinfo
  - Fixed: missing buffer_puts_quote_html

15/05/2001
==========

Need inactive icons on message display screen.
  - DONE

Text under icons too small.
  - Made them a bit bigger.

Draft <-> Role integration
  - Select role on compose screen
  - Trigger role based on address book.
  - NO: at least not on this interation.

09/05/2001
==========

Objective:
==========

Have something useful by next MDCM (9th May).
  - DONE!


04/05/2001
==========

Looking for silly MM bug.

Session disconnect.

03/05/2001
==========

Logging
-------

Need to log normal + abnormal events   - 2/3 days work.
  - DONE

Extra options  - about two days work.
-------------

Simple options:
 - Use icons. 
 - Size of Small and Large Compose windows: DONE

Advanced options, disabled by default: DONE using USER_LEVEL
 - Roles               : DONE
 - Postpone message    : DONE
 - Spell check         : DONE
 - Large/Small window. : DONE

Prettify the interface
 - DONE (some tweaking remains?)

Unmark after aggregate operation
 - DONE (test fully!)

02/05/2001
==========

Icons
-----

Need to apply icons
  - Need separate HTTP and HTTPS listeners!

Easy route:
 - repeat headers x 2

Harder route:
 - build up list and apply.

01/05/2001
==========

Fixed ISO-8859-1 address display. Question: other charsets?

30/04/2001
==========

Assorted things that I've decided not to do.
--------------------------------------------

Printing?
  - should be reasonably easy for plain text.
  - what to we do about inline graphics etc?
  - NO: Not unless we are specifically asked for this.

Private version of rfc822_parse_adrlist based on pool allocation
might be useful. Would remove lots of fun memory allocation.
  - Quick look suggests lots of duplication from c-client.
  - Private library of address parsing code might be appropriate!



Spelling list
-------------

 - need ability to add and remove words from the personal dictionary

Search
------

Search on date, status and size

addr.c
------

Nasty mess: must be a better way to do this!
  - DONE. (Stolen some code from Pine to estimate size of ADDR!)


Misc
----

Eliminated a few strlen and sprintf calls.

msgno should be long throughout: check.


27/04/2001
==========

Fixed include and attachments options
-------------------------------------

  - string_get_lws_line was broken. Sigh...

Fixed "reply" bug:
  Need to check Makefile dependancies are correct

Clean up session structure :: Unknown
--------------------------

Move data out into sub-structures             :: NO
  - Probably not: just clean up into groups.  :: DONE

Possible to support multiple folders?  :: DONE
  - inbox and other.

Why do we have two IMAP sessions open at the moment?
  - FIXED

Use multiple IMAP sessions:  DONE
  INBOX
  postponed messages
  One other folder != inbox

Cleaned up cmd_restart code: DONE

Cleaned up init code: only needs single IMAP session to get started

26/04/2001
==========

Postponed messages :: 2 days
------------------

Proper postponed message folder
  - "Postpone" button saves to folder
  - "Compose" looks for postponed messages (have flag)
  - "Need to check for postponed messages at startup


25/04/2001
==========

Option screens  :: 1 day  :: DONE in entirity
--------------

Make options screens work properly.
  - Need three copies of preferences data:
      initial  : In case we want to rewrite .prayer folder
      active   : Working options
      working  : for updates within options screen before commit
 - Primitives:
     Create new empty preferences list
     Generate copy of preferences
     Free up list of preferences.
 - Question: use pools?
     Preferences don't change very frequently.

Move abook and dictionary outside prefs.

Address parsing  :: 1 day
---------------

Use local versions of c-client routines for both address book lookup
and message sending. Also alt_addr stuff. DONE

Use cdb database of CRSid->Full name mapping in both places.
  - Come back and do this later

11/04/2001
==========

- Clean up folder list                  - DONE
- integrate directory cache             - DONE
- spread options over multiple screens  - DONE (Add more options!)

Address book lookup:
  Recursive aliases?                 - DONE
   Need some way to detect loops.    - DONE
   Could be more efficient!

Directory cache:                       DONE
  - at the moment invalidate whenever anything changes
  - possible to make this more efficient?

cmd_mailboxes()/cmd_save()/cmd_rename():
  - Sort out confusion and duplication
  - (suggest three separate cmd_files with common code factored out)
  - Done.



10/10/2000
==========

- "compose/fresh".

- Expunge with large numbers of messages -> out of memory
    last < first at line 236 of cmd_list_simple().

-  variant on the "Re: Re:" bug, now only applicable if the Re: is
   capitalised (RE:)

- purge in zoom mode crashes 
    need to invalidate zoom map and recalculate session->marked.

- Undisclosed Recipient list.

- Replying to 
  From: "Eburne, Katharine" <katharine.eburne@ic.ac.uk>
  strips off the quotes and sends to
  To: Eburne@cam.ac.uk, Katharine <katharine.eburne@ic.ac.uk>

- Should always download as octet/byte-stream

- Attachment names.

- Aggregate save misses stream->nmsgs

Things done by 10/10/2000
=========================

cmd_compose():
  Large window mode:                DONE
  Sort out silly interdependancies: DONE
  Add undo option:                  DONE

cmd_list():
  Should prefetch envelope data as single mail_xxx operation. Currently
  many round trips to server.

cmd_list():
  Move search stuff to bottom of screen: DONE

FULL hdrs in cmd_display

Things done by 01/10/2000
=========================

Replace popen() in cmd_send with some sane code.
  - DONE (Need to test properly!)

Add personal dictionary to spell check: DONE
  - Stage 1:
      Accept for spell check run
        - add to tmp personal dict
        - File processed line by line:
          need to remember accepts for current line (request->pool fine!)

  - Step 2:
     Add perseonal dictionary
     - Store as "Dict: " lines in prefs message.
     - send to /tmp file each time speller starts
     - record changes to personal dictionary immediately
        => don't have to parse /tmp file.

  - Try to clean up variable names used in struct speller:
      - multiple I/O streams rather confusing at the moment.

Things done by 11/08/2000
=========================

Important:
==========

Need some way to pass IP address of client on to backend to check okay.
Suggest special header line at start of request that is only allowed
on proxy connections.


DON'T worry about HTTP/1.1
  - make sure that HTTP/1.0 code is stable
  - HTTP 1.1 fron end can be bolted on afterwards,
     shouldn't affect rest of the code

Add support for Cookies.

Need unique session IDsx

Other
=====

Improve error reports.

request_* code should use any of CR, LF and CRLF for line break
  - Break on CR or LF
  - If CR swallow LF if next character
  - not urgent, should be able to replace method by method

Need to fix hash_insert

Server response 501 (Not implemented) for anything that we recognise,
don't implement.

OLD list
========

Replace message -> buffer (so name reflects actual use)
  - New methods
     bputc    (macro)
     bputchar
     buffer_seek_offset    (simple implementation)
     buffer_printf         (simple implementation: use sprintf)
     bprintf

Add simple explanatory message body to error messages
  - response_error(status)
  - All responses other than following must have Content-Length header:
      1xx (informational, not used in 1.0)
      204 (no content)
      304 (not modified)

0.9 request
  => reply with imformative message.
  DONE. Need some way to test.

%XY decoding in URLs
  - Don't think that we actually need this for Webmail only app.

Check ".." checking correct
  - don't expect ".." to be vaoid

Add nodup option to hash_update.

Need versions of hash and buffer data types which work without pool
  - suggest that NIL => no pool
  - use alloc (malloc with fatal) instead of pool_alloc
  - pool_alloc could use malloc if NIL


Get rid of ALL fixed length buffers that might involve user input?

POST requests must content Content-Length in 1.0
  - should generate 400 (bad request) if they don't

BUFFERSIZE of 0 for
   buffer_create etc should be same as iostream_create
   - use system provided default

URL encoding:
  Currently using '@XY' to encode special characters for both
  URLs and GET forms. WING only uses '@' in URLs where '%'
  might be interpreted by Apache. Do the same thing?

  DONE: %XY used everywhere apart from filenames which are embedded
  directly in URLs


HTTP 1.0 only server:
  Possible to merge iostream and buffer, reduce data copying
   only IO operations that we actually need are:
     read entire request
     write entire response.

  Actually rather messy as we need to calculate Content-Length in
  header. Simplest solution is to generate response body, then header
  and combined. Requires multiple small writes without iostream.