#ident	"@(#)smail:ToDo,v 1.77 2003/06/18 00:20:36 woods Exp"

Things that should be done before the next minor release (patches are,
of course, gratefully accepted!):

Important Bugs:
--------------------

- DO NOT RUN SMAIL AS ROOT!!!!

  + the daemon will setuid(nobody:smail) and re-exec itself after it has
    a file descriptor already bound to port 25 (it will have to be
    started by root on most systems, of course)

  + the queues will be writable by the group 'smail' (not "mail") and
    owned (and also writable) by the user 'smail'.

  + the main smail/sendmail binary will be setgid-smail (not 'mail') so
    that it can write to the queues (but owned by root, of course).

  + checkerr will be run as smail:smail

  + local delivery to spool files will be done only by a separate
    setgid-'mail' agent ala *BSD mail.local, with kernel locking where
    possible, *.lock files only where absolutely necessary.

  + initial local spool file creation, if necessary, will be done by a
    tiny setuid-root helper on systems that have a root-only chown(2)
    [one exists somewhere already -- search the net]; they will be owned
    by the user, group 'mail', and be mode 660.  The spool directory
    will be mode 555 if kernel locking is possible, else 575 & group
    'mail' if necessary for *.lock files (and in all cases owned by
    root, of course).  the helper will _only_ create an empty spool file
    if one does not already exist, and it will determine the pathname to
    use based on the user name given to it on its command line.

  + all mail readers will be expected to either use a small helper to
    safely copy the spool file to a private place (ala movemail, which
    will use kernel file locking when possible or be setgid-mail and use
    *.lock files if necessary), or to use kernel file locking to access
    the spool file (and hopefully copy it away to a private place while
    th user does his/her thing); mail readers (including movemail) will
    be "encouraged" to keep the zero-length spool file after emptying
    it; and use of setgid-mail for readers will be very Very VERY
    strongly warned against (anything more complex than the old V7
    /bin/mail has had many bugs in this regard for decades now).

  + .forward files will have to be world readable (or at least readable
    by the group 'smail'), *or* "Forward to" support can be used.

  + delivery to files and pipes done only as nobody:nogroup regardless
    of where they're expanded from or who "owns" the address.  Anyone
    wanting more will be advised to deliver to an intermediate
    lock-protected spool and to run a collector daemon/periodic-job as
    with the ultimately desired identity, though nothing will prevent
    the delivery to a pipe from execing a setuid binary.

  + access to locked files will only be attempted for a limited amount
    of time and messages will be left in the queue if delivery is
    unsuccessful because of the presence of any pre-existing a lock

- make sure foreign data is safely formatted wherever it's used:

  + make sure TXT records returned from RBL lookups, as well as strings
    included in smtp_hello_reject_hosts, cannot violate SMTP message
    response rules.

  + make sure domain names and other data returned from the DNS cannot
    violate SMTP message response rules or RFC 822 header text rules.

  + similarly always run raw network data through strvis(3) before sending
    it back out to the network in the likes of an SMTP response message,
    or before logging it.

  + use strvis(3) to do this.  borrow from 4.4BSD for other systems.

- think about always truncating the bodies of bounces....

- fix bug where qualify_domain() isn't called for addresses specified on
  the command line (i.e. it only seems to be called when '-t' is used)

- use ftruncate() to remove partially written messages in appendfile.c
  if ERR_135 [if possible].

- investigate extra <>'s in received for bounces (and other places):

  Received: from most.weird.com (4544 bytes) by most.weird.com
	via sendmail with P:bsmtp/D:user/T:local
	(sender: <MAILER-DAEMON>) (ident <MAILER-DAEMON> using unix)
	id <m0x2LJo-00076wC@most.weird.com>
	for <<woods>>; Sat, 23 Aug 1997 14:53:52 -0400 (EDT)
	(Smail-3.2.0.98-Pre 1997-Aug-19 #7 built 1997-Aug-20)

  Extra <>'s also appear in message log entries too... most of the time.

  They may also be screwing up RFC-821 local part quoting....

  Or maybe they're just an artifact of the display code not checking
  first to see if there are already enclosing angle brackets or not.

- deal with un-qualified local hostnames when there's no qualify file in
  some sane way....  (the qualify.c stuff is perhaps overloaded and
  shouldn't be used to qualify both local names in outgoing headers at
  the same time as being used to qualify destination hostnames).

- fix "from_field" to never allow "From:" to go missing and if it's nil
  do something appropriate....

- fix db lookup parser to allow '#' in left-hand side (if quoted?) [aliasfile]

- investigate the Apparently-To: being set, while input_addr not being
  set: (no To/Resent-To/Cc/Bcc, etc., header in data, just envelope
  "MAIL FROM:")
	
	Received: from [204.92.254.3] by most.weird.com
		via sendmail with smtp (ident woods using rfc1413)
		id <m0udnxb-00076qC@most.weird.com>
		for <unknown>; Tue, 9 Jul 1996 21:20:59 -0400 (EDT)
		(Smail-3.2 1996-Jul-4 #1 built 1996-Jul-4)
	Apparently-To: foo@anet

  perhaps $input_addr should be set from envelope (always?).

- investigate smail vs. MH via SMTP and BCC.  Seems the BCC line can end
  up in the initial Received header.  The exact address that appears in
  the first Received header will vary if there are multiple destination
  addresses.

    Received: from woffi.planix.com([204.29.161.34]) (1436 bytes) by whome.planix.com
	via sendmail with P:esmtp/D:aliases/R:inet_hosts/T:smtp
	(sender: <andreas@planix.com>) 
	id <m0x3NUM-0008NDC@whome.planix.com>
	for <partners@planix.com>; Tue, 26 Aug 1997 11:25:02 -0400 (EDT)
	(Smail-3.2.0.97 1997-Aug-19 #2 built 1997-Aug-25)
    Received: from localhost.planix.com(localhost[127.0.0.1]) (1104 bytes) by woffi.planix.com
	via sendmail with P:esmtp/R:inet_hosts/T:smtp
	(sender: <andreas@planix.com>) 
	id <m0x3NUL-000EExC@woffi.planix.com>
	for <customers_hidden@planix.com>; Tue, 26 Aug 1997 11:25:01 -0400 (EDT)
	(Smail-3.2.0.97 1997-Aug-19 #2 built 1997-Aug-19)
    To: customers@planix.com (to /dev/null), partners@planix.com (an alias)
 >> Dcc: customers_hidden@planix.com (a private alias to everyone)

  Note also that MH uses 'Dcc' instead of 'Bcc' for normal (direct)
  blind carbon and that this header may not be stripped either!

- do something about the premature lower-casing of user names.  Users
  with upper case characters may not be able to receive mail (or at
  least read the stuff they've received....)  The correct solution is
  probably to provide another field in struct addr in which the
  un-adulterated user-id can be stored for use in the "local"
  transport's filename expansion.  I.e. the "user" director, with the
  'ignore-case' attribute set, will do a caseless match of the user-id
  against the mailbox portion of the address, and then the actual
  user-id with case preserved can be used in generating the mailbox
  spool filename.  [PR#295 notes that getpwbyname() in pwcache.c
  explicitly lowercases the user name passed to it before a getpwnam()
  search is proposed and the PR actually suggests removing this
  lowercasing (so that the case is preserved in the cache), but still
  doing a case-insensitive search through the password file, though it
  doesn't pay heed to the ignore-case attribute, nor does it provide for
  storing the case-preserved user-id in struct addr.]

- fix aliasfile parser to allow case sensitive aliases (ala above?)
  [keep in mind the lists director uses "lists/${lc:user}"]

- fix expand_string() et al to always return newly allocated storage.

- turn down the verbose logging of failed locks, if known other smail
  process holds lock....  eg:

	02/28/96 12:07:36: open_spool: /local/var/spool/smail/input/0trpIB-00076nC: lock failed: Permission denied

  Unfortunately this will probably require re-writing the spool locking
  functions to use pid-in-a-lock-file mechanisms.  [effectively fixed in
  3.2.1 for systems that return EAGAIN if lock_fd() meets another lock?]
  [it has been noted that there may be real race conditions in here!]

- check out re-writing From: if from '-f'

- Make sure "From:" and "To:" are always generated correctly for all
  locally originating mail and never for anything else.

- check out what's going on with Apparently-From being added multiple
  times [Apparently-From should be gone from 3.2.1].

- stop smail from generating those horrible Apparently-* headers now
  that the envelope is completely available in the default received
  header [Apparently-From should be gone from 3.2.1].


Incomplete Features:
--------------------

- implement an outbound ACL to prevent delivery to destinations listed
  in either smtp_output_reject_hostnames, smtp_output_reject_ipaddr,
  smtp_output_reject_dnsbl (addrs), or smtp_output_reject_rhsbl
  (hostnames).  Maybe these can be checked during verify too so that
  they'll even fail at RCPT TO:  time, though that would then require
  that at least the domain/host forms be excepted if they are
  "islocalhost()".  We may need smtp_output_reject_except_{host,ipaddr}
  lists too/anyway.  This will allow the like of inputs.relays.osirusoft
  to be used to block even command-line users from sending to a known
  open relay.

- implement a client-source-address based ACL to allow restricted EXPN
  usage (smtp_expn_allow?).

- make sure RFC-[2]821 local-part quoting isn't too aggressive and
  re-quoting things that don't need re-quoting.

- check to be sure we use IANA registered protocol names, etc. in
  received headers and such places.

- don't allow bogus A RR's to "match" (0, 255.255.255.255, 127/8,
  RFC-1918 addresses etc.).  Probably need to provide a config variable
  that contains a list of "bogus" addresses, something like
  smtp_bad_mx_targets but more general for all A RRs.  Perhaps names
  given in HELO would skip this check if the client address matches in
  smtp_remote_allow or something.  What about names for internal MX
  hosts though?  How do we know if they're "internal"?

- When command-line recipients are given, but there's no "To:" header,
  add one like "To: undisclosed-recipients:;", just as Postfix does.

- think about making mailq's "-E" work for runq too.  [unfreezemail
  effectively does this though]

- add an option to 'mailq -E' that'll only match if the undelivered
  addresses are local or not to make it easier to find and unfreeze
  large batches of messages which might now be locally deliverable (this
  could be especially useful for Cyrus IMAP sites where many messages
  get frozen because of quota restrictions).  [this is partly done now
  in a hackish way by the tempfail retry logic in util/checkerr]

- implement client name loop detection [compare HELO name with what we
  give in the 220 startup message, and vice versa] (not that it ever
  seems to be needed, but perhaps some idiot will mis-configure things
  so badly that such a simple check will save their necks).

- modify the host retry locking mechanism to allow for multiple
  concurrent deliveries to a given target host.  Perhaps each sender
  locks the retry file only temporarily, not for the whole duration of
  its delivery attempt, and if the file is new (or empty?) or contains
  only fewer than the concurrency limit of "pid N" lines then it appends
  its own "pid N" line to the end of the file, unlocks it and goes on.
  If the file exists but contains an error number and message then it
  does the normal retry duration timeout first.  The concurrency limit
  should be (optionally) specified in the retry configuration so that
  different destinations can be given different concurrency limits.

- the retry file should allow hostnames for tcpsmtp driver targets,
  though this might require the PTR to be looked up for every address
  being tried (or maybe if there are no matching IP#s then we fall back
  to looking up the target name from which we resolved the IP we're
  about to try connection to? -- problem is this doesn't match the retry
  file name and we may have an API layering problem getting that name)

- the retry file should allow for CIDR subnets.  Currently the 'tcpsmtp'
  transport driver calls retry_host_lock() with a hostname that is
  concocted by converting the target IP address to a string with
  inet_ntoa().  Perhaps we could implement this by testing the hostname
  to see if it can be converted by inet_aton(), and if so then in
  match_retry_domain() instead of looking through the retry table with
  is_suffix(), use match_ip_net() instead.

- maybe the retry file should also allow for hostname REs too?  If so
  then are they all REs (how do we maintain the leading dot
  compatability?) or should we just use the quoting trick ala aliases?

- Normally we don't want to restrict users who are not using our mail
  server to relay their outgoing messages but rather only to receive
  (and probably re-route remotely again) their incoming messages
  (i.e. virtual domain users).  Such users will not usually be using
  clients listed in smtp_remote_allow.  For finer grained
  smtp_local_sender_restrict control we should be able to tell the
  difference between an address routed via $hostnames or $more_hostnames
  and one routed via some outside router like one using the 'rewrite'
  driver.  Maybe we need a full list of domains for which we do
  anti-spoof checks.  In the mean time if you host virtual domains like
  this then you'd best disable smtp_local_sender_restrict.

- try to include the "ORIG-TO:" field in the logs when a message bounces
  (i.e. in the "Failed" log entry) -- otherwise it's almost impossible
  to see what the input address was.

- think about how to include the "ORIG-TO:" field from the "Delivered"
  log entry in the received header -- for SMTP this should be the
  original envelope recipient address for this particular delivery (what
  do we do when a delivery has collapsed multiple addresses when
  avoiding duplicate delivery?  It looks like only the "first" will
  appear in the log, whatever "first" means....)

- properly fix all the other director drivers to have RE-capable prefix
  and suffix attributes (like what was done recently for the user driver)

- think about adding a verify_command to the pipe transport driver so
  that VRFY and RCPT TO: commands, as well as '-bv', can actually check
  if a delivery will succeed.  This is primarily most useful when local
  delivery is done via a pipe driver and where the delivery agent has
  some easy way to report if delivery might fail due to quota violations
  or other problems (eg. with Cyrus IMAP).

- this happens sometimes when multi-homed with multiple IP subnets on
  the same segment and connecting to a peer's "alternat" address....

	07/07/1999 17:47:12: [4931] remote EHLO: questionable operand: 'becoming.weird.com': from root@becoming.weird.com source [204.29.161.180]: Remote address PTR lookup failed (Unknown host).

   This would probably be fixed by always greeting with the name
   matching our actual source address [getsockname() and then
   gethostbyaddr() or getnameinfo()], and would re-invent/remove the
   meaning of 'primary_name'.

- fully support $max_message_size [perhaps also add a new option
  $truncate_oversize_bounce or similar with default ON].

- think about allowing $listen_name to be set on command line too [if
  this is used for more than one domain then you'll need separate config
  files anyay, so just use -C; but if you are using this to avoid having
  SMTP on some interfaces then this info may be easier to manage in one
  place in the /etc/rc* files or whatever].

- do something to make aliasfile parsing identical across lookup protos.
  (related to 'db lookup parser' bug above?)

- Put the following in default.c for SVR4's local, pipe, & file transports:

	remove_header="Content-Length",
	append_header="${if !header:Content-Type :Content-Type: text}",
	append_header="Content-Length: $body_size",

- think about how to integrate checkerr and savelog so that security
  violations can be snarfed from logfile just after it is cycled.
  Perhaps a new over-all maintenance script (smailmaint?) could do the
  work and there would only be one crontab entry necessary.  Note that
  there's no need to use the antiquated savelog on systems that have a
  newsyslog(1) capable of not compressing the .0 file (eg. my version!).
  [syslog logging would also change all of this since then security
  violations will get higher priority from syslog if the admin so
  desires...]

- add an "always" attribute to the directors drivers, esp. aliasfile.

- add 'senders' and 'senders_except' attributes to directors and routers
  to implement restricted aliases, transports, etc.

- think about allowing hostnames in match_ip() by doing a reverse lookup
  on the address and matching the resulting PTR(s) with any hostname
  patterns [regex's too, or just glob(3), or just domain suffixes?].
  Remember to always do the safe thing when no PTR is found --
  i.e. return a code saying that a test was not possible (either
  temporary error indicator if DNS times out, or permanent if
  authoritative NXDOMAIN) and let the calling code can do the "safe"
  thing (eg. reject a relay attempt).

- think about making smtp_remote_allow and other users of match_ip() and
  match_re_list() capable of specifying a file lookup mechanism in a
  list element:

       smtp_remote_allow="localnet:10/8:192.168/16:\
		${lookup:sender_host_addr:ipsearch{
				/etc/smail/remote.allow}:$value}"

  where "ipsearch" iterates the [new] match_ip() function over all the
  values in the file.  (does this mean keeping the double compare?)
  (the file should probably be cached in-core and treated as a list if
  it's not too big).  See next item too about how to specify the
  variable containing the value being searched for instead of magically
  knowing what it is as in the above example.

- think about adding a new magic variable name like "key" that the
  caller of expand_string() can set in a dummy addr structure so that
  expansion of ${lookup in a list-style variable can be done reliably
  without having to know what variable is used for matching (eg. with
  $sender_host_addr in the previous item and $sender in the next one)

- think about fixing parsing of all list-style variables so that they
  can all always optionally include an element that is run through
  expand_string(), in particular so that ${lookup can be used.  The
  first trick here is in making sure there's some way to always specify
  the value being searched for in the list, and making sure each
  variable's definition documents the expected lookup variable(s).  The
  second trick is making sure the expansion results in something useful
  so that the caller makes sense of the lookup result.  For hostname and
  IP# lookups this should be as simple as expanding the the searched-for
  key if the lookup succeeds, or the key prefixed by '!' if not.

  + eg. this would allow collapsing smtp_sender_reject and
    smtp_sender_reject_db into just the former.  In this particular
    example the caller would have to arrange to have the expected lookup
    variable set to the appropriate value:

	smtp_sender_reject:".*@[^@]*\\.localdomain;bogus domain!:\
		${lookup:sender:lsearch{dead-mail.senders}
		then {$sender;$value}
		else {!$sender}"

- think about adding 'DNS' and 'RDNS' db search protocols for ${lookup.

- Think about splitting lsearch and USE_LSEARCH_REGEXCMP into a plain
  old lsearch and a new "research" (is this a bad name? ;-) [JPR
  suggests "grep", how how about "grepsearch"?] for straight RE linear
  searches.  Think about not using double quotes to trigger the RE match
  in "grepsearch", but rather doing it for every key value.  Think about
  a combined lsearch+grepsearch that would do what lsearch+REGEXCMP does
  now with the double-quote trigger, but maybe also not require the
  double-quote trigger.

- adjust the error messages in config file parsing to include at least
  the line number, and anything else helpful, not just:

	05/07/1997 15:40:59: /local/etc/smail/config: parse error: unexpected end of attribute

- think about adding eqic{, ltic{, gtic{ operators that unify the case
  of their arguments before testing.

- think about changing the "var" portion of the eq{ et al operators to
  be a fully expanded value, not just a variable name (which would make
  the eqic{ et al operators suggested above effectively redundant).

- add support for Kiem-Phong Vo <kpv@research.att.com> Vmalloc library,
  particularly debugging support [partly done].  Also add hooks to build
  with sfio (i.e. without the stdio layer).

- document ${eval: if it turns out to be useful anywhere but with -bP.

- re-write aliasfile.c in the style of the fwdfile.c with a finish_*()
  function, etc.

- think about allowing multiple recipients at RCPT_CMD time for SMTP
  bounce messages but then denying them at the DATA phase.  This may
  cause a lame sender to retry the bounce ad-infinitum though....

- add someone's regex library to pd/regex (Ozan's?, PRCE?) and use that
  if the code's not ported to the current system's equivalent (or always
  use it?)

- make the startup log message more verbose (version, build, build date,
  release date, etc.) [use $smtp_banner ???]

- figure out how to do the configuration for per-transport (or
  even per-target?) relaying control.

- pass a flag to fill_attributes() so that it can print a more
  meaningful error message that indicates if an unknown attribute is
  expected to be either a generic attribute, or a driver-specific
  attribute (possibly either the word "generic" or the driver name).

- the error message string returned by parse_header() doesn't indicate
  which header line the problem was with, never mind which specific
  address in the case of address parsing problems.

- implement optional $max_mailbox_size [optionally as a colon separated
  list of "user=size" tokens with something like '*' as the default user
  and "nolimit" to unset per user].  Or use ${lookup}?

- add configurable reserve space for spooldirs ($min_spooldir_free?)

- be careful about never filling the logfile too (can we instantly defer
  connections if we're out of resources like this?)

- try to ensure all variables are run through expand_string().

- add a way to supress warnings for smtp_helo_broken_allow.

- Microsoft IDIOTS:

  220 exchange1.ACC.WORKFORCE.COM Microsoft ESMTP MAIL Service, Version: 5.0.2195.1600 ready at  Tue, 6 Feb 2001 13:05:01 -0800 
  EHLO proven.weird.com
  250-exchange1.ACC.WORKFORCE.COM Hello [204.92.254.15]
  250-TURN
  250-ATRN
  250-SIZE
  250-ETRN
  250-PIPELINING
  250-DSN
  250-ENHANCEDSTATUSCODES
  250-8bitmime
  250-BINARYMIME
  250-CHUNKING
  250-VRFY				# it's _NOT_ optional!
  250-X-EXPS GSSAPI NTLM LOGIN
  250-X-EXPS=LOGIN
  250-AUTH GSSAPI NTLM LOGIN
  250-AUTH=LOGIN			# bogus -- it's just "AUTH"
  250-XEXCH50
  250-X-LINK2STATE
  250 OK				# what's this BS!?!?!?
  quit
  221 2.0.0 exchange1.ACC.WORKFORCE.COM Service closing transmission channel

- checkerr should maybe try to find the original message-id for double
  bounces and look for related log entries for it too, then we could see
  right in its report the original source of the failing message.

- checkerr should pre-parse DEAD_MAIL_ERE_FILE so that flags ala Postfix
  regexp_table(5) can be used, or maybe we need an enhanced version of
  egrep that can take expressions from tables using the Postfix
  regexp_table(5) syntax.  (this is to avoid using 'egrep -i')

- look for more places where xprintf(), dprintf(), and str_printf()
  could use new '%S' (like %*s in printf(3)) could be used.

- add more to the API defined in list.c/list.h and make more use of it.

- think about how to safely and portably typedef uid_t and gid_t
  [autobuild?]

- think about some way to support SysVr3.x's broken SIGCLD [really?].


New Features:
-------------

These are primarily things that should wait for the next major release.

- Implement LMTP support, especially for delivery.

- With LMTP verification should probably go all the way to doing an LMTP
  VRFY command to hopefully get over-quota notification right away.

- Think about a config variable that could (maybe $log_events?) that
  could control which items are logged and which are not [or wait for
  syslog support?]

- think about implementing RFC 1891 (ESMTP DSN support).

- write a minimal mailstats replacement (new log file format only)
  [real stats, not just what logsumm does]

- implement '-R'

     -Rstring	    Go through the  queue  of  pending	mail  and
		    attempt  to	 deliver any message with a reci-
		    pient containing the specified string.   This
		    is useful for clearing out mail directed to a
		    machine which has been down for awhile.

- implement ETRN from RFC 1985 ala the above (patch already available,
  but needs some performance enhancements and support for '-R').

- implement other standards-track SMTP extensions....

- possible make the daemon children change their ps command line text to
  show what they are currently doing (on systems where this is possible)

- teach substitute() to recognize the variable names listed in
  conf_attributes, etc.(?)


Miscellaneous:
--------------------

- remove nested includes from routers/bind.h and transports/tcpsmtp.h

- investigate this weird log message fragment:

	ORIG-ID:<199604230758.AA13625@post.tandem.com\POS,$ZNET^U5>

  (possibly related: what'll happen if a message-ID header has other
  crap, and even continued lines, in it too?)

- install ".so" (soelim) manual pages with their full longer names on
  systems with longnames (do we detect this feature dynamically, or do
  we rely on a configuration item?) [need to fix up xrefs too?]

- we should add IsValid*() checking?  from:
  <URL:ftp://ftp.cert.org/pub/cert_advisories/CA-96.04.corrupt_info_from_servers>
  [one place this should be done is in addr.c:check_target_and_remainder()]

- maybe we shouldn't allow trailing dots in HELO/EHLO greeting name,
  except for "localhost." of course.

- read RFC-2821 and RFC-2822 even more carefully.

- think about not stripping comments from aliases, etc., and providing
  GCOS info; esp. for EXPN and VRFY, perhaps re-using smtp_info to
  control.

- Should the "real_user" director set ignore_alias_match?

- consider allowing multiple whitespace characters to act as one when
  speparating words in a string parsed by expand_string().

- think about the possible benefits of having separate DBG_DRIVER types
  for each of the different kinds of drivers (router, director,
  transport).

- clean up the duplication between COPY_STRING() and copy().

- clean up remaining use of deprecated index(), strchr(), bcopy(), whatever.

- an interval of '1y' prints as '52w1d5h45m36s'.