regularia implementatia projectia

    ________________________________________some mainlines
   |                                        |
   $  allow setting explicit dollar in env, e.g. {$type}{roman}
   #  many built in tests will not equate {} and \{ and \}.
   -  %: import trees?
   -  let cannot distinguish error from bounce.
   -  stress test with embedded \0's: nulbyte.azm
   -  \:!: syntax is pretty horrid; introduce \:{!} instead ?
   -  let portability of isnan, isfinite. on CWI, <ieeefp.h>
   -  chaining apply/sort/grep              |  [_stagem_]
   $  let precision, -0.0, 10**100          |
   $  is \, really needed?                  |
   -  macro packages should not depend on zoem builtin macros.
   -  apply different keys to same data, tag apply-keys
   -  automate running regresstion/stress tests
   $  enable customization of \<> syntax    |
   -  ifdef{marker}{\done} logic in zmm files
   -  other encodings for '\\' '{' and '}' for better unprotecting/protecting.
   -  html unit type, html/troff units; design
   -  separate TODO in man-faq-doc/core, cleanse it.
   -  \write{key}{foo}{device}{foobarzut} / or KV interface
   KV rethink interfaces, why not more key-val args? [def_expand] [overload vararg]
   X  interpolation skips \!{} scope        |
   X  inspect does not support \' syntax in callback
   ?  preparse macro definitions            |
   ?  inline file syntax: \:{foo}           |
   ?  register#3, tied to sink; register exported macros
   ?  allow \10, \11? varargs alleviate most of that
   #  extensions are ugly.
   ## enable setting primitives aside in own namespace. \'while{}
   ## and allow zoem primitives to be overwritten (but warn).
   ## \:/ line-count screw-up               |
   #  zoem weakness: escape character. no length-encoded data.
   |________________________________________|


    ________________________________________glorious syntax
   |                                        |
   |  \a-zA-Z_       primitive/user syntax  |
   |  \$             dictionary stacks      |
   |  \"xy::12+@#"   user key               |
   |  \'             primitive namespace    |
   |  \:             comment                |
   |----------------------------------------|
   |  \-             dash                   |
   |  \|             newline                |
   |  \~             nbsp                   |
   |  \\  \{   \}    glyphs                 |
   |  \*{ }          glyphs syntax          |
   |----------------------------------------|
   |  \<             xml etc                |
   |  \%{ }          data                   |
   |----------------------------------------|
   |  \!             delay                  |
   |  \@{  }         at scope               |
   |  \&{  }         and scope              |
   |  \`   `         formatted              |
   |  \={foo}        inline files           |
   |  \,             nothing                |
   |________________________________________|




!/ redo format implemtation (mcxTingKAppend)

-  \list{
      {join}{:}
      {apply}{_#1{}}
      {split}{-}
      {grep}{_#1{}}     \: inspect ?
   }{data}


#
---->8------>8------->8------>8------->8------>8---
huhuhu
\write{\__fnout__}{copy}{testtesttest}
hahaha
   concatenates testtesttesthahaha; device filter does not.
---->8------>8------->8------>8------->8------>8---
-  The copy filter does not update the state to
   be no longer BOL.
-  Note that this is funny given \__fnout__.


\apply{_#2{\!{\1=\2\@{\N}}}}{
   \inspect{{mods}{posix,discard-nmp}}{([^-])}{_#1{{\1}}}{a-b-c-d}
}

.
----------------------------------------
{a}{b}{c}{d}



-  by default do not do ws squashing - it's more pud than zoem.

-  document let/f/fv precision behaviour.




-  format:
      ~{<pat>}{{left}{right}}

$? precision_g: currently relative to absolute value.
   Now 0.00000009 == 0.00000001.
   (it's currently meant for *basic* stuff).

-  let: when does -0.0 occur? \let{0.99999999 -1 }


-  document ctr.zmm somewhere.
-  allow '>' in \<\foo>; somehow manage to map it to &gt;.
?  \%#3 as callback

-  mod_initialize composites as tings (so no need to create/delete them)


def_expand
-  if def is expanded, it will be like this:
      \def{{mode}{bcwxz}}{foo}{bar}
         bottom
         conditional (if not set)
         warn (if set)
         expand
         z allow masking zoem primitives

-  how to do pipes with system: invoke shell !?

_stagem_
-  how about chaining applies? [then also sort, grep]
   See examples/loop.azm, and consider textmap approach.

#  it is currently possible to translate \\ and \{ and \} ? #yes
#  tr translates \@{} as well, e.g. \tr{{from}{@}{to}{F}}{\@{a}}

?  register BEGIN_FILE, END_FILE, to be run at beginning and end of each input
   file.  (with __file__ set to file name)

KV dofile; scope for more control as well as abolishing old interface.

KV def/set/defx/setx are candidates as described further
   below.

KV write
-  would diversions be useful?  this is perhaps just
      \write{key}{signature}{filter}{data}
   or better
      \write{{type}{key}{ident}{signature}{filter}{name}}{data}

KV tildex/protect/vanish/"" are candidates.

?  tildex#2 primitive?
   \tildex{{mode}{uzr}}{data}

-  what is the status of \writeto and environments?
?  how about enforcing well-formed-ness only at the filter level?
-  yamKeyGet does not return a const Ting. Why not?
?  This would work by inserting special at-scope tokens.
-  how about allowing \$"foobar" ?
-  add -P flag to add paths to searchpath.  -I already taken.
?  -s, setx not supported. -sx
-  reserve \-- and \--- similar to TeX. But with other bindings.
!  \text{foo}
-  traceput c argument is funny. better (symbolic) interface needed.
#  apparently formatted skips \<> scope.
-  \while{\inspect{}} will only work in boolean mode.
?  need to reset special level when \special{} is issued (check)?
?  seg 0 stack -1 ?
-  tracing: at base file level, add line numbers to keys.
#  \@{\S} is meaningful even with \@{\w}.
-  can I remove table#5? mm not entirely replaceable by apply.
#  writeto does not work in interactive mode.
?  is there a use for stuff like substr, pos ?
-  inline files may cause premature exit (no recovery).
#  zinsert *does* strip comments.
#  \def{%%{foo}}{3} does sth.  (it sets \%{%{foo}}).

#  in stdia, writeto{foo} does not work (but it gives the impression
   it does). Except that \writeto{\__fnout__}{copy}{bar} works.

## inspect
   stop-iter-on-match    then what with the other blocks? no good.
   se-and-copy           apply for side-effects and output
                         -> nonsense, side effects passed in result text.

#  inspect#2 bug which is hard to fix:
   \inspect{posix{\\b}{()}}{ab cd ef gh}
   \b matches; pointer is set to next char; \b matches again.

?? some way to associate env with \<> syntax. automatic counters etc.

?  how about 'x' operator to aid with tables?  \x{-}{30}
   dangerous name though.

-  implement linked lists in special1 with grim code (filter.c)

?  what about filtering strings? good for anything?
   would best be integrated with byte-syntax.
   it would require some syntax for selecting and deselecting special tables.

-  do not reset (push/pop) line count in stdia.

o  does a #0 key also enter the while loop in which \k's are sought
   and checked?

/  make explicit framework for opening/writing/appending files,
   both design and interface.
   currently: indexed by file path name.

-  \zinsert{}, \finsert{} syntax to include a single line.
   [and iterating? dont do what perl does]

-  not a good way yet to join a list. apply with if yes, but cumbersome.
   (_stagem_)

#  \write{stdout}{device}{enter something\@{\N> }}
      writes spurious newline *if* -o - is used, otherwise ok.
   Understandable: newline in input is simply flushed, fixed by
   escaping or removing those newlines.

-  nobody outside segment.c should need yamSegFree.
   but apply2 is a special case .... parts of it should better be
   encapsulated by segment.c and parse.c

-  this should fail ?
   ||| at scope in and scope:
   ||| \@{\&{\foo}}
   abcfoo_start{ foo_at }foo_enddef
   simply says unknown escape '@' ..

#! xml syntactic sugar, SEGMENT_CONSTANT
   I've tried to screw it up a couple of times, so far no luck.
   Does it really behave exactly like the rest of zoem wrt delay and nested digest?
   [errors cannot be caught by \catch#2, that's one difference]


-  unicode input: UCS-16 UCS-32
      + fgetwc
      -> need data structure support
   unicode input: utf-8
      + fgetc
      -> need nothing, can do nothing
   zoem unicode encoding; \#{402} \#{x192}
      -> need output hook mechanism, either string mapping (e.g. entities)
         or putwc-like thingy.
   zoem byte encoding: \'{32} \'{x20}

-  formatted or device scope: sth to specify that only leading
   whitespace should be removed ?
   \`^`

#  \catch{error}{abc\write{\__fnout__}{device}{\<zut> hih \</foo>}def}
   results (understandably) in 'hih abc'

!  format: sth to specify no padding at right side for centered
   and substring alignment.  (to remove trailing spaces)

!  long term: possible to unify all the character-iterating code?
   checking escapes, jumping blocks etc.  (assemble into a single file)
      yamProtect
      yamUnprotect
      yamReadData
      yamLength

!/ \@{\W} is needed for indenting, but also implies squashing.
   indenting and squashing pbb need to be decoupled.
   what are reasonable rules?  do I stack indent as well????
   apparently yes.  is there a difference between device and plain scope?

?  some syntax to remember previously opened file name.
   e.g. with clmformat the first output could be used
   as log file.

-  if \writeto{..} is given before anything is output, the
   default output should not be opened, perhaps. but that
   would require some ugly check on the key index ...

at_scope
-  if you put \@{\w} at the beginning of line, what happens with
   stacked newlines, stacked spaces?

-  how about optifying \<> to variant \@{<} .. \@{>} rather than
   \@{< .. >}.

   then \<\zut> (\zut -> '>') would automatically get mapped to <&gt;>
   simply by character filtering.  [would there be a problem with delay scope??]
   [basic problem; mixing different character translations and scopes
   is annoyingly difficult; perhaps the first stage X has to utilize
   e.g. the \@{..} second stage syntax].

-  can urlencode simply be done using inspect + predefined grape keys?
   register a grape space for zoem primitive use; e.g. __
   assumption[first '?' must not be encoded] anything else?
,  how about \urlencode primitive.
   httpref would just use that, and it would neatly localize everything.
   urlencoding is context dependent: first '?' denotes the search part (always?)
   how to discern which & -> &amp; ; which & -> %26
   user must do the &amp; (or #38; for that matter) part.

#  yamInputIncrLc; could parse.c know by itself from the seg
   whether to call it justly or not?
   Right now incrlc checks the txt against hd[idx].txt,
   to see whether we are at the outer level in a regularly
   processed file. Presumably parse.c has no way of checking
   this; a fresh seg linked list is created for each digest.

#  what happens when inline files simply don't end ?
   zoem (p)errs ok but the line number is funny; it pertains
   to the last finished chunk.

-  \inspect{{mods}{posix}}{(([a-z])*)}{_#2{\!textmap{{word}{ucase}}{\1}}}{ab234}
   does not work: no atom warning. is this a posix feature?

#  \system{perl}{{-e}{'print "hi\\n";'}} fails
   \system{perl}{{-e}{print "hi\\n";}} works. shell quote hell.
   \system{exec}{{date}} fails; (shell builtin, not a system binary)

!  more primitives could act like '%': being able to take variable
   number of arguments.
   it would be an additional property of a primitive.

   BUT the parser has to know this; for '%' it knows by virtue of
   the first character.

   So, such primitives would e.g. have to share some prefix. like '%'.
   \%system{a}{b}{c}

   OTOH, grape keys could have been coded to just use a vararg.



-  NOTE parsing stuffs everything in arg1_g;
   %#1 then *again* extract arguments.
   this is so that the macro location mechanism need not be adapted.
   (we can simply search for %#1)

-  xml sugar:
   make '<' and '>' yamSpecial tokens (259 and 260), so that the
   functionality can be supported for other formats too.
   this requires some additional rules which are not yet clear.

-  interactive:
   \| and \~ etc are ignored ??

!  how does nested \zinsert{stdia} work?
   it actually seems to work, but what is happening with fp's meanwhile?

mqmq


?  at directive: don't write newlines (shorthand for cumbersome
   translate expression)?

apply-keys
-  what when i want to apply different keys to the same data?
   currently difficult wrt backslash i believe.

   \def{foo#1}{[\1]}
   \def{bar#1}{{\1}}
   \def{zut#1}{<\1>}
   \apply{_#1{\!\1{a}\|}}{{foo}{bar}{zut}}
   does not work because after expansion apply uses
   anon key containing '\\1'

 ! \apply{_#1{\1{a}\|}}{{\!foo}}
 ! *does* work

   \def{$foo#1}{[\1]}
   \def{$bar#1}{{\1}}
   \def{$zut#1}{<\1>}
   \apply{_#1{\!$\1{a}\|}}{{foo}{bar}{zut}}
   works; apply uses
   \$\1

   \def{foo#1}{[\1]}
   \def{far#1}{{\1}}
   \def{fut#1}{<\1>}
   \apply{_#1{\!f\1{a}\|}}{{oo}{ar}{ut}}
   works also.

   dotfoo
   A conceivable feature 
   is that during parsing
   \.foo is simply interpreted as \foo
   this would enable callbacks, e.g. for shipping typeface information:
   bf, it, tt etc.
   \apply{_#1{\!.\1{a}\|}}{{foo}{bar}{zut}}
   but it is seems too special purpose to be good.
   \def{bf#1}{bf[\1]}
   \def{it#1}{it[\1]}
   \def{st#2}{\apply{_#1{\!{}\1{\2}\|}}{{foo}{bar}{zut}}}

   nuther solution is to make cb_bf, cb_it, cb_tt keys that map
   to bf, it, tt.

-  can it be useful to keep track in yamfilterdata, the state in
   which the last character was output (at or plain)?
   This pertains also to the squashing problems and to the \S\| issue.

-  how about a '\register#2{name}{descr}' command, so that it's easy
   to find all macro's supported by a package.
   .zmm files are not parsed by -i flag. perhaps

   zoem -r man.zmm
   should list all registered macro's in man.zmm



}~{_}~{_}~{_}~{_}~{_}~{_}~{_}~{_}~{_}~{_}~{_}~{_}~{_}~{_}~{_}~{_}~{_}~{

   -  A very OOOLD todo. nobody knows what it means anymore,
      but it is pretty deep.
   \(@array)
   \(%hash)
         evaluating expressions will require splicing into the string
         in the previous segment, which is equivalent with inserting
         a segment below and fiddling with ofsets.

}~{_}~{_}~{_}~{_}~{_}~{_}~{_}~{_}~{_}~{_}~{_}~{_}~{_}~{_}~{_}~{_}~{_}~{