regularia implementatia projectia

0  reduce this ridiculous TODO

FF far future (thinking)
cc corner case
R  robustness
N  note
j  jot down a thought
j? jot down a thought, but when would it be useful/needed?
u  useful feature (needs eg: example)
?  huh? what was I thinking?
w  documentation


    ________________________________________some mainlines
   |                                        |
   -  better fixed/adaptable layout for manual, index.html (aephea)
   -  import file1 -> finsert file2 does not work, finserted stuff is still under import restriction.
      -> (came up in zpresenter macros)
   -  \begin{}{{#2}{a}} *silently* overwrites primitive.
  FF  Inhume 0-terminated strings in tingea/ting
   -  provision for string/input validation (e.g. fib example, number)
   -  product of array leads to integer overflow; should however change to double. (let library).
   w  zinsert/finsert different rules for file inclusion (do not use directory stack), different fail modes. 
  cc  interactive \| and \~ etc are ignored?
  FF  preparse macro definitions?
   j  begin: a way to check whether user set a key.
   j  'export' and documentation mode for def for tight coupling with documentation. 
      -> {doc}{doc-string} in set#3, would register documentation, accessible as \%{__doc__}{par#1}
      \register#2{name}{descr} to find all macro's supported by a package.
      -> zoem -r man.zmm could list all registered macro's in man.zmm
   j  length: strip leading, trailing white space.
   ?  writepush, writepop that remember last stream?
   j  allow \''$foo syntax ?
   j  should failed set be fatal? (yes; probably refers to named dictionary: needs-eg)
   j  \$__xargs__ is expanded; may lead to double expansion?
   j? syntax to set special level to previous value.
   w  document io/sink/filter framework
   w  \write{bar}{device}{\inform{zut}} overwrites __fnwrite__ (ok warning)
   -? further organize the filter code      |
   -  zoem should only enter interactive if isatty(stdout, stdin)
   ?  ability to create/access (by prefix) new stacks?
   N  apply creates all the key invocations and concatenates them (rather than iterating) -- useful when it is needed, if wasteful.
   ?  traverse-tree is easy to write: now generalize tree data?
   w  enter_interactive vs sourceStdia      |
   w  document ctr.zmm somewhere.
   w  vararg parsing: unary argument {} delimited argument requires {{}}.
   j  filter.[ch], espcially at ' ' and '\n' stuff.
      -> fixmefixmefixme -> indent is printed after newline regardless of what follows.
      -> note n_spaces means 'number of stacked spaces'. We do not know the number of *printed* spaces.
   N  precision_g: currently relative to absolute value.
      -> Now 0.00000009 == 0.00000001.  (it's currently meant for *basic* stuff).
   N  \system{perl}{{-e}{'print "hi\\n";'}} fails
      \system{perl}{{-e}{print "hi\\n";}}
   FF xml sugar:
      make '<' and '>' yamSpecial tokens (259 and 260), so that the
      functionality can be supported for other formats too.
      this requires some additional rules which are not yet clear.
   R  - document/consider what happens with \write{foo}{device}{\eval{\zinsert{tim}}}
      where the file tim issues \include.   |
      - document/resolve why finsert does not take inline files.
      - generally research nested inclusions and prevent/catch/alert weirdness
   j  something to push special definitions and then pop the exact same set (possibly useful for format#2).
      ->-> translate constant definitions at interpretation to at sequences?
  FF  something to strip device scope from copy filter: should see just (semantic) content?
      ->-> for \*{} sequences take key rather than value.
   ?  for extensibility: a facility to match an argument against a certain block structure.
   -  have f and fv supplied by util/let.h. OR DROP THEM COMPLETELY.
      -> document let/f/fv precision behaviour.
   -  \while{\inspect{}} will only work in boolean mode.
   w  code-doc SEGMENT_CONSTANT SEGMENT_DIGEST
   ?  format#2: explicit control over phase of padding.
   ->-> with more arguments: cycle through padding strings. mcxTingRoll().
   ?  format#2: temporary mapping of character set!?
   ?  format#2: left-padding and right padding
   ?  format#2: typeset in a box.           |
   j  investigate pipes with system
   ?  format#2: ability to store/reuse parsed format strings (clmformat)
   ?  format#2 or roff: column support      |
  cc  unprotecting-tr-protecting '\foo\@{\N}' should do what exactly?
   -  let precision, -0.0, 10**100          |
   -  formatted specifier to remove ^\s*    |
 ? !  \set{{diag}{NOOOOO}}{foo}{bar} (customized diagnostic)
   /  audit format implementation for mcxTingKAppend
   $  audit all mini and special syntax parsers. inspect#2, formatted#1.
   -  let portability of isnan, isfinite.   |
   ?  parsescopes {{a}b{c}} gives blocks a and b{c}. used somewhere?
   $? inspect mode where offset is always incremented by 1 or user-specified.
   $? introspection: stack/segment nummer, file nesting depth .. \reflect#x
   $? query internals (dump keys on pattern)|
   $? enable customization of \<> syntax    |
   $? implement \%free in \undef or \set{{free}{data,tree,node}}{%{foo}}{}
   $? implement character filtering by finite state machine driver
   $? implement grep.                       |
   ?  a \__xxxx__ unique global tag for the current dictionary (concat of names)
 ------                                     |
   $  comprehensively analyze/categorize '\\' '{' and '}' encoding issues
   $  silent overflow with overly long %{1}{2}..{} sequences
   $  audit atoi code, integer overflows .. |
 ------                                     |
   #  allow setting explicit dollar in env, e.g. {$type}{roman}
   #  \write{key}{foobarzut} / or KV interface / or just append mechanism.
   #  \:!: syntax is pretty horrid; introduce \:{!} \:{/} instead.
   #  \set{{{if|unless}{\let{\c>1}}}}{foo}{bar}
   #  format evaluate vararg val.           |
   #  \done should quit current file whatever chunk it is: extra status bit?
   #  parse.c case '+' -- what does it do? A: filter level
   #  \set{a}{-1} \let{\a+1} yields 0.0 (improved simplistic punctuation check)
   #  \''foo for straight access to global user keys.
   #  many built in tests will not equate {}|and \{ and \}.
   X  interpolation skips \!{} scope        |
   X  inspect does not support \', \'' syntax in callback
   ?  inline file syntax: \:{foo}           |
   ?  register#3, tied to sink; register exported macros
   ?  allow \10, \11? varargs alleviate most|of that
   #  hide filter* struct in filter.c.      |
   #  extensions are ugly.                  |
   #  use checkusrsig in entry.c, -s option.|
   ## enable setting primitives aside in own|namespace. \'while{}
   ## and allow zoem primitives to be overwritten (but warn).
   ## formatted \` syntax is pretty ugly.   |
   ## \:/ line-count screw-up               |
   ## missing \end{foo} not reported.
   |________________________________________|


    ________________________________________glorious syntax
   |                                        |
   |  \a-zA-Z_       primitive/user syntax  |
   |  \$             dictionary stacks      |
   |  \"xy::12+@#"   user key               |
   |  \'             primitive namespace    |
   |  \''            global user namespace  |
   |  \:             comment                |
   |  \:{..}         special comment        |
   |----------------------------------------|
   |  \-             dash glyph             |
   |  \|             newline glyph          |
   |  \~             non breaking space     |
   |  \\  \{   \}    glyph literals         |
   |  \*{ }          glyph syntax           |
   |----------------------------------------|
   |  \<             xml etc                |
   |  \%{ }          data                   |
   |----------------------------------------|
   |  \!             delay                  |
   |  \@{  }         at scope               |
   |  \+{k}          filter level           |
   |  \&{  }         and scope              |
   |  \`{  }         formatted              |
   |  \={foo}        inline files           |
   |  \,             nothing                |
   |----------------------------------------|
   |  \`  `          deprecated             |
   |  \*  *          deprecated             |
   |________________________________________|


---->8------>8------->8------>8------->8------>8---
huhuhu
\write{\__fnout__}{copy}{testtesttest}
hahaha
  concatenates testtesttesthahaha; device filter does not.
---->8------>8------->8------>8------->8------>8---
-  Note that this is weird given \__fnout__ is the current output stream
*  The copy filter does not update the state to be no longer BOL; general
   issue of filter processing + output stream interactions.

-  mod_initialize composites as tings (so no need to create/delete them)

#  it is currently possible to translate \\ and \{ and \} ? #yes
#  tr translates \@{} as well, e.g. \tr{{from}{@}{to}{F}}{\@{a}}

?  register BEGIN_FILE, END_FILE, to be run at beginning and end of each input
   file.  (with __file__ set to file name)

?  tildex#2 primitive?
   \tildex{{mode}{uzr}}{data}

-  what is the status of \writeto and environments?
?  how about enforcing well-formed-ness only at the filter level?
-  yamKeyGet does not return a const Ting. Why not?
?  This would work by inserting special at-scope tokens.
-  how about allowing \$"foobar" ?
-  add -P flag to add paths to searchpath.  -I already taken.
?  -s, setx not supported. -sx
-  reserve \-- and \--- similar to TeX. But with other bindings.
!  \text{foo}
-  traceput c argument is funny. better (symbolic) interface needed.
#  apparently formatted skips \<> scope.
?  need to reset special level when \special{} is issued (check)?
?  seg 0 stack -1 ?
-  tracing: at base file level, add line numbers to keys.
#  \@{\S} is meaningful even with \@{\w}.
-  can I remove table#5? mm not entirely replaceable by apply.
#  writeto does not work in interactive mode.
?  is there a use for stuff like substr, pos ?
-  inline files may cause premature exit (no recovery).
#  zinsert *does* strip comments.
#  \def{%%{foo}}{3} does sth.  (it sets \%{%{foo}}).

#  in stdia, writeto{foo} does not work (but it gives the impression
   it does). Except that \writeto{\__fnout__}{copy}{bar} works.

## inspect
   stop-iter-on-match    then what with the other blocks? no good.
   se-and-copy           apply for side-effects and output
                         -> nonsense, side effects passed in result text.

#  inspect#2 bug which is hard to fix:
   \inspect{posix{\\b}{()}}{ab cd ef gh}
   \b matches; pointer is set to next char; \b matches again.

?? some way to associate env with \<> syntax. automatic counters etc.

-  implement linked lists in special1 with grim code (filter.c)

?  what about filtering strings? good for anything?
   would best be integrated with byte-syntax.
   it would require some syntax for selecting and deselecting special tables.

-  do not reset (push/pop) line count in stdia.

o  does a #0 key also enter the while loop in which \k's are sought
   and checked?

/  make explicit framework for opening/writing/appending files,
   both design and interface.
   currently: indexed by file path name.

-  \zinsert{}, \finsert{} syntax to include a single line.
   [and iterating? dont do what perl does]

#  \write{stdout}{device}{enter something\@{\N> }}
      writes spurious newline *if* -o - is used, otherwise ok.
   Understandable: newline in input is simply flushed, fixed by
   escaping or removing those newlines.

-  nobody outside segment.c should need yamSegFree.
   but apply2 is a special case .... parts of it should better be
   encapsulated by segment.c and parse.c

-  this should fail ?
   ||| at scope in and scope:
   ||| \@{\&{\foo}}
   abcfoo_start{ foo_at }foo_enddef
   simply says unknown escape '@' ..

#! xml syntactic sugar, SEGMENT_CONSTANT
   I've tried to screw it up a couple of times, so far no luck.
   Does it really behave exactly like the rest of zoem wrt delay and nested digest?
   [errors cannot be caught by \catch#2, that's one difference]


-  unicode input: UCS-16 UCS-32
      + fgetwc
      -> need data structure support
   unicode input: utf-8
      + fgetc
      -> need nothing, can do nothing
   zoem unicode encoding; \#{402} \#{x192}
      -> need output hook mechanism, either string mapping (e.g. entities)
         or putwc-like thingy.

#  \catch{error}{abc\write{\__fnout__}{device}{\<zut> hih \</foo>}def}
   results (understandably) in 'hih abc'

!  format: sth to specify no padding at right side for centered
   and substring alignment.  (to remove trailing spaces)

!  long term: possible to unify all the character-iterating code?
   checking escapes, jumping blocks etc.  (assemble into a single file)
      yamProtect
      yamUnprotect
      yamReadData
      yamLength

!/ \@{\W} is needed for indenting, but also implies squashing.
   indenting and squashing pbb need to be decoupled.
   what are reasonable rules?  do I stack indent as well????
   apparently yes.  is there a difference between device and plain scope?

?  some syntax to remember previously opened file name.
   e.g. with clmformat the first output could be used
   as log file.

-  if \writeto{..} is given before anything is output, the
   default output should not be opened, perhaps. but that
   would require some ugly check on the key index ...

at_scope
-  if you put \@{\w} at the beginning of line, what happens with
   stacked newlines, stacked spaces?

-  how about optifying \<> to variant \@{<} .. \@{>} rather than
   \@{< .. >}.

   then \<\zut> (\zut -> '>') would automatically get mapped to <&gt;>
   simply by character filtering.  [would there be a problem with delay scope??]
   [basic problem; mixing different character translations and scopes
   is annoyingly difficult; perhaps the first stage X has to utilize
   e.g. the \@{..} second stage syntax].

-  can urlencode simply be done using inspect + predefined grape keys?
   register a grape space for zoem primitive use; e.g. __
   assumption[first '?' must not be encoded] anything else?
,  how about \urlencode primitive.
   httpref would just use that, and it would neatly localize everything.
   urlencoding is context dependent: first '?' denotes the search part (always?)
   how to discern which & -> &amp; ; which & -> %26
   user must do the &amp; (or #38; for that matter) part.
   http://stackoverflow.com/questions/275150/xhtml-and-ampersand-encoding

#  yamInputIncrLc; could parse.c know by itself from the seg
   whether to call it justly or not?
   Right now incrlc checks the txt against hd[idx].txt,
   to see whether we are at the outer level in a regularly
   processed file. Presumably parse.c has no way of checking
   this; a fresh seg linked list is created for each digest.

#  what happens when inline files simply don't end ?
   zoem (p)errs ok but the line number is funny; it pertains
   to the last finished chunk.

-  \inspect{{mods}{posix}}{(([a-z])*)}{_#2{\!textmap{{word}{ucase}}{\1}}}{ab234}
   does not work: no atom warning. is this a posix feature?

!  more primitives could act like '%': being able to take variable
   number of arguments.
   it would be an additional property of a primitive.

   BUT the parser has to know this; for '%' it knows by virtue of
   the first character.

   So, such primitives would e.g. have to share some prefix. like '%'.
   \%system{a}{b}{c}

   OTOH, grape keys could have been coded to just use a vararg.

   NOTE parsing stuffs everything in arg1_g;
   %#1 then *again* extract arguments.
   this is so that the macro location mechanism need not be adapted.
   (we can simply search for %#1)

!  how does nested \zinsert{stdia} work?
   it actually seems to work, but what is happening with fp's meanwhile?

?  at directive: don't write newlines (shorthand for cumbersome
   translate expression)?

-  can it be useful to keep track in yamfilterdata, the state in
   which the last character was output (at or plain)?
   This pertains also to the squashing problems and to the \S\| issue.