The Unity library: developer notes
==================================


This document contains brief notes on the structure of the Unity
library, aimed at developers who wish to add new functionality (for
example, a new language), _or_ who wish to build the source from a
checkout (which is intricate enough, that it probably counts as a
‘developer’-ish task).

The Unity library (currently) supports four grammar variants and two
implementation languages (C and Java).

The grammar in `src/grammar/unity.y` is a compendium grammar for the
four grammar variants, and is language-agnostic.  It is not, as it
stands, a valid yacc input file The grammar variants are set off from
each other using the various `@` flags, and the file has
language-specific material inserved into it using the `@!` flags found
in the rule action parts.  These flags are interpreted by a small
filter in `src/grammar/tools/stripat.c`.  The flags are documented at
the start of that program, but their meaning is probably fairly clear
from the use in this file.  The master grammar file is processed by
stripat to produce language- and variant-specific yacc input files,
which are subsequently compiled by the makefiles in the
language-specific parts of the tree.

The bulk of the tests are in a precisely-formatted CSV file in
`src/grammar/testcases.csv`; see the top of the file for documentation.
This file is pre-processed in the same way as `unity.y`, to produce four
variant-specific test files, which are interpreted by the programs run
by the 'make check' targets in `src/c` and `src/java`.

The build is admittedly rather intricate (OK, I confess, it's a bit
insane), with a certain amount of makefile magic.  It could possibly
be simplified.  However the main complicating annoyances are (i) that
the `src/java` and `src/c` builds have internal dependencies on ../grammar
because they need the `unity-<lang>.y` grammar files for building and
`testcases-*.csv` for testing; and (ii) the Java dependencies are
incompletely expressed.

It's useful to include `-Wall` in the CFLAGS when running .`/configure`.
To run the C tests with valgrind, call

    % make RUNTESTS='/path/to/valgrind --leak-check=yes ./runtests' check

The Java version has more extensive test coverage than the C version.

If a memory allocation fails, the library aborts rather than trying to
recover.  The C library does not have large memory demands: if it runs
out of memory, something quite major is wrong.


Building from a repository checkout
-----------------------------------

To build from a source checkout, cloned from
<https://heptapod.host/nxg/unity> (Mercurial),
you need to download, or have installed, rather more.

Building _might_ be as simple as:

    % ./bootstrap
    % ./configure PATH=path/to/byaccj:$PATH  # adjust path to suit
    % mvn dependency:copy-dependencies
    % ./configure PATH=path/to/byaccj:$PATH
    % make check

(the first `./configure` is to create the `pom.xml` which the `mvn` command relies on).

If you use [Nixpkgs](https://github.com/NixOS/nixpkgs) (which is a
Good Thing), then you can set up most of the dependencies with
`nix-shell setup.nix` (but at the time of writing, `byaccj` isn't
included there, so you'll have to download it yourself).

Tools:

  * autoconf
  * [byaccj][], plus optionally bison, plus flex or lex
  * Java, specifically JDK 8
  * [doxygen][] and [graphviz][] if you wish to build the C documentation

Make sure these are all in the path before configuring, for example by setting
PATH as one of the `./configure` arguments).  Byaccj is required to
generate the Java parsers; it will also work for generating the C
parsers, if bison happens not to be present.

The code is currently intended to be built using JDK 8.  Porting it to
a more recent would be reasonable on the face of it, but we want to be
able to support potentially quite old Java versions, so this would
need some careful planning.

Some of the source code is generated using a Java program,
and therefore you do need Java (and possibly `byaccj`) to be present,
even if you only want to build the C library.

Specific packages suggestions:

  * FreeBSD 13: install `pkg` packages
    `autoconf byaccj maven graphviz doxygen gmake zip`
    (and dependencies) and build using `gmake`.
  * Debian 12: install `apt` packages
    `autoconf byacc-j flex maven graphviz doxygen make`
    and either `gcc` or `clang`.
  * Alma Linux 9: install `dnf` packages
    `autoconf flex maven`, and either `gcc` or `clang`;
    byaccj, Mercurial (!) and doxygen must be installed by hand.

Java dependencies:

  * [jflex][]
  * [JUnit4][]
  * [Mulgara MRG][]

These dependencies can most conveniently be obtained using Maven
(`% mvn dependency:copy-dependencies`);
the Java dependencies must live in `<build-directory>/lib`.

Doxygen is optional:
if it is not in the PATH, the C documentation will be skipped.

_Note on installing byaccj_: after downloading [byaccj][], (1) tweak
its Makefile to remove macOS-specific compiler and build options (even
on macOS), (2) do `make yacc`, and finally (3) install the result as
`byaccj`, by hand, in some location which is findable in the path at
`./configure` time.



Making a release
================

_The notes and instructions in this section are (probably) for Norman only_
(and yes, I'm sure heptapod/gitlab could automate these):

  * Possibly update the version number in `configure.ac`
    – is the ‘likely next release’ there the actual one (eg, 1.2 rather than 1.1.1)?
    The date is taken from the repository tag.
  * Check that the `RELEASE-NOTES.txt` file is up to date
  * Check in, and `hg tag` the revision with the version number in the
    release.  This may be the same as that in configure.ac (eg, `1.1`), or with a
    beta identifier appended (eg, `1.1-b1`) -- see [semver.org][].
  * Hg update (back) to this tag.
  * `% autoconf; ./config.status --recheck && ./config.status && make clean && make check && make webpage-tarball.tar`
  * `cd $(DIST); ./configure && make check` (just to check everything's bundled up correctly)
  * If everything is OK, push to the repository (otherwise rollback, fix, etc...)
  * Update to the tip, again.
  * If this was a release version (as opposed to a beta), then
    increment the version number in `configure.ac` to the likely next release
    (eg `1.1` -> `1.2`);
    add another blank stanza for the likely next release (full or beta) to `RELEASE_NOTES.txt`;
    and check in.

Package upload
--------------

To make a package, and distribute it on heptapod, see the
[documentatation there](https://heptapod.host/help/user/packages/package_registry/index).

Distribute (eg) `unity-1.1-b1.zip`, `unity-1.1-b1.jar` and the
javadocs/doxygen bundles (inside `webpage-tarball.tar`) by uploading to the
[heptapod package registry](https://heptapod.host/nxg/unity/-/packages).

It's necessary first to create a [personal access token at
heptapod](https://heptapod.host/help/user/profile/personal_access_tokens.md).
Save that token, `FOO`, in a file `token` with content `private-token:FOO`.
That can have a fairly short lifetime, and simply giving it general `api` access
is fine in this case.  Note that Unity has heptapod project number 1283.

    % curl --header @token \
           --upload-file unity-1.1-b1.zip \
           https://heptapod.host/api/v4/projects/1283/packages/generic/unity-dist/1.1-b1/unity-src-1.1-b1.zip
    % curl --header @token \
           --upload-file unity-1.1-b1.jar \
           https://heptapod.host/api/v4/projects/1283/packages/generic/unity-dist/1.1-b1/unity-1.1-b1.jar
    % curl --header @token \
           --upload-file unity-1.1-b1-javadocs.zip \
           https://heptapod.host/api/v4/projects/1283/packages/generic/unity-dist/1.1-b1/unity-javadocs-1.1-b1.zip
    % curl --header @token \
           --upload-file unity-1.1-b1-doxygen.zip \
           https://heptapod.host/api/v4/projects/1283/packages/generic/unity-dist/1.1-b1/unity-cdocs-1.1-b1.zip

These PUTs create a ‘package’ `unity-dist/1.1-b1`.

Distribution – making a heptapod ‘release’
------------

On the [heptapod release page](https://heptapod.host/nxg/unity/-/releases):

  * create a new release
  * select the appropriate tag and add release notes
  * create link: to the ‘package’ created above – eg
    `https://heptapod.host/nxg/unity/-/packages/3` labelled eg
    ‘Package files: distributed sources and jarfile’, marked as type ‘Package’
    (would it be better to link individual src and jar files in here?)

Optionally, unpack `webpage-tarball.tar` on distribution page
(<https://nxg.me.uk/dist/unity/>), and upload  `$(DIST).tar.gz` and
`$(DIST).jar` to the same place.

The VOUnits standard document requires the presence of
`src/grammar/unity-grammars.zip`, built using the Makefile in that directory.



Other random remarks
====================

Below here are rather random remarks which I don't want to completely
forget, but which are unlikely to be of very broad interest.

Standards
---------
Relevant pseudo-standards docs:

  * IAU style manual, section 5.1, 1989
        https://www.iau.org/static/publications/stylemanual1989.pdf
  * OGIP memo OGIP/93-001, 1993
        https://heasarc.gsfc.nasa.gov/docs/heasarc/ofwg/docs/general/ogip_93_001/
  * Standards for Astronomical Catalogues, Version 2.0, section 3.2, 2000
        https://vizier.u-strasbg.fr/vizier/doc/catstd-3.2.htx
  * FITS, section 4.3; see v3.0 (W.D. Pence et al., A&A 524, A42, 2010
        doi:10.1051/0004-6361/201015362) and v4.0
        https://fits.gsfc.nasa.gov/fits_standard.html (all)

Standards documents:

  * BIPM SI brochure
        http://www.bipm.org/utils/common/pdf/si_brochure_8_en.pdf
  * IEC kibibit standard:  IEC 60027-2: 2005, third edition,
        Letter symbols to be used in electrical technology – Part 2:
        Telecommunications and electronics.  See also ISO/IEC 80000-13, which
	'ancels and replaces subclauses 3.8 and 3.9 of IEC 60027-2:2005'.
  * ISO 80000.  Part 1 has general remarks, including discussion of formatting
        units, and lists the kibi-multiples, with reference to part 13.

Use of solidus: the CDS document is ambiguous; but the OGIP and IAU
styles flatly contradict each other on how many solidi can be used in
an expression: OGIP says they apply only to the following unit, IAU that
there can be only one; this matches BIPM, who say (Sect 5.1) "[a] solidus
must not be used more than once in a given expression without brackets
to remove ambiguities".  They also disagree about whether '.' or ' ' is
used for multiplication.

Sebastien's IVOA Naples notes about units:
<http://www.ivoa.net/internal/IVOA/InteropMay2011Semantics/VOUnits_Semantics.pdf>

The unit-manipulation library UDUNITS has various interesting features,
and is available at <http://www.unidata.ucar.edu/software/udunits/>.  It
supports reading and writing units (in a non-standardised syntax), and
some operations for defining new units, and doing arithmetic on them.


Java class-path warnings
------------------------

It _may_ be necessary in some circumstances to build src/java/unity.jar with

    cd src/java; make BOOTCLASSPATH='-bootclasspath path/to/rt5.jar' unity.jar

But this is probably unnecessary in versions later than JDK1.5.



Software used
-------------

This library incorporates code to implement the
au.com.bytecode.opencsv package, from opencsv.sourceforge.net.  There
are no local modifications to that code (see file
src/grammar/tools/Notes for discussion).  That library is Copyright 2005
Bytecode Pty Ltd, and is distributed with an Apache-2.0 licence.



Other flex/yacc-like systems
----------------------------

  * http://www.antlr.org/
    (big multi-language system, produces parsers with runtimes)
  * http://byaccj.sourceforge.net/
    Port (more-or-less) of Berkeley YACC to Java.  Standalone parser
  * http://www2.cs.tum.edu/projects/cup/
    Parser-generator for Java
  * http://jflex.de/
    Lex for Java, intended to work with CUP and BYacc/J
  * http://www.dabeaz.com/ply/
    Python Lex-Yacc
  * http://www.freenet.org.nz/python/pybison/
    Another Python Bison-a-like




Norman Gray  
<https://nxg.me.uk>


[byaccj]:	http://byaccj.sourceforge.net/
[doxygen]:	http://www.doxygen.org
[graphviz]:	http://www.graphviz.org/
[jflex]:	http://jflex.de/
[JUnit4]:	http://junit.org
[Mulgara MRG]:	http://code.google.com/p/mrg/
[semver.org]:	https://semver.org