Changes since Pike 8.0.610 (release 12)

New Features
------------

o Sql.mysql

  In unicode decode mode, characters outside the BMP are now
  encoded as UTF-8 encoded UTF-16. This works around that
  MySQL/MariaDB don't allow characters outside the BMP in utf8
  strings (and instead require the character set to be declared
  as utf8mb4, which has quite a few side effects).


Bug fixes
---------

o Runtime

  - Fixed NULL dereference (SIGSEGV) in _disable_threads() if
    another thread is holding the compiler lock.

  - Fixed loss of Pike_interpreter_pointer (race, --with-debug only).

    The Pike_interpreter_pointer must not be altered by threads not
    holding the interpreter lock...

o Calendar

  - Updated tzdata to 2018e.

o Compiler

  - The machine code generator for ppc64 now generates correct code
    under ABI v2 (ppc64le)

  - Fixed an incompatibility between the machine code generator on
    ppc and GCC 7.

  - Support inherit of CompilerEnvironment.

    Fix some bugs triggered in the supporter subsystem when
    executing in an inherited CompilerEnviroment.

o Filesystem.Monitor

  - Fix NULL-dereference in bump().

    Survive st being zero in bump().

o Fuse

  - Switched the Fuse module to use a different libfuse API.

    The way we used to use libfuse got first deprecated and support
    for it is now removed from the libfuse git repository entirely.
    In addition, our previous use of said libfuse API triggered a
    bug resulting in a segmentation fault in at least libfuse 2.9.2
    and 2.9.7. We now use libfuse in a less elegant but more usual
    way, and do not trigger the bug anymore.

  - Handle errors in Fuse operations (i.e. Pike code implementing
    FUSE filesystems) gently by reporting ENOSYS instead of
    exiting the process.

o Gmp.mpq

  Fix numbers between -1.0 and 1.0 sometimes missing a leading 0.

o Pike.identify_cycle

  Fix various issues with LFUNs throwing errors.

  LFUNs used by mapping operations may throw errors; this caused
  identify_loop_visit_leave() to also throw errors. The rest of
  the identify_cycle code was not happy about this and

  - Forgot to unlock the mc_mutex. This caused all following calls
    to Pike.identify_cycle() and Pike.count_memory() to hang.

  - Leaked memory.

  The above issues are now avoided by instead using the addresses
  of objects as indices in the affected mapping. It also fixes
  the issue with hangs if there are objects with lfuns calling
  Pike.identify_cycle() or Pike.count_memory().

o Protocols.HTTP.Server

  Fix an issue with accounting of sent bytes that caused responses
  to requests over HTTPS to not finish on success, but instead the
  whole connection to time out. Data for the first request was
  usually fully written, but the connection neither was properly
  closed nor re-used.

o Protocols.WebSocket

  - Added low_connect() and low_websocket_accept().

    These make it easier to test the error handling in
    the module, and to replace the HTTP-layer.

  - Improved compliance with RFC 6455 section 4.1.

    o The nonce is now actually random.

    o The http version, method and headers are validated.

    o Header case-sensitivity issues have been fixed.

  - The module now has a testsuite.

  - Improved HTTP-header camel-case consistency.

o Sass

  - Errors thrown by handle_sass_import() can now be caught and
    handled by the caller of compile_file() and compile_string().

  - Added the option "sass_syntax".

    This option will parse the input as having indented Sass
    syntax. This option only has effect in compile_string().

  - Added method sass2scss() to convert Sass syntax to SCSS syntax.

o Sql.pgsql

  - Fix regression bug since rel 12 that hangs queries returning over 1024
    records.

  - Fix all rare-deadlock-causes logged in the last two years.

o Stdio.Buffer

  Eliminate race condition in read_cstring().

o Stdio.UDP

  Fixed wait() failing with ENOTSOCK on NT.

o Thread.Farm

  - Fixed lost error in provide_error() exception handling.

  - Avoid delayed deallocation of jobs by clearing references immediately
    after execution.

o Tools.Standalone

  - extract_autodoc: Terminate on failure.

    Reenable termination on extraction failure.


Building & Tools
----------------

o Java

  - The location of libjvm is now detected on Ubuntu 18.04.

  - Detect and support OpenJDK 10.

o Postgres

  Support Postgresql 10 and later.

o Sass

  Improved detection of C++ runtime library on MacOS X.


Changes since Pike 8.0.498 (release 11)
----------------------------------------------------------------------

New Features
------------

o Crypto.Hash.SCRAM

  New class to support SCRAM authentication.

o Concurrent

  - Add map_with() as an alias to flat_map().

  - Add a convenience variant to zip().

  - depend() can be used to collect multiple dependencies before
    finalising the Promise.

o Debug.find_all_clones()

  New convenience function to find all objects that are clones of a
  program.

o MIME

  Added a setter for the message boundary prefix.

o Sql.pgsql

  - Added support for PostgreSQL 10's stronger password authentication
    (SCRAM-SHA-256).

  - Slightly reduced overhead.

  - Support IEEE binary float database-wireformat.

  - Support NUMERIC database-wireformat.

  - Prevent new queries from starting in the local_backend.

  - Make sure huge queries are not fully loaded into memory.

o Sql.Sql

  Make big_typed_query() gracefully fallback to big_query().

o SSL.File

  Support query_fd() and set_buffer_mode() methods.

o Stdio.File [NT]

  A major clean-up and consistency improvment of the I/O layer on NT.

  - All filesystem paths are now encoded in UTF-8 from Pike-code's
    point of view. Previously this was a mix of Latin-1 and UTF-16,
    with some stuff affected by the filesystem default character set.

    This makes the entire filesystem accessible from Pike-code.

    This new behavior can be detected by the presence of the constant
    Stdio.__HAVE_UTF8_FS__.

  - Added new low-level fd handling functions.

    This should fix several potential race-conditions in the
    I/O layer on NT.

o Stdio.Terminfo

  Support the new terminfo format from ncurses 6.

o Thread.Farm

  Added Thread.Farm()->set_thread_name_cb() to help applications
  that monitor thread creation and termination for the purpose
  of tracking thread names.

o Thread.ResourceCount

  New module to allow threadsafe race-condition-free counting of
  resources.

o Tools.Standalone.check_http

  - Allow redirect responses.

  - Support --expect option.

o Web.Sass

  This is a module that interfaces to the SASS compilation
  library "LibSass" (https://github.com/sass/libsass).


Bug fixes
---------

o ADT.CritBit

  - Due to an off-by-one error the last limb of bignums was never
    compared.  This resulted (sometimes) in unequal bignums comparing
    equal inside of the tree.

o Compiler

  - Improved variant robustness.

  - Fixed type leak for getters/setters implemented with
    variant functions.

o Calendar

  - Updated tzdata to 2018d.

  - Calendar.TimeRanges.NullTimeRange now has a working __hash().
    The expression ([ Calendar.TimeRanges.nulltimerange : 17 ])
    no longer gives a backtrace.

o Concurrent

  - Less filling, deadlock-free, faster, available without running
    backend.

  - Handle an empty argument array properly in results().

    Previously code such as:

      Concurrent.Future f2 = Concurrent.results(({}));

      f2->on_success(lambda(array(string) a) {
          werror("success: %O.\n", a);
        })
        ->on_failure(lambda(mixed err) {
            werror (describe_backtrace(err));
          });

    would lead to a backtrace because the future was destructed
    prematurely.

  - Readable/understandable documentation.

o Crypto.CCM

  Fixed initialization of the iv.

o Crypto.Hash

  Censor the password argument from backtraces in crypt_hash().

o Filesystem.Monitor

  - Fixed some backend_check rescheduling isusues.

    adjust_monitor() didn't reschedule the backend_check() call_out if
    the adjusted monitor already was at the head of monitor_queue.
    This caused polls to be delayed in some common cases.

    register_path() didn't reschedule the backend_check() call_out.
    If the newly registered monitor ends up at the head of the
    monitor_queue, the backend_check() call_out likely needs to be
    rescheduled.

  - Fixed issue with next_poll being set to zero causing polling every
    second.

  - Fixed indexing the NULL value error when InotifyMonitor was used
    in combination with a filter_file() that doesn't accept
    everything.

  - Fixed incorrect creation of new monitors.

    Monitors were created using monitor() on the top level rather than
    using Monitor::monitor(), which prevented the symlinks' overridden
    DefaultMonitor from setting the correct state on newly created
    monitors.

    We now force a check on the directory monitor instead, which will
    pick up the new file instantly and create the sub monitor.

  - Fixed race condition on deletion.

    This fixes a race that could occur on rapid exists => delete =>
    exists transitions, where the directory monitor never noticed the
    change but the sub monitor was removed from the monitor lookup
    mapping. The directory monitor is now notified on sub-monitor
    release.

  - Added set_stable_time() function.

  - Removed obsolete special case for co_id == 1.

o HTTPLoop:

  Fixed some potential race conditions.

o Image.Image

  - Improved argument checking in apply_curve().  Non-integer array
    values lead to use of uninitialized curve values.

o Image.PVR

  Fixed size check for twiddled images.

o MIME

  Support trailing newline in MIME part epilogues.

o Protocols.HTTP.Query

  - Fixed incoming chunked transfers (timed async works now, chunk
    options are properly 'parsed'). Trailer headers are now correctly
    parsed.

  - Improved handling of multiple headers of the same kind.

  - Fixed a bug where base64 encoded HTTP basic auth tokens were
    encoded in a pretty-printed format including line breaks, which
    lead to malformed HTTP requests on long username and password
    combinations.

  - Do not wait for SSL.File()->close() to complete when called
    from the destruct() callback.

o Protocols.HTTP.Server

  - The header parser exception mode now doesn't immediately throw an
    exception.

  - Attempt to ensure that data is sent before terminating an
    HTTPS connection.

o Protocols.LDAP

  Fixed race condition on initializing LDAPS.

o Search.Database.MySQL

  Fixed issue where blobs didn't get updated after the initial blobs
  got full.

o Search.Utils

  Updated debug string format.

o Sql.pgsql

  - Suppress spurious 'File not open' blurts on stderr when the
    connection is still pending and we inspect the connection object.

  - Signal EOF early before closing the SQL result portal and
    thoroughly cleanup bufcon/stashcount to prevent occasional
    deadlocks on asynchronous object destructions.

  - Suppress rare "access in destructed object" messages.

  - Simplified fundamentally flawed reconnect logic, made failures
    consistent.

  - Repair SSL connect logic.

  - Update documentation.

  - Repair fetch_row_array().

  - Ensure that all database errors result in exceptions (eventually).

  - affected_rows() and status_command_complete() now wait until they
    have a meaningful result to report.

  - Closing the database will wait until all running queries have been
    committed.

  - Report correct number of affected_rows() for INSERTs.

  - Eliminate a rare deadlock when running many simultaneous queries
    on a single filedescriptor.

o mktime/System.TM

  - Make timezone management consistent (especially UTC handling).

  - Fix memory leak in strftime().

o Standards.IIM

  Disabled debug output for unknown segment markers.

o Stdio

  - Restore the FD_SETSIZE on NT to 65536.

    Due to #include file restructuring, the FD_SETSIZE value on NT had
    fallen back to the default (64). This issue has probably affected
    all releases of Pike since 7.9.2.

  - mv() [NT]

    Fixed single byte buffer overrun in mv() on NT.

    A NUL-terminator was written outside a malloced buffer. This
    caused intermittent crashes after unusual use of mv().

    Fixes the crash in [PIKE-90].

  - send_file()

    Improved support for TLS/SSL; wait for TLS handshake to complete
    before calling the sendfile done callback when sending an empty
    string. This fixes the issue where the connection got closed
    before TLS handshaking was completed, confusing the other side.

o Image

  Creating images with mode "grey" now works as documented,
  e.g. Image.Image(10,10,"grey",123);


Optimizations
-------------

o mktime/System.TM

  Considerable speedup and a reduction in codesize.

o Unicode

  Avoid scanning the string in NFC mode if the string is 8-bit,
  as NFC normalization of 8-bit strings is the identity function.


Building & Tools
----------------

o Hilfe

  Fix commands having priority over code. [LysLysKOM 22552693].

o Image.SVG

  Fall back to using the unprefixed pkg-config if the arch-prefixed
  doesn't exist.

o Mysql

  Support MariaDB 10.2 headerfiles.

o export

  - Fixed fix_configure.

    This was broken in multiple ways:

    * Did not consider timestamp on aclocal.m4
    * Did not consider post_modules
    * Did not check configure scripts directly under modules or
      post_modules
    * When fixing modules, it looked for the configure scripts in the
      build directory rather than the source directory
    * Did not use the run_autoconfig wrapper

    This has caused xenofarm to export snapshots with out of date
    configure scripts.

o precompile

  Support precompilation with Pike 8.1 again.

o run_autoconfig

  Fix localdir when relative and not "./"

o smartlink

  Fixed multiple issues on Darwin & NetBSD.

o Whitefish

  Use the standard VPATH.

o Yp

  Support using libtirpc. Glibc 2.26 has removed the Sun RPC
  implementation.


Changes since Pike 8.0.466 (release 10)
----------------------------------------------------------------------

New Features
------------

o Crypto.ECC

  - Added get_point() to Crypto.ECC.Curve.

  - Crypto.ECC.Curve.Point objects can now be compared with equal().

o JWK

  - Added Web.encode_jwk().

  - Added jwk() to signatures (Nettle.Sign, Crypo.RSA, Crypto.ECC.Curve.ECDSA)

  - Added jwk() to Crypto.Hash.HMAC.

  - decode_jwk() on a ECC private key now returns an ECDSA object (and a
    Point object for an ECC public key).

o Standards.PKCS

  Support for decoding PKCS #10 public keys through
  Standard.PKCS.parse_public_key.

Bug fixes
---------

o Protocols.DNS

  Added hooks for controlling error handling in Protocols.DNS.server.

o Protocols.HTTP

  Fixed a memory leak in the HTTP header parser upon receiving
  malformed HTTP headers.

o Protocols.WebSocket

  Fixed a case where the onopen callback would get the wrong argument.

o Search

  Fixes some broken SQL queries causing data corruption.

o Sql.sqlite

  list_fields() now throws an error when used on a non-existent table.

o Stdio.Buffer

  - Some methods (add_int*, input_from, add_padding) did not trigger
    a call to the fd output function. When using an Stdio.Buffer instance
    in Stdio.File buffer mode, this could result in new data not being
    written.

  - Due to an error in the realloc strategy, in some cases unread() did
    not succeed after reading or writing data to a fd.

o Threads

  Fixed a race condition when creating a new thread while threads
  are disabled.

o WhiteFish

  - Fixed issues with merging of empty sets.

  - Fixed rank calculation on intersection of sets.

  - Fixed multiple memory leaks.

  - Improved behavior when provided with corrupted input.

  - The module is now tested by the normal testsuite.

Compiler
--------

o Improved syntax error recovery, including eliminating some fatal
  conditions.

o Fixed a memory leak in the preprocessor.

Runtime
-------

o Fixed NULL-dereference by the GC when running count_memory() on a
  Pike compiled with --with-dmalloc.

o Fixed fatal "Didn't find gc marker as expected" at exit on a Pike
  compiled with --with-cleanup-on-exit.

o Signal handling

  Fixed an error where processes would share the same queue for
  pending signals after using fork().  As a result, a signal sent to
  one process could trigger a signal handler in the other process.

Building & Tools
----------------

o Configuration

  Fixes to the detection of MariaDB, MySQL and Inotify.

o mkpackage

  Fixed issue where the error code from extra platform tests wasn't
  propagated all the way to exit from the install script.


Changes since Pike 8.0.438 (release 9)
----------------------------------------------------------------------

New Features
------------

o Concurrent

  The Concurrent module simplifies asynchronous code by synchronizing
  events in different ways. As an example the connect() function shown
  below will respond with a Concurrent.Future object that at some point
  will represent either a connected socket or a failure.

    Concurrent.Future connect(string host, int port)
    {
      Stdio.File con = Stdio.File();
      Concurrent.Promise p = Concurrent.Promise();
      if( !con->async_connect(host, port, lambda(int success)
        {
          if(success)
            p->success(con);
          else
            p->failure("Failed to connect to "+host+":"+port+"\n");
        }))
      {
        p->failure("Failed to open socket.\n");
      }
      return p->future();
    }

  The returned future can then be used in various ways.

    // On success, call make_request(con, query). On failure call
    // werror(msg).
    connect(host, port)
      ->on_failure(werror)
      ->on_success(make_request, query);

    // On success, call make_request(con, query1) followed by
    // make_request(resp, query2), where resp is the return value from
    // make_reqest.
    connect(host, port)
      ->then(make_request, werror, query1)
      ->then(make_request, werror, query2);

    // Call bridge_ports(con1, con2) when both connections are
    // established.
    Concurrent.all(connect(host1, port1), connect(host2, port2))
      ->then(bridge_ports, failure);

    // Call make_request(con) once either of the connections are
    // established.
    Concurrent.race(connect(host1, port1), connect(host2, port2))
      ->then(make_requet, query);

o Debug.Inspect

  Allows for interactive debugging and live data structure inspection
  in both single- and multi-threaded programs.

  Example:
  In the program you'd like to inspect, insert the following one-liner:
      Debug.Inspect("/tmp/test.pike");

  Then start the program and keep it running.
  Next you create a /tmp/test.pike with the following content:
      void create() {
        werror("Only once per modification of test.pike\n");
      }

      int main() {
        werror("This will run every iteration\n");
        werror("By returning 1 here, we disable the stacktrace dumps\n");
        return 0;
      }

      void destroy() {
        werror("destroy() runs just as often as create()\n");
      }

  Whenever you edit /tmp/test.pike, it will automatically reload
  the file.

o Protocols.DNS.server

  Derived classes can now override report_decode_error() and
  handle_decode_error() to change how errors while decoding a DNS
  packet are reported and handled.

Bug fixes
---------

o ADT.Heap

  Fixed heap corruption when the same object is pushed more than
  once. It will now be considered as calling adjust().

o FSEvents

  EventStreamMonitor now works with other backends.

o Filesystem.Monitor

  - Reduced initialization latency.

  - Changed the polling heuristic.

    This is intended to reduce poll (and notification) delays in
    some circumstances.

  - Improved robostness of adjust_monitor().

  - Removed some dead and obsolete code.

  - Scan accelerated monitors too.

    Both the Inotify and FSEvents APIs claim that they support
    notifications on eg network file systems, and while they will
    succeed in notifying on changes performed by the local host
    on such file systems, they will not on changes performed by
    other hosts. To avoid missing such changes these monitors
    need to be actively scanned too.

  - Fixed lost acceleration with InotifyMonitor

    Recover from race-condition where when a file was repeatedly
    created and deleted, acceleration was lost.

  - Fixed symlink issue with InotifyMonitor.

  - Added support for more fine-grained tracing of monitors.

o Inotify

  Addressed an issue where the backend might be stuck in pending
  indefinitely.

o mappings

  Fixed an off by one error in random(mapping) that randomly caused
  values of type PIKE_T_FREE from the freelist to be exposed to Pike.

o Nettle.Hash

  Fixed NULL-dereference in Nettle.Hash()->crypt_hash().

o Parser.HTML

  Fixed a condition where Pike would run out of stack space for large
  documents.

o Search

  Fixed a race condition when updating the database.

o Sql.pgsql

  - Sped up BEGIN/COMMIT statements.

  - Preserve the initial error message in case of multiple error messages
    during the same transaction.

  - Flush out unseen error messages upon connection close to stderr.

  - Closed a prepared-statement-cache race when the same statement
    is offered multiple times before it finalises the cache entry.

  - Fix sync errors with the database in case of multiple running
    statements on a single connection that generate multiple errors.

  - Eliminate spurious sync errors on pipe-lined transactions
    (multiple transactions in flight on a single file descriptor).

  - CancelRequests (to cancel running queries) caused sync errors with
    the database when multiple queries were in flight.

  - Eliminate transient error messages on database restarts.

o Sql.SQLite

  When using query bindings, strings are now always stored as strings;
  previously, 8-bit strings were stored as binary objects (BLOBs) and
  wide strings were stored as unicode strings, which caused 8-bit strings
  to be unequal (and thus not returned) when used in a standard string query.

  Binary values may be stored as BLOBs using bindings by wrapping the
  binary string in a multiset. For example, to store the value "myBinary"
  as a BLOB value using a binding, use: (<"myBinary">).

o Standards.BSON

  - Fixed incorrect encoding/decoding of Binary data.

  - Bugfixes for the decoding of Binary values of subtype 2, Binary
    values of subtypes above 0x7f and Datetime values with negative
    values.

o Standards.JSON

  Fixed integer overflow in decode() on 64-bit platforms.

o Stdio.Buffer

  - Fixed a crash when attempting to create a rewind key on a buffer
    returned by read_buffer().

  - Fixed integer overflow in read_json().

Building
--------

o Configure

  Attempt to detect the tool prefix depending on ABI.

  Improves detection of pkg-config.

o Java

  Detect system installation of JavaVM framework on MacOS X.

o Stdio

  Improved support for NetBSD.


Changes since Pike 8.0.404 (release 8)
----------------------------------------------------------------------

New Features
------------

o Calendar

  Updated Calendar to use timezonedata from tzdata2017a.

o GL & GLUT

  Prefer Quartz implementation on systems that have it (OS X).

o GTK2

  - Add -x objective-c for Darwin.

  - Support linking against other implementations than X11.

o Protocols.HTTP

  Connecting to HTTP proxies over HTTPS is now supported.

o Web.decode_jwk()

  Support decoding of symmetric (ie HMAC) keys.

o Web.encode_jws() and Web.decode_jws()

  Added simplified API for dealing with JSON Web Signature values.

Bug fixes
---------

o Runtime

  - Variant dispatcher

    Improved string range checking.

    Fixes mismatches due to te lazy string range (ed string(8bit))
    exceeding the typed string range (eg string(7bit)) for narrow strings.

o mkpackage

  Support multiple sub-packages with the same name.

  Gnu-tar doesn't like being asked to extract the same file multiple
  times from the same tar-file. It claims that the second file doesn't
  exist in the archive.

  This can happen when a sub-package has been replaced with a fixed version.

o smartlink

  Don't override MACOS_DEPLOYMENT_TARGET if it's been specified.

  This should prevent "compiled against version x but linking against
  version y" warnings.

o Arg & ADT.Struct

  Survive use by old (ie 8.0.232) pike binaries.

  Roxen 6.0.130 uses a Pike 8.0.232 binary with current modules.

o Backend.PollDeviceBackend

  Make set_signal_event_callback() optional.

  This fixes the warning "An expression of type function({ Backend =
  object(implements _static_modules.Builtin()->defaultBackendClass) } :
  void) cannot be assigned to a variable of type function({ Backend =
  object(implements _static_modules.Builtin()->Backend) } : void).".

o bool

  Make sure that 0 is part of the range for the bool type.

o Charset

  Fixed encoding errors for two characters in the GSM-03.38 charset.

o Debug

  Fixed typo in mask in check_callback_chain().

  Fixes infinite loop in circularity detection.

o Gdbm

  Support building with gdbc 1.13.

  Fixes obscure compilation error when building with gdbm 1.13
  due to a symbol clash with the symbol "fatal" used as an argument
  in the macro definition of gdbm_set_errno().

o Mapping

  Removed extraneous and conflicting prototype.

  Fixes linking issues on NT.

o Protocols.HTTP

  - Fixed do_async_proxied_method() losing the timeout during TLS
    negotiation.

  - Now fails immediately if Query is unable to open a socket, instead
    of waiting 120 seconds for a time out.

  - Allow connection reuse also when the server is a hostname.

o Thread.Mutex

  - Perform a thread yield on mutex onlock in an attempt to reduce
    thread starvation.

o Sql.pgsql

  - Added an extra synchronisation condition, and increased parallelisation
    to resolve all detected race conditions (which in rare circumstances
    (many parallel queries on a single filedescriptor) could have resulted
    in a deadlock).


Changes since Pike 8.0.388 (release 7)
----------------------------------------------------------------------

New Features
------------

o Mysql.SqlTable

  sizeof() on an SqlTable now returns the number of rows in the table.

o Web

  - Updated {en,de}code_jwt() to support HMAC signatures.

  - Added decode_jwk{,_set}(), which decode RFC 7515-style JOSE keys.

Runtime
-------

o Stdio.Buffer & String.Buffer

  Allocated constant program identifiers for both of the above.

Bug fixes
---------

o Protocols.DNS

  The size and precision fields of T_LOC records are now properly
  decoded.

o System.TM

  When a System.TM object was created without any arguments some
  operations would crash on Windows, and possibly other platforms.

Building
--------

o Configure

  Fixed issues with the LDSHARED test on some BSD-derivatives.

  Detect systems that provide bswap16,32,64 in their libc.

o Stdio

  Fixed compilation issue on platforms without readdir_r().


Changes since Pike 8.0.370 (release 6)
----------------------------------------------------------------------

New features
------------

o Thread

  - _sprintf() improvements: Thread.Mutex now prints the ID of the thread
    holding the lock, and thread IDs are shown as hexadecimal numbers.

  - Thread.Farm now supports a callback for thread creation and termination
    for the purpose of tracking thread names.

Bug fixes
---------

o String character range

  Fixed a bug where a string could incorrectly have its character
  range start point set to 0 when added with an empty string. This in
  turn would incorrectly block the string from being used by functions
  that require strings without null characters as argument or
  incorrectly take optimization paths.

o VCDiff

  Fixed type of decode_chunk().

o Parser.XML

  UTF-8 encoded documents with initial BOM were not decoded correctly.

  Serializing Parser.XML.Tree objects will now entity escape
  characters that can not be encoded in the selected output charset.


Changes since Pike 8.0.358 (release 5)
----------------------------------------------------------------------

New features
------------

o Calendar

  Updated Calendar to use timezonedata from tzdata2016j.

o Nettle

  Added version().

o Refdoc

  Updated to new Pike site layout.

Bug fixes
---------

o JSON

  Fixed a bug in the unicode handling of Standards.JSON.decode(), which
  treated codepoints slightly below the surrogate range as invalid.

o pike -x pmar_install

  Works again and properly verifies PMAR checksums.

o ::_indices(), ::_values(), ::_types(), ::`->(), ::`->=()

  Fixed several argument checking bugs in the magic_* functions.

o Fixed some bugs in the pmar installer.

o Msql

  Fixed some compilation issues.

o Sql.pgsql

  Fixed broken reconnect behaviour when the database connection dies.

Runtime
-------

o Added workaround for incompatibility in MS Windows 10's Linux
  emulation layer.


Changes since Pike 8.0.276 (release 4)
----------------------------------------------------------------------

New features
------------

o Calendar

  Updated Calendar to use timezonedata from tzdata2016i.

o Crypto.ECC

  Zero-pad short signature integers.

o Protocols.HTTP.Server.HeaderParser

  It is now possible to enable non-normalization on the entire object,
  and not just on a per call basis. It is also possible to disable
  support for folded headers, as per IETFs latest specification.

o CompilerEnvironment()->lock()

  This is a class that exposes the compiler internal lock.

o Gz.deflate()->clone()

  This allows for testing compression.

o gdb_backtraces()

  gdb_backtraces() is now available even --without-debug.

o ADT.Heap()->low_pop()

  This function is analogous to low_peek().

o Protocols.WebSocket

  Parse query variables in WebSocket Requests.

Bug fixes
---------

o cpp

  Fixed stringify operator on strings containing escaped double quotes.

o mkpackage

  Support $EXTRA_PLATFORM_TEST.

o Threads.Condition

  Fixed double free at cleanup on exit.

o Fixed multiple issues with cleanup on exit.

o Whitefish

  Fixed wrong signedness.

o Standards.EXIF

  Improved robustness of the EXIF parser.

o Protocols.HTTP.Session

  Ignore malformed expiry dates in cookies.

o Protocols.WebSocket

  Fixed handling of truncated frames.

o _Roxen

  Backported multiple fixes from Pike 8.1.

o Compiler

  Fixed broken range optimization.

  Removed optimization that considered the hostname() constant.

  Don't reference count direct cyclic references via mixed variables.

o MasterObject

  Protect against the same file being compiled concurrently in
  multiple threads.

  Cast to program and cast to object should now be thread-safe.

  Survive RESOLV_DEBUG being enabled.

o Stdio.FakeFile

  Improved sendfile compatibility.

o Process.create_process

  Improved behavior under high signal pressure.

o Sql.pgsql

  Improved robustness against stray destructs and exceptions.

o Standards.BSON

  Fixed circular dependency.

o Mappings

  Multiple optimizations in m_delete() and friends.

  Added flag MAPPING_FLAG_NO_SHRINK.

Building
--------

o Nettle

  Since there are distributions that have removed nettle_secp_192r1
  and nettlesecp_224r1 from the nettle library, these curved are now
  disabled by default. Enable them by compiling with
  --with-weak-curves.


Changes since Pike 8.0.240 (release 3)
----------------------------------------------------------------------

New features
------------

o Protocols.HTTP.Server.HeaderParser

  feed() can now be forced not to normalize (ie lower_case)
  the header names.

o Protocols.WebSocket

  Added client mode.


Bug fixes
---------

o Documentation

  Fixed some AutoDoc markup typos.

o Fixed assertion failure in multiset handling.

  Indexing of non-empty multisets containing no objects or functions,
  with an object implementing lfun::`<() et al could cause assertion
  failures.

o Filesystem.Monitor

  Fixed potential recursive mutex locks on destroy().


Changes since Pike 8.0.182 (release 2)
----------------------------------------------------------------------

New features
------------

o Roxen.http_decode_string() now decodes UTF-16 surrogate pairs.

o Protocols.HTTP.Query now always sends a content-length header.

o Report feature flags for System.FSEvents and System.Inotify.

o Updated Calendar to use timezonedata from tzdata2016c

o Added module Apple.Keychain that can extract certificates.

o Made Standards.X509 aware of more OS specific ways of retriving certificates.

o JOSE (JSON Object Signing and Encryption)

  Some low-level API support has been added to the Crypto and Web
  modules to support parts of RFC 7515 - 7520.


Bug fixes
---------

o The machine code generator on ia32 now aligns the stack
  on a 16 byte boundary. This makes it possible to call
  code that uses SSE2 instructions.

o Exceptions thrown in GTK signal handlers no longer cause
  segfaults.

o The SSL client code now interprets the RFCs more leniently to
  address some interoperability issues.

o Fixed bugs in cipher selection that prevented null-ciphers from
  being negotiated by the SSL server (if allowed by the Context
  object).

o Multiple fixes for the handling of accelerated
  Filesystem.Monitor monitors, which could sometimes
  lose notifications.

o Improved handling of response content-length in Protocols.HTTP.Query.

o Fixed case where unfinished programs could lose their parents.

o Fixed "Invalid service"-error from connect() on Fedora Core 4.

o Fixed next_result() in Sql.sql_array_result.

o Fixed sizeof() in Array.Iterator, Multiset.Iterator and
  String.Iterator.


Testing
-------

o Reduced the memory requirements of Tools.Shoot (pike -x benchmark)


Building and installing
-----------------------

o Multiple fixes for building on Solaris.

o Improved scanning for Odbc header files.

o Fixed some issues with --with-abi flags not propagating
  correctly to the modules.


Changes since Pike 8.0.164 (release 1)
----------------------------------------------------------------------

New features
------------

o gc

  It is now possible to run the gc in a quick mode to just cut
  weak references from a specific container.


Bug fixes
---------

o ADT.Heap

  The Element class lost track of its position when remove() was used.

o Filesystem.Monitor

  * Improved thread safety and robustness

  * Inotify-related fixes.

o pgsql

  The new driver in 8.0 suffered from filedescriptor leaks if the
  database connections were not explicitly close()d.
  Fixes have been applied which eliminate the filedescriptor leak and also
  ensure that the filedescriptors are closed the instant the object
  loses all references.

o programs

  The runtime now attempts to wait for any compilations in progress to
  complete before complaining about cloning of unfinished programs.
  This fixes a race-condition when the same program is compiled from
  different threads.

o Runtime

  Fixed multiple potential NULL-dereferences.

o SSL.File

  Fixed breaking of circular references when the peer terminates
  the connection.

o Standards.URI

  Support the userinfo field containing '@'.

o Stdio.File

  The second argument to set_buffer() should now work.

o Testsuite

  Updated the SHA3_* testvectors to FIPS 202.


Building and installing
-----------------------

o Multiple fixes for building on Solaris 11.


Pike 8: Changes since Pike 7.8
----------------------------------------------------------------------

New language features
---------------------

o The compiler now only blocks other threads attempting to use
  the compiler instead of disabling all other threads.

o Added a way to access the local, and not the overloaded,
  implementation of a symbol.

  As an example, given the classes:
    | class Test
    | {
    |     int a( ) { return 1; }
    |     int b( ) { return local::a(); }	// New
    |     int c( ) { return a(); }
    | }
    |
    | class Test2
    | {
    |    inherit Test;
    |    int a() { return 42; }
    | }

  Both Test()->b() and Test2()->b() will return 1, but Test2()->a()
  and Test2()->c() will return 42.

o Added new syntax that can be used to return the current object as if
  it was a class it is inheriting

    The syntax is X::this, where X is the inherited class.

    The returned object will not include any symbols not available in
    the class X, but your overloaded methods (and other identifiers)
    will be the ones returned when the object is indexed.

o Added a way to easily inherit the implementation of a subclass in an
  inheriting class.

    | inherit ::this_program

  The syntax refers to the previous definition of the current class in
  the inherited class, and is typically used with inherit like:

    | inherit Foo;
    |
    | // Override the Bar inherited from Foo.
    | class Bar {
    |    // The new Bar is based on the implementation of Bar in Foo.
    |    inherit ::this_program;
    |    // ...
    | }

o Added new syntax for referring to and naming inherits.

    | inherit "foo.pike";
    | inherit "foo/bar/baz.pike" : "foo/bar";
    | // ...
    |   "foo.pike"::foo();
    |   "foo/bar"::baz();

o Look up of named inherits now also looks in indirect inherits.

    | class A { int fun() {} }
    | class B { inherit A; }
    | class C {
    |   inherit B;
    |   // ...
    |     // The function fun may here be referred to as any of:
    |     B::A::fun();
    |     B::fun();
    |     A::fun();	// New.

o Implemented the variant keyword.

  The keyword was present before, but did not actually do anything
  useful.

  This is used to do function overloading based on the argument types.

  As an example:
    | class Test1 { int b(); }
    | class Test2 { int a(); }
    |
    | class Foo
    | {
    |     variant string foo( object q ){ return "Genericfoo"; }
    |     variant string foo( Test1 x ) { return "Test 1"; }
    |     variant string foo( Test2 x ) { return "Test 2"; }
    |
    |     variant string bar( int(0..0) s ) { return "0"; }
    |     variant float bar( int(1..1) s ) { return 1.0; }
    |     variant int bar( int(2..2) s ) { return 2; }
    | }

  A common use-case is functions that return different types depending
  on the arguments, such as getenv:

    |  string|mapping(string:string) getenv( string|void variable );

  can now be rewritten as

    |  variant string getenv( string variable );
    |  variant mapping(string:string) getenv( );

  which gives significantly better type-checking.

o The type-checker has been rewritten.

  Mainly it is now much better at finding type errors, and most error
  messages are more readable.

o Allow '.' to be used as an index operator in the few places it did
  not work before.

o "Safe" index

  Copied from a popular extension to other C-like languages.

  X[?ind] is equivalent to ((auto _ = X),(_ && _[ind]))
  while X?->ind is equivalent to ((auto _ = X),(_ && _->ind))

  The main use is 'deep indexing' where some elements can be 0:

    | request_id?->user?->misc?->x

  vs

    | request_id && request_id->user && request_id->user->misc
    |  && request_id->user->misc->x

o Added the '?:' operator for compatibility with other C-like
  languages. It is identical to '||' in Pike.

o The && operator changed, when doing A && B, and A is false, keep A
  instead of returning 0.

  This is especially useful then A is not actually 0, but an object
  that evaluates to false, or UNDEFINED.

o Fixed symbol resolution with deep inherits and mixins.

o Added multi-character character constants.

  'ILBM' is equivalent to (('I'<<24)|('L'<<16)|('B'<<8)|'M').

  Unlike how it works in some C compilers the native byte order is
  never relevant.

o Added new syntax for literal-string constants

  #{, #( and #[ starts a literal string, and it is ended by the
  corresponding end marker: #}, #) and #] respectively.

  No character is modified at all inside the literal string, including
  newlines, \ " and '.

  So, the string #["\n\'##] will be equivalent to "\"\\n\\'#".

  The main use-case is to write code in code:

  | string code = #[
  |  void main(int c, array v) {
  |    string x = "";
  |    foreach( v[1..], string elm )
  |      x += reverse(elm)+",";
  |    write("Testing: %s\n", reverse( x ));
  | #];

  The three different start/end markers might be useful if you write
  code in code in code, since there is no way to quote the start/end
  markers.

o Added a shorthand syntax for integer ranges: xbit, where x is a
  number between 1 and 31. This can be used as an example to indicate
  that a string is 8 bits wide only: string(8bit)

  This is equivalent to the range (0..255) but can be easier to parse.

  Similarly int(1bit) is an alias for 'bool', and int(12bit) is the
  same as int(0..4095).

o 'this::x' is now equivalent to 'this_program::x' -- access the
   identifier x in the current object.

o Weak references to stuff with a single reference are now
  removed when the container is resized.

  This means that weak references can be cut without having to
  wait for the full gc to run.


New preprocessor features
-------------------------

o Support for the ", ##__VA_ARGS__" cpp feature.

  This makes the ‘##’ token paste operator have a special meaning when
  placed between a comma and a variable argument. If you write

    | #define err(format, ...) f(debug)werror("ERROR: "+format, ##__VA_ARGS__)

  and the variable argument is left out when the err macro is used,
  then the comma before the ‘##’ will be deleted. This does not happen
  if you pass an empty argument, nor does it happen if the token
  preceding ‘##’ is anything other than a comma.

o The define __COUNTER__ has been added. It is a unique integer value,
  the first time the macro is expanded it will be 1, the next time 2
  etc.

o The preprocessor can now be run with a cpp prefix feature.

  This is currently used by the precompiler to run two levels of
  preprocessing, one using "#cmod_" as the prefix and the other "#".

o Dynamic macros

  You can now add programmatic macros. There is currently no syntax
  that can be used to define these while compiling code, but you can
  add them from one program before compiling plug-ins/modules.

  The main use is macros like DEBUG(...) and IFDEBUG() that would
  expand to something if a debug setting is enabled in the module but
  nothing otherwise, or, to take an actual example from the Opera Mini
  source code:

    | add_predefine( "METRIC()",
    |     lambda( string name, string ... code )
    |     {
    |         string type = type_of( code );
    |         string aggregate = aggregate_of( code );
    |         string format = format_of( code );
    |         code -= ({ 0 });
    |         return replace( base_metric_class,
    |                     (["ID":(string)next_metric_id++,
    |                       "NAME":stringify(name),
    |                       "TYPE":type,
    | 			   "AGGREGATE":aggregate,
    | 			   "FORMAT":format,
    |                       "CODE":make_value_function(code),
    |                    ]));
    |      });


  That is, a macro that needs to do some calculations, and rewrite the
  code more than is possible in normal macros.

  This one expands something along the lines of

    | METRIC("requests", Summarize, PerSecond,
    |        floating_average_diff(requests));

  into

    | class Metric_requests_01
    | {
    |    inherit Server.Metric;
    |    constant name = "transcoder:requests";
    |    constant type = Float;
    |    constant format = PerSecond;
    |    constant aggregate = Summarize;
    |
    |    float value() {
    |       return floating_average_diff(requests);
    |    }
    | }

o Dependency declarations

  It is now possible to use the CPP directive #require to specify a
  condition that must be true for the file to be seen by the resolver.
  This would typically be the inherited C part of a module or a system
  call.

    | #require constant(__WebKit)
    | inherit __WebKit;


Optimizations
-------------

o New significantly faster block allocator

  The free in the old one was O(n^2), which means that as an example
  creating a lot of objects and then free:ing them mainly used CPU in
  the block allocator.

  This fix changed the ordo of one of the tests that did that very
  thing (binarytrees) from O(n^2) to O(n), and as such is more than a
  simple percentual speedup in some cases, but it always improves the
  performance some since the base speed is also faster.

o Power-of-two hashtables are now used for most hashtables

  This speeds up mappings and other hashtables a few percent, and also
  simplifies the code.

o Significantly changed x86-32 and an entirely new AMD64/x86-64
  machine-code compilation backend

  The main feature with the x86-32 edition is that it is now using
  normal function call mechanics, which means that it now works with
  modern GCC:s.

  The x86-64 backends has been rewritten so that it is easier to add
  new instructions (x86-64) and opcodes (Pike) to it, and it is
  significantly more complete than the x86-32 one.

o Svalue type renumbering

  PIKE_T_INT is now type #0, which makes certain things significantly
  faster.

  As an example, clearing of object variables is now done
  automatically when the object is created since all object memory is
  set to 0.

  The same thing happens when clearing values on the stack.

o Svalue type/subtype setting changed

  This halves the number of instructions used to push a value on the
  stack in most cases.

  The speedup is not large, but noticeable.

o And on a related note, we now lie to the compiler about the
  const:ness of the Pike_interpreter_pointer variable.

  This significantly simplifies the code generated by the C-compiler
  for stack manipulation functions, the stack pointer is now only
  loaded once per code block, instead of once per stack operation.

  This saves a lot of code when using the stack multiple times in a
  function, and should be safe enough, albeit somewhat unconventional.

  If nothing else the binary size shrunk by about 5%.

o string(x..y) (constant) types

  The strings now keep track of the min/max values in addition to two
  new flags: all-uppercase and all-lowercase.

    | > typeof("foo");
    | (1) Result: string(102..111)

  This is used to optimize certain operations, lower_case, upper_case,
  search and string_has_null for now. It could be added to other
  places in the future as well.

  A fairly common case is where you are doing lower_case or upper_case
  on an already lower or uppercased string. This is now significantly
  faster.

o Several other optimizations to execution speed has been done

  + object indexing (cache, generic speedups)

  + lower_apply, changes to apply in general

    Taken together these individually small optimizations speeds up at
    least pike -x benchmark more than 5%.

  + A lot of opcodes implemented in machine-code for x86-64

    This speed up the loop benchmarks close to a factor of 3. But then
    again, most real code is nothing like that benchmark.

  + Several new opcodes added

    As an example an observation was that most branch_if_zero is
    followed by an is_eq, is_lt or is_gt or similar. Those opcodes
    always return 1 or 0. So, two new opcodes, quick_branch_if_zero
    and quick_branch_if_non_zero were added that map directly to three
    x86-64 opcodes, there is no need to check the types, do a real
    pop_stack etc.

  + Demacroified some code, resulting in smaller code-size

    This makes things faster, it is often better to branch to a small
    block of code than to have it inlined in a lot of different
    locations on modern architectures.

o Faster hash-function for strings

  If your CPU has the crc32 instruction (modern x86 mainly) the
  hashing is now an order of magnitude faster, albeit the risk of
  attacks using known crc:s in things like http request headers (to
  force long hashtable chains and slow things down) is greater, but:

  + Also added siphash24, and use that when the string hash table
    becomes inefficient.


Deprecated features and modules
-------------------------------

o Tools.PEM and Tools.X409 deprecated

  Use the corresponding modules in Standards.

o The built in security sandbox is now deprecated

  Unless somebody wants to take ownership of the feature and keep it
  up to date the security system will be removed in the next stable
  release.

o The compiler now warns about switch statements where the cases
  aren't enumerable, since these will generate random failures if the
  code is read from a dump.

o strlen() now only accepts strings

o Gdbm.gdbm is now known as Gdbm.DB

o Yabu.db and Yabu.table renamed to Yabu.DB and Yabu.Table

o The keyword 'static' will now generate deprecation warnings.


Removed features and modules
----------------------------

o Removed facets

  The optional (and not enabled by default) support for facet classes
  has been removed, since it was only partially done and not really
  used.

o It is no longer possible to compile Pike without libgmp.

  Bignums are now a required language feature

o The old low-level 'files' module has been renamed to _Stdio

o The old _PGsql helper module has been removed.

o 'GTK' is now GTK2 if GTK2 support is available, earlier it defaulted
  to GTK1 as a preference over GTK2.

o Locale.Charset

  The charset module is now available on the top level as 'Charset'

o The ancient syntax for arrays (string * was an alias for
  array(string)) has now been removed completely.

o Compatibility for Pike versions before 7.6 is no longer available.

o decode_value can no longer decode programs using the 'old style'
  program encoding.

  Since the format has been deprecated since feb 2003, and those
  programs could not be decoded anyway due to other issues it is not
  much of a loss.


New modules
-----------

o Pike.Watchdog

  A Watchdog that ensures that the process is not hung for an extended
  period of time. The definition of 'hung' is: Has not used the
  default backend.

  To use it simply keep an instance of the watchdog around in your
  application:

    | Pike.Watchdog x = Pike.Watchdog( 5 ); // max 5s blocking

  An important and useful side-effect of this class is that the
  process will start to respond to kill -QUIT by printing a lot of
  debug information to stderr, including memory usage, and if Pike is
  compiled with profiling, the CPU used since the last time kill -QUIT
  was called.

o Crypto.Password

  A module that can be used to verify and create passwd/ldap style
  password hashes.

  It tries to support most common password hashing schemes.

o Debug.Profiling

  Tools useful for collecting and format for output the profiling
  information generated when Pike is compiled --with-profiling.

o NetUtils

  This module contains a lot of functions useful for the storage and
  processing of network addresses, it supports IPv4 and IPv6.

o ADT.CritBit

  Mapping-like key-sorted data structures for string, int and
  float-keys (ADT.CritBit.Tree, ADT.CritBit.IntTree,
  ADT.CritBit.FloatTree). Implemented in C.

o Standards.BSON

  A module for working with BSON serialized data. See
  http://bsonspec.org/

o Geography.GeoIP

  Does geolocation of IPv4-numbers using databases from maxmind.com or
  software77.net

o Protocols.WebSocket

  An implementation of the WebSocket (RFC 6455) standard, both server
  and client

o Image.WebP

  Encoder and decoder for the WEBP image format.
  More information about the format can be found on
  https://developers.google.com/speed/webp/

  Requires libwebp.

o Serializer

  APIs useful to simplify serialization and deserialization of objects
  Mainly it allows you to easily iterate over the object variables,
  including the private ones.

  + Serializer.Encodable
    A class that can be inherited to make an object easily
    serializable using encode_value.

o Filesystem.Monitor (and the low level System.Inotify +
   System.FSEvents)

  Basic filesystem monitoring.

  This module is intended to be used for incremental scanning of a
  filesystem.

  Supports FSEvents on MacOS X and Inotify on Linux to provide low
  overhead monitoring; other systems currently use a less efficient
  polling approach.

o Mysql.SqlTable

  This class provides some abstractions on top of an SQL table.

  At the core it is generic and could work with any SQL database, but
  the current implementation is MySQL specific on some points, notably
  the semantics of AUTO_INCREMENT, the quoting method, knowledge about
  column types, and some conversion functions. Hence the location in
  the Mysql module.

  Among other things, this class handles some convenient conversions
  between SQL and Pike data types

o Parser.CSV

  This is a parser for line oriented data that is either comma,
  semi-colon or tab separated. It extends the functionality of the
  Parser.Tabular with some specific functionality related to a header
  and record oriented parsing of huge datasets.

o ZXID

  ZXID is a library that implements SAML 2.0, Liberty ID-WSF 2.0
  and XACML 2.0. Used for single sign-on.

  This module implements a wrapper for ZXID. The interface is similar
  to the C one, but using generally accepted Pike syntax.

o Git

  A module for interacting with the Git distributed version control
  system.

o Val

  This module contains special values used by various modules, e.g. a
  Val.null value used both by Sql and Standards.JSON.

  In many ways these values should be considered constant, but it is
  possible for a program to replace them with extended versions,
  provided they don't break the behavior of the base classes defined
  here. Since there is no good mechanism to handle such extending in
  several steps, Pike libraries should preferably ensure that the base
  classes defined here provide required functionality directly.

o __builtin

  The __builtin module is now a directory module, so that it can
  provide a suitable namespace for code written in Pike intended for
  being inherited from modules written in C (cf precompile).

o Web.Auth

  Support for federated authentication with OAuth (Web.Auth.OAuth and
  Web.Auth.OAuth2). There are several ready to go modules for some big
  OAuth authentication service providers:

  + Web.Auth.Facebook
  + Web.Auth.Github
  + Web.Auth.Google
  + Web.Auth.Instagram
  + Web.Auth.Linkedin
  + Web.Auth.Twitter

o Web.Api

  Modules and classes for communicating with various RESTful web
  api's. Currently with ready to go modules for these services:

  + Web.Api.Facebook
  + Web.Api.Github
  + Web.Api.Google.Analytics
  + Web.Api.Google.Plus
  + Web.Api.Instagram
  + Web.Api.Linkedin
  + Web.Api.Twitter

o VCDiff

  Glue for the open-vcdiff differential compression library. More
  information can be found on http://code.google.com/p/open-vcdiff/


Incompatible changes
--------------------

o Parser.XML.Tree: Fixed handling of namespaced attributes.

  Attribute namespaces starting with "xml" (except for "xmlns") are
  now handled just like any other attribute namespaces. This means
  that eg the attribute "xml:lang" will be expanded to
  "http://www.w3.org/XML/1998/namespacelang" in the mapping returned
  by get_attributes(), and not kept as "xml:lang" when namespace
  handling is enabled.


Extensions and new functions
----------------------------

o Bz2.File added

  It implements a Stdio.File like API, including support for the same
  iterator API that Stdio.File has, allowing for convenient line
  iterations over BZ2 compressed files.

    | foreach( Bz2.File("log.bz2")->line_iterator(); int n; string line )

o Both sscanf and sprintf can now handle binary floats in little
  endian format

  %-8F would be a 64 bit IEEE float binary value in little endian
  order.

o Image.JPEG

  + decode now supports basic CMYK/YCCK support

  + exif_decode is a new function that will rotate the image according
    to exif information

o Image.BMP now supports some more BMP:s.

  + Added support for vertical mirroring (images starting at the
    bottom left)

  + Added support for 32-bit (non-alpha) BMP:s.

o String.Buffer

  It is possible to add sprintf-formatted data to a String.Buffer
  object by calling the sprintf() method. This function works just as
  the normal sprintf(), but writes to the buffer instead.

o String.range(str)

  This returns the minimum and maximum character value in the string.

  The precision is only 8 bits, so for wide strings only character
  blocks are known.

o String.filter_non_unicode(str)

  This function replaces all non-unicode characters in a Pike string
  with 0xfffd.

o SDL.Music added to SDL.

  Allows the playback of audio/music files.
  Requires the SDL_mixed library.

o System.TM

  Low-level wrapper for struct tm.

  This can be used to do (very) simple calendar operations. It is, as
  it stands, not 100% correct unless the local time is set to GMT, and
  does mirror functionality already available in gmtime() and
  localtime() and friends, but in a (perhaps) easier to use API.

o decode_value now throws the error object Error.DecodeError.

  Useful to catch format errors in the decode string.

o Process.daemon

  The daemon() function is for programs wishing to detach themselves
  from the controlling terminal and run in the background as system
  daemons.

o Debug.pp_object_usage()

  Pretty-print debug information, useful to get debug information
  about object counts and memory usage in Pike applications.

  Uses the new _object_size lfun, if present in objects, to account
  for RAM-usage in C-objects that allocate their own memory.

o Mysql

  + Added support more modern client libraries (incl. MariaDB)

  + Mysql.mysql now has methods to query the id or SQLSTATE of the
    last error.

o pgsql

  A complete rewrite of the existing driver. Changes in random order:
  + Eliminates an obscure and rare Pike-internals corruption bug.
  + Drops the _PGsql CMOD in favour of pure Pike using Stdio.Buffer.
  + Fully threaded, event and callback driven operation.
  + Allows for query setup and row fetches to be spread out over
    an arbitrary number of threads.
  + Maximum parallelism over a single filedescriptor (better than before).
  + New interface: fetch_row_array() and callback driven.
  + Less filling and faster than the existing driver.

o Protocols.DNS

  + Prevent endless loops in maliciously crafted domain names.

  + Add QTYPE T_ANY to DNS enum EntryType in DNS.pmod.

  + Handle truncated labels

  + TCP client and server support

o Thread no longer inherits Thread.Thread (aka thread_create)

o Thread.Farm now might work

o Cmod precompiler:

  + You can now use #cmod_{define,if,ifdef,include} etc to do
    preprocessing using CPP before the .cmod file is processed by the
    precompiler.

    This preprocessing is done with the Pike precompiler, while the
    C-compiler preprocessor will then be used on the generated file
    (like before)

  + inherit "identifier"
    -- inherit the program returned by calling master()->resolve() on
       the specified identifier. Useful to inherit code written in Pike.

  + Object types on return and argument types are now resolved
    dynamically when needed.

    This means that the typing information is significantly more
    strict.

    If you use classes that are not defined in the file (or
    src/program_id.h) they will be resolved when your .so-file is
    loaded.

    Note: Circular dependencies do not work (currently, at least).

    Also, the parser still often has problems with types containing
    '.'. You can use object(X.Y) instead of X.Y.

o String.levenshtein_distance()

  The function calculates the Levenshtein distance between two
  strings. The Levenshtein distance describes the minimum number of
  edit operations (insert, delete or substitute a character) to get
  from one string to the other.

  This can be used in approximate string matching to find matches
  for a short string in many longer texts, when a small number of
  differences is expected.

o System.sync()

  Synchronizes the filesystem on systems where this is possible
  (currently windows and UNIX-like systems).

o System.getloadavg()

  Return the current 1, 5 and 15 minute system load averages as an
  array.

o access()

  Check if a file exist and can also return if it is readable and or
  writeable for the current process.

o glob()

  The glob function has been extended to accept an array of globs as
  the first (glob pattern) argument.

  In this case, if any of the given patterns match the function will
  return true, or, if the second argument is also an array, all
  entries that match any glob in the first array.

o Stdio.UDP():

  + added IPv6 multicast support

  + added set_buffer

o Stdio.Port():
  + Added client and server support for TCP_FASTCONNECT

    To connect using this TCP extension simply pass the data as the
    fifth argument to connect.

    The server support is automatically enabled if possible when a
    Stdio.Port object is bound to a port.

  + Added support for SO_REUSEPORT. In this mode multiple Stdio.Port
    instances can bind to the same port (mainly used with the ports
    being opened in different processes).

    This allows automatic load sharing between the processes, incoming
    connections will only be sent to one of them by the OS.

    The presence of these features are indicated by constants in
    Stdio.Port: Stdio.Port.SO_REUSEPORT_SUPPORT and
    Stdio.Port.TCP_FASTOPEN_SUPPORT (although the same code should
    work regardless of the existence of TCP fast open in both server
    and client mode, it can be useful to know if support is available)

  + Added a query_fd() method


o Stdio.Buffer():

   This is a byte buffer (unlike String.Buffer which is a string
   buffer, and thus contains unicode characters and not bytes) that is
   optimized for both reading from it and adding data to it at the
   same time.

   The main intent of the class is buffering for binary and non-binary
   protocol parsing and output, and also buffering when using
   non-blocking I/O.

o Stdio.File():

  + send_fd and receive_fd

    These functions can be used to send and receive an open
    file-descriptor over another file-descriptor. The functions are
    only available on some systems, and they generally only work when
    the file the descriptors are sent over is a UNIX domain socket or
    a pipe.

  + Changed internally to remove one level of indirection.

    The Stdio.File object no longer has a _Stdio.Fd_ref in _fd. They
    are instead directly inheriting _Stdio.FD.

    _fd is still available for compatibility, but internally it is gone.

  + The callbacks now get the file object as the first object if no
    other id has been set.

  + Added support for buffering via Stdio.Buffer.

    This can be used to greatly simplify the writing of non-blocking
    code.

    - When read and/or write buffer is enabled the corresponding
      callbacks get a buffer object as the second argument

    - The only thing the callback has to do is add data to the buffer
      or read from it, depending on what callback it is.

    - The write callback will now only be called when the buffer
      contains no more output, and data that is not read in one read
      callback is kept for the next time data arrives.

  + Fixed grantpt() on Solaris failing with EPERM.


o Unicode databases updated to 7.0.0 from 5.1.0

  This is the latest released Unicode database from unicode.org.

o The Search search engine module has seen several fixes

  + Added support for padded blobs. This improves performance when
    incrementally growing blobs. This feature is only enabled if
    Search.Process.Compactor says this is OK, see the documentation
    for more information.

  + Several locking optimizations, specifically, avoid locking and
    unlocking for every single iteration when merging and syncing
    blobs.

  + Charset conversion fixes

  + Fixes for queries where the same world occur multiple times
    ('foo and bar and foo')

o pike -x benchmark

  + Output format changed

  + Also added support for JSON output.

  + The results should be more consistent.

  + Added options to allow comparison with a previous run.

o pike -x check_http

  This new tool can be used to check http/https connectivity to a host.

o New stand-alone tools added to make it possible to build
  documentation without the Pike build tree

  + autodoc_to_html
    AutoDoc XML to HTML converter.

  + autodoc_to_split_html
    AutoDoc XML to splitted HTML converter.

  + git_export_autodoc
    Exports a stream of autodoc.xml suitable for git-fast-import.
    Used on pike-librarian.

o Readline tries to set the charset to the terminal charset

  This makes it possible to write non-7bit characters on a terminal if
  the terminal supports it.

o Fixed units in pike --help=kladdkaka

o Several changes has been done to the GTK2 module

  + GTK2.DrawingArea no longer crash in draw_text if you pass odd
    parameters.

  + draw_pixbuf can now be passed width and height -1, which makes it
    take the size from the passed image.

  + GDKEvent no longer crash when you extract strings from them

  + accelerators now work

  + Fixed RadioToolButton

  + signal_connect can now connect a signal in front of the list

  + Several fixes to Tree related objects

  + GTK2.SourceView added

  + GTK2.Spinner added

o A few issues were fixed that were found by Coverity

  + Fixed memory leak in Math.Transform

  + Fixed two compares that were written as assignments (errno checks
    for EINTR for sockets)

o System.get_home + System.get_user

  (mostly) Cross-platform ways to get the user name and home
  directory.

o System.AllocConsole, System.FreeConsole and System.AttachConsole for
  Windows NT

  These are useful to create or close the console window that is shown
  for Pike programs.

o Process - forkd

  Forkd can be used to more cheaply create new processes on UNIX like
  systems.

  This is done by first starting a sub-process that is then used to
  create new processes.

  If your main process is large, this is significantly faster than
  using the normal create_process, and does not risk running out of
  memory for the forked (temporary) copy of the main process that is
  created.

o MacOSX CoreFoundation support in the backend

  This makes it somewhat more likely that native libraries can work
  with Pike.

o Better IPv6 support.

  This includes detection of IPV6 mapped IPV4 addresses
  (::FFFF:i.p.v.4) and full support for IPv6 in the UDP code.

o Asynchronous Protocols.SNMP client

o Fixes to Process.run, Process.spawn_pike and friends.

  + Support OS/2 path conventions

  + Fixed multiple issues with search_path()/locate_binary()
    - locate_binary() is now more reliable on Windows
    - Now invalidates the cached path is PATH is changed
    - Uses locate_binary to scan the path
    - spawn_pike() now uses search_path()

  + You can now optionally have System.spawn_pike pass predefines,
    program and include path to the spawned Pike, in addition to the
    module path.

o Lots of autodoc fixes

  A lot more of the previously existing, but broken, documentation is
  now readable.

o predef::types

  This is equivalent to values and indices, but instead gives the
  types for each value.

  Basically only useful for objects.

o Builtin._get_setter

  This function returns a setter for a variable in an object. The
  setter, when called, will set the variable value to the passed
  argument.

o Parser.XML.Tree fixes

  + Several namespace improvement and handling fixes

o New charsets

  A lot of ISO-IR charsets added:
    9-1, 9-2, 31, 232, 234, 231 (aka ANSI/NISO Z39.46, aka ANSEL) 230
    (aka TDS 565) 225 (SR 14111:1998), 197/209 (sami) 208 (IS 434:1997)
    207 (IS 433:1996), 204,205 and 206 (aka 8859-1, 8859-4 and 8859-13
    with euro) 201, 200, 138 (ECMA-121) 198 (ISO 8859-8:1999) 182, 181,
    189 (TCVN 5712:1993, aka VSCII) 167, 166 (aka TIS 620-2533 (1990)),
    164, 160, 151 (NC 99-10:81), 68 (APL), 59 (CODAR-U), 202 (KPS
    9566-97). Fixed CSA_Z242.4

o Several fixes to Protocols.HTTP

  + Improved Protocols.HTTP.Query.PseudoFile
    (significantly better Stdio.Stream simulation)

  + Do not use hard coded Linux errno:s

  + Case insensitive handling of header overrides in do_method

  + Fixed broken check for URL passwords when querying

  + Add more descriptive HTTP responses along with a mostly complete
    list of codes

  + Handle non-standards compliant relative redirects

  + Cleaner handling of async DNS failures

  + Handle chunked transfer encoding correctly when doing async
    queries

  + Fixes for the proxy client support

  + Several keep-alive handling fixes

  + Server:
    - More forgiving MIME parsing for MSIE
    - Fixed range header handling
    - Fixed parsing of broken multipart/form-data data
    - Added optional error_callback to attach_fd
    - The response processor (response_and_finish) now treats the
      reply mapping as read-only.
    - Support if-none-match (etag:s)
    - Ignore errors in close when destroying the object
    - Fixed ordo of sending large chunks of data without using files

o Multiple threads can now call the Backend `() function (the function
  that waits for events).

  The first thread will be the controlling thread, and all callbacks
  will be called in it, the other threads will wake when the
  controlling thread is done.

o dtrace support (on MacOSX)

  Pike now supports dtrace events on function enter and leaving (and
  when stack frames are notionally popped, for functions doing
  tailrecursion).

o Standards.JSON.encode can now get the initial indentation level
  specified.

  This is rather useful for recursive calls encode in pretty print
  mode (common for objects with encode_json methods).

o Added Pike.identify_cycle(x)

  Checks to see if x contains any circular structures.

  This can be useful while optimizing to identify reference cycles in
  Pike data structures, so that the need for garbage collection can be
  reduced.

o Most math functions (log, pow, exp sin, cos etc) can now take
  integers as arguments in addition to a floating point number.

  The result will still be a float, the argument will be converted.

o The random(), random_string() and random_seed() might be more random

  On computers with a hardware pseudo random generator random() can
  return significantly more random numbers, however, this means that
  random_seed is a no-op on those machines.

  A side-effect of this is that random_string is now actually
  significantly faster on at least x86 cpu:s with rdrnd.

  Note: If you want cryptographically random data please use
  Crypto.Random.random_string unless you know for sure the random data
  returned by the RDRND instruction is random enough.


SSL
---

o SSL now supports TLS 1.0 (SSL 3.1), TLS 1.1 and TLS 1.2.

o Several identifiers have changed names:

  SSL.alert -> SSL.Alert
  SSL.connection + SSL.handshake -> SSL.{Client,Server,}Connection
  SSL.context -> SSL.Context
  SSL.packet -> SSL.Packet
  SSL.session -> SSL.Session
  SSL.sslfile -> SSL.File
  SSL.sslport -> SSL.Port
  SSL.state -> SSL.State

o SSL.File: Changed client/server selection API.

  Client and server operation is now selected by calling either
  connect() (client-side) or accept() (server-side) after creating the
  SSL.File object.

  Blocking handshaking mode is selected by calling set_blocking()
  before either of the above.

o SSL.File: Redesigned I/O state machine.

  This reduces the complexity and risk of bugs in the I/O handling.

o SSL support for lots of new cipher suites added:

  + AEADs and modes:

    - CCM and CCM-8

    - GCM

  + Certificates

    - ECDSA

  + Ciphers

    - AES and AES-256

    - CAMELLIA and CAMELLIA-256

  + Key exchange methods

    - DH and DHE

    - ECDH and ECDHE

  + All suites currently defined consisting of combinations of the
    above (and earlier existing) have been added (~190 suites in
    total).

o TLS Extensions added:

  + ALPN (Application Layer Protocol Negotiation) (RFC 7301).

  + EC Point Formats (RFC 4492).

  + Elliptic Curves (RFC 4492).

  + Encrypt then MAC.

  + Fallback SCSV.

  + Heartbeat (RFC 6520).

  + Max Fragment Length (RFC 6066).

  + Padding.

  + Renegotiation info (RFC 5746).

  + Signature Algorithms (RFC 5246).

  + SNI (Server Name Indicator) for both client and server (RFC 6066).

  + Truncated HMAC (RFC 6066).

o Improved protection against various SSL/TLS attacks:

  + BEAST protection (aka 1/(n-1)).

    Improve resilience against the BEAST client-side attack, by
    splitting the first data packet into two, where the first only
    contains a single byte of payload.

  + Heartbleed protection.

    The protocol code can probe the peer for the Heartbleed
    vulnerability, and aborts the connection with
    ALERT_insufficient_protection if so.

  + Lucky 13 protection.

    Attempts to have HMAC calculation take the same amount of time
    regardless of padding size.

o SSL.Context: Improved cipher suite selection:

  + Filtering of weak cipher suites.

  + Suite B (RFC 6460).

o SSL.Context: Support multiple concurrent certificates.

  This allows a server to eg have both an RSA and an ECDSA
  certificate.


Crypto and Nettle
-----------------

o Nettle refactored

  CBC cipher mode is now twice as fast.

o Nettle 3.0 supported.

  The new APIs in Nettle 3.0 are now detected and utilized.

o Crypto.GCM

  GCM (Galois Counter Mode) cipher mode added.

o Blowfish and Serpent support fixed in Nettle

o Crypto.PGP

  Added support for SHA256, SHA384 and SHA512 as hash functions.
  Expose the used hash and key types in the out data

o Crypto.Arctwo

  The 1-128 bit cipher Arctwo is now provided as a block cipher in
  Crypto. This cipher is only intended for compatibility with OLD
  third party code, and should NOT be used for new development.

o Crypto.Camellia

  The 128/256 bit cipher Camellia is now available as block cipher in
  Crypto.

* Crypto.ECC

  Elliptic Curve operations are now supported when compiled
  with Nettle 3.0 or later.

o Crypto.SALSA20 and Crypto.SALSA20R12

  The 128/256 bit cipher SALSA20 is now available as a stream cipher
  in Crypto. SALSA20R12 is SALSA20 reduced to just 12 rounds.

o Crypto.SHA3_224, Crypto.SHA3_256, Crypto.SHA3_384 and Crypto.SHA3_512

  The SHA-3 secure hash algorithm has been added in multiple variants.

o Crypto.GOST94 and RIPEMD160

  The lesser used hash functions GOST R 34.11-94 (RFC 4357) and
  RIPEMD160 have been added.

o Crypto.RSA and Crypto.DSA

  The key generation for RSA and DSA are now done by Nettle. This
  results in 90% faster key generation for RSA. Key generation for DSA
  is 10 times slower, but produces better quality keys.

o Crypto.Hash

  Added support for pbkdf1 from PKCS#5v1.5 and pbkdf2 from PKCS#5v2.0.

o Crypto.Random

  The underlying algorithm for Crypto.Random has changed from a overly
  cautious Yarrow implementation to Fortuna on top of system random
  sources. About 7000x faster than before.

o Standards.PEM

  + Added some support for encrypted PEM files

o Standards.X509

  X509 was moved from Tools to Standards and has been refactored and
  bug fixed. It is now possible to extend both validation and creation
  of certificates with new cipher and hash algorithms. A range of new
  algorithms are supported for both RSA and DSA:

    RSA MD2
    RSA MD5
    RSA SHA-1
    RSA SHA-2-256
    RSA SHA-2-384
    RSA SHA-2-512
    DSA SHA-1
    DSA SHA-2-224
    DSA SHA-2-256

  Note that the API has changed compared to Tools.X509 and there is
  now a single make_selfsigned_certificate() method for both RSA and
  DSA, though it takes the same arguments. In addition a hash function
  and serial number can be supplied. The hash function defaults to
  SHA-2-256.


Incompatible C-level API changes
--------------------------------

o New svalue layout

  The most obvious change is that the macros TYPEOF() and SUBTYPEOF()
  are now actually needed to directly access the type and subtype of
  an svalue, the svalues no have 'type' and 'subtype' members.

  There are also a few additional macros used to set both the type and
  subtype of an svalue at the same time:

    SVAL_SET_TYPE_SUBTYPE(sval,type,subtype) and
    SVAL_SET_TYPE_DC(sval,type)

  (these are only neede if you do not use the usual push_* macros)

  They are useful because they (usually) compiles to significantly
  more compact code, especially if the type and subtype are
  compiletime constants. The _DC variant will either leave the subtype
  unchanged or set it to 0, useful when you do not care about the
  actual subtype (which is, really, most of the time).

o get_storage() returns void*

  There is no need for casts in non-C++ code.


Building and installing
-----------------------

o -fvisibility=hidden is now the default

  This means that PMOD_EXPORT is now actually needed on systems like
  Linux and MacOSX. It also means that the binary is slightly smaller
  and faster.

o clang compilation fixes (bignum overflow checks, misc)

  It is now possible to compile Pike using a modern clang compiler.

o Removed bundles

  Pike no longer comes with copies of some libraries, and the support
  for that in the makefile has been removed.

o Several OS/2 and windows compilation fixes

o C89 assumed

  The configure tests will not check for functions defined in C89
  anymore.


Lots of bug fixes
-----------------

o Fixed symbol resolution with deep inherits and mixins

o Fixed PNG 4bpl indexed mode with alpha

o The _sprintf LFUN now works with %F

o foreach( arr[-two()..], string x), where two() returns 2 will no
  longer iterate over the first element in the array three times or
  crash.

o Postgres.postgres_result now properly decodes binary blobs and
  strips trailing spaces from CHAR columns.

o Fixed a typo from 2001-05-05 that caused a lot of countries to
  reside on the continent ',Europa' instead of "Europa" in
  Geography.Countries.

  Obviously the continent is not that often checked.

o Fixed interesting bug in the constant number parsing for octal
  numbers in escapes.

  For whatever reason 8:s were accepted for every location except the
  first in octal numbers, so 078 was considered an alias for 0100.

  This could cause issues at times, and will result in string parsing
  being somewhat different:

  |  Pike v7.8 release 700 running Hilfe v3.5 (Incremental Pike Frontend)
  |  > "\078";
  |  (1) Result: "@"
  |  Pike v8.0 release 3 running Hilfe v3.5 (Incremental Pike Frontend)
  |  > "\078";
  |  (1) Result: "\a8"

o A lot more, see the (more or less) full changelog for more info:

   http://pike-librarian.lysator.liu.se/index.xml?m=pike.git&start=forever&branch=7.9,8.0&template=logonly.inc