:orphan:

JPype 0.7 Core ChangeLog
========================

Here is the "complete" log of the changes I think I made.

Module changes
--------------

* Moved Python module objects to namespace ``PyJP`` so that they are consistent
  with a Python module namespace.  Renamed module classes presented to 
  ``_jpype`` extension to ``PyJP*`` to match the internal classes.  Though not
  exactly a standard convention, the types were internal anyway and having
  the names match the C structure makes it more clear what resource is being 
  accessed.  It also eliminates the confusion between ``jpype`` and ``_jpype``
  resources.

* Removed all usage of Capsule from the extension module. This was bridging
  between Python versions and had to be replicated on old platforms.  As the
  capsules were functioning as crippled objects, they could not have methods
  of their own.  Thus functionality that properly belonged to a specific class
  would get pushed to the base class.  This affected former capsules of
  ``JPObject``, ``JPProxy``, and ``JPArray``.  These are now formal classes in the
  module as ``PyJPValue``, ``PyJPProxy``, and ``PyJPArray``.

* Moved the initialization of each class to the ``__init__`` function.  Thus
  rather than creating the resource at the top level ``_jpype`` module (such as 
  ``_jpype.findClass('cls'))``, the resource is created by allocating a new 
  object (such as ``_jpype.PyJPClass('cls')``).

* The presentation of ``JPArrayClass`` has been merged as a generic ``JPClass``.  
  The only requirement for creation of an array instance is that the supplied 
  ``PyJPClass`` satisfy ``isArray()``.

* Removed direct dependencies that objects holding resource be exactly the
  type in ``jpype`` module.  This reduces the restrictions in the underlying 
  Python layer and allows for multiple classes such as ``JavaArray``, ``JWrapper``,
  and ``JavaClass`` to all be recognized as holding resources.  This simplifies
  some paths in the ``jpype`` module where we needed to simply access a single
  method during bootstrapping and we were forced to construct complete
  classes necessitating the order of resource loading.

* Remove ``JPObject`` concept and replaced it with ``JPValue``.  ``JPValue`` holds
  the type of the object and a ``jvalue`` union.  Both ``JavaClass`` and
  ``JWrapper`` now point to these classes as ``__javavalue__``.  Anything with a
  ``__javavalue__`` with type ``_jpype.PyJPValue`` is now recognized as being a
  Java object.

* Changed the recognization of a ``JavaClass`` to any object holding
  ``__javaclass__`` with type ``_jpype.PyJPClass``.  This allows array classes,
  object classes, and wrappers classes to be used together.

* Added hooks to direct convert ``PyJPClass`` to a ``PyJPValue`` with a type of
  ``java.lang.class`` and an object to the class. This replaces the need for
  calling ``forName`` to get to the existing class.

* Changed ``PyJPField`` and ``PyJPMethod`` to descriptors so that we do not
  need to mess with  ``__getattribute__`` and ``__setattr__`` in many places.
 
* Eliminated the unnecessary class bound method.


C++ Reorg
~~~~~~~~~

* Reorganized the type tree in the C++ layer to better match the Java
  structures.  

* Flattened out the redundant layers so that ``JPType`` is now ``JPClass``
  corresponding to an instance of a ``jclass``.

* ``JPClass`` is not the base class.  Arrays are now objects and have 
  base classes and methods.

* Split ``JPClass`` into a separate type for each specialized object class for
  boxed, ``java.lang.Object``, and ``java.lang.Class`` which all required
  specialized conversion rules.

* Boxed, string, base ``java.lang.Object`` and base ``java.lang.Class`` are now
  specialized with their required conversion rules.


Path reduction
~~~~~~~~~~~~~~

* Removed ``HostRef`` and all of its usage.  It was a halfway memory
  management method.  To be passed around it was being held as a dynamically
  allocated pointer type with no static presence for cleanup.  This defeats
  the point of having a smart point wrapper if the smart pointer is being used
  as a pointer itself.  Thus it was only as safe as the user applied
  conventions rather than safe by design.

* Replaced all the ``HostRef`` methods and ``JPy*`` Python object wrappers with
  a new smart pointer concept (namespace ``JPPy``).  This removes the
  redundant host and ``JPy*`` wrapper layers.

* Removed multiple optimization paths such as bypassing between ``jchar`` and
  ``unicode`` if the size matched.  These paths were for speed reasons, but they
  could only be tested on particular machines.  Thus it was difficult to tell
  if something was broken.  It is better to have one tested code path
  that is slight slower, then a faster path that is busted.

* Removed dead class ``JPCharString``.

* **(bug)** Replaced all string handling with conversion through UTF8.  Java and
  Python use different UTF8 encodings and thus those paths that were trying
  to short cut directly through from one system to another were badly flawed.
  By forcing a conversion to and from each time a Java string or Python string
  are passed eliminates conversion problems.  This should resolve user issues
  having to do with truncating extended unicode characters.

* Combined all code paths in ``canConvertToJava`` and ``convertToJava`` to use the
  ``JPValue``

* Combined code paths from ``check`` and ``get`` for ``JPValue``, ``JPClass`` and
  ``JPProxy`` ``get`` patterns when fetching from Python.  Almost always we want
  to use the object immediately and just check if we can.

* Removed the entirely redundant Primitive type ``setRange`` and ``getRange``.
  That code was entirely dead because it could not be reached. Renamed the
  direct methods as they now have the same function.

* Removed ``JPTypeName``.  This concept will be phased out to
  support lambdas. ``TypeManager`` now used ``getCanonicalName()``.
  Transferred responsibility for conversion to native names to Python module 
  interface.

* Introduced named classes for all specialized instances of classes to be
  held in ``TypeManager`` namespace. Thus converted most of the "is this type" to 
  comparison of ``JPClass*`` pointers in place of string level comparisons.  

* Removed near duplicate methods.  ``JProxy`` was requesting slightly altered
  copies of many conversions to support its usage.  These operations could
  be supported by just splitting to two existing methods.  Thus we could
  eliminate a lot of stray methods that served this specialized purpose.

* ``JPArray`` is now a method holder rather than the primary object like
  ``JPBoundMethod``.  All array objects in Python now hold both a ``__javaarray__``
  and a ``__javavalue__``.  This eliminates need for special paths for 
  arrays.

* ``_getClassFor`` is now overloaded to work with array classes.  Thus
  asking for a ``JClass('[java.lang.Object;')`` will now correctly 
  return a JavaArrayClass.

* Constructing a string now shortcuts to avoid methodoverload resolution on 
  new instance if given a Python string.

* Reworked the GIL handling.  The previous model was doing all the release
  locks on the JPJni calls automatically for almost all jni transactions.
  This would be fine, except that many utility functions were using those same
  calls regardless of whether is was a good time to release the lock.  This
  ultra fine grain locking was effectively allowing any call to JPJni methods
  to become a break point, including those calls in critical sections such as
  ensureTypeCache and TypeManager::findClass.  Any time it loaded a class or
  looked up a name it could be interrupted and thus end up in a corrupt state.
  Thus I moved all of the GIL calls to those places where we call user code on
  the type returns and the object constructors.  Thus cuts the number of GIL
  transactions greatly and eliminates the need to deal with trampling global
  resources.  The refactor exposed this a bit more because the removal of
  TypeName meant that we did a lot more transactions to get the class name.
  But that does not mean the flaw was not there before.  If our tests cases had
  been any more aggressive about creating class instances during execution it
  would have overrun the TypeManager table and all would have failed.

* Removed the previous default option to automatically convert
  ``java.lang.String`` to either a Python string or a unicode when returning
  from Java.  This does mean some string operations now require calling the
  Java string method rather than the Python one.  Having strings not convert
  but rather remain on the jvm until needed cuts the conversion costs when
  working with Java heavy code.  I added a caching mech so that if we need to
  convert the string multiple times, we don't pay additional over the previous
  option.

* A special ``toString`` method was added to ``PyJPValue`` to convert Java 
  strings to Python strings. This can convert Java string resources to 
  Python ones on request.


Proxy changes
~~~~~~~~~~~~~

* Proxy as implemented previously held only a pointer to the proxy object
  and from this proxy object it lookup up the callable using either a
  dictionary or an instance.  The majority of the resources were held
  by the ``jpype.Proxy``.   This was replaced with a more general function
  in which the ``PyJPProxy`` proxy holds two resources.  One is an object
  instance and the other is a lookup function that turns the name to a 
  function definition. This supports the same use cases but eliminates
  the need for finding resources by convention.  There is no need for 
  the proxy in Python to have any specific layout other than holding a
  PyJPProxy as ``__javaproxy__``.  Thus allowing alternive structures 
  such as Proxy by inheritance to work.

* Memory handling was changes slightly as a result so that the reference
  queue is now responsible for cleaning up the proxy.  Proxy handle instances 
  are generated whenever the proxy is passed to Java.  Thus we form no
  counting loops as the proxy has no reference to the handles and the 
  handles hold a reference to the proxy. 


Exception changes
~~~~~~~~~~~~~~~~~

* Changed all exception paths to use ``JPypeException`` exclusively.  The prior
  system did way to much in the Exception constructors and would themselves
  crash if anything unusual happened making changing of the system nearly 
  prohibitive to debug.  Everything bubbles down to ``toJava`` and ``toPython``
  where we perform all the logging and pass the exception off.  This also
  centralizes all the handling to one place.

* This pulls all the logic from ``JPProxy`` so that we can now reuse that 
  when returning to any Java jni native implemented function.

* Same thing for Python, but that was already centralized on ``rethrow``.

* Reworked exception macros to include more info and introduced ``JPStackInfo``.
  It may be possible to connect all the stack info into the Python traceback 
  (via a proxy class) to present a more unified error reporting.  But this
  work is currently incomplete without a Python layer support class.

* Integrated ``JPStackInfo`` into tracer to give more complete logs when
  debugging.


Code quality
~~~~~~~~~~~~

* Applied a source formatter in netbeans.  It is not perfect as it tends to 
  add some extra spaces, but it does make faster work of the refactor. 
  Custom spacing rules were applied to netbeans to try to minimize the total
  changes in the source.

* Improved error handling where possible.

* Rework ``JPTracer`` so that reporting from places that do not have a formal
  frame or could not properly throw (such as destructors) and still appear in
  the trace log.  All ``TRACE`` macros were moved to ``JP_`` so that were less
  likely to hit conflicts.  Removed guards that complete disabled Tracer from
  compiling when ``TRACE`` was not enabled so that unconditional logging for
  serious failure such as suppressed exceptions in destructors can report.

* Defensively added ``TRACE`` statements whenever entering the module for a
  nontrivial action so that errors could be located more quickly.

* Removed ``MTRACE`` layer as Java local frame handles all cleaning tasks for
  that now.

* Replaced TRACE1, TRACE2, TRACE3 with a variodic argument macro ``JP_TRACE``
  because I am too lazy to remember to count.

* Renamed functions to best match the documented corresponding function in
  the language it was taken from.  Thus making it easier to find the needed
  documentation.  (Ie  ``JPyString::isString()`` becomes 
  ``JPPyString::check()`` if the corresponding language concept is 
  ``PyString_Check()``).  This does mean that naming is mixed for the 
  Java/Python layers but it is better to be able to get the documentation 
  than be a naming idealist.

* Used javadoc comments on header of base clases.  These strings are picked
  up by netbeans for document critical usage.

* Moved method implementations and destructors out of headers except in 
  the case of a truly trivial accessor.  This has a small performance loss
  because of removal of inline option.  This reduces the number of
  redundant implementation copies at link time and ensures the virtual
  destructor is fixed in a specific object.  We can push those back to the 
  header if there is a compelling need.


``jpype`` module changes
---------------------------

Because these do affect the end user, we have marked them as enhance, change, remove, bug fix, or internal.

General
~~~~~~~

* **(enhance)** ``__all__`` added to all modules so that we have a well defined
  export rather that leaking symbols everywhere. Eliminated stray imports in
  the jpype namespace.

* **(enhance)** Add ``@deprecated`` to ``_core`` and marked all functions that are
  no longer used appropraitely.  Use ``-Wd`` to see deprecated function warnings.

* **(enhance)** Exposed ``JavaInterface``, ``JavaObject``, ``JavaClass`` so that they
  can be used in ``issubclass`` and ``isinstance`` statement.
  ``JavaClass.__new__`` method was pushed to factory to make it safe for external
  use.

* **(enhance)** mro for Java Classes removes ``JavaInterface`` so that
  ``issubclass(cls, JavaInterface)`` is only true if the class not derived from
  ``JavaObject``.

* **(enhance)** All classes derived from ``java.lang.Throwable`` are now usable as
  thrown exceptions.  No requirement to access special inner classes with 
  exception types.  Exceptions can be raised directly from within
  a Python context to be passed to Java when in proxy.  Throwables now 
  use a standard customizer to set their base class to the Python
  Exception tree. Deprecated ``JException``

* **(enhance)** ``args`` is a property of ``java.lang.Throwable`` containing the
  message and the cause if specified.  

* **(enhance)** ``JChar`` array now converts to a string and compares with string
  properly.  Conversion uses range so that it does not try to convert
  character by character.

* **(remove)** ``JByte`` array is not a string type.  It is not a string in Java
  and should not be treated as a string without explicit conversion.
  Conversion path was horribly inefficient converting each byte as a Python
  object.  Test marked as skip.

* **(change)** Array conversion errors produce ``TypeError`` rather than
  ``RunTimeError``.

* **(enhance)** ``JArray`` now supports using raw Python types as the specifier for
  array types.  It will convert to the most appropraite type or return an
  error.  

* **(remove)** property conversion customizer is deactivated by default.  This
  one proved very problematic.  It overrided certain customizers, hid
  intentionally exposed fields, bloated the dictionary tables, and interferred
  with the unwrapping of exception types.  We can try to make it an optional
  system with ``import jpype.properties`` or some such but it will still have all
  those problems.  Best to kill this misfeature now.

* **(enhance)** ``JArray`` classes now have ``class_``.  We can access the component
  type.  This makes them more consistent with ``JClass``.  (required for
  testing)

* **(enhance)** Use of constructor call pattern eliminated the need for use of a 
  separate factory and type.  Thus we are back to the original design in 
  which we only need to expose a small number of "types".  This was applied to 
  ``JArray``, ``JClass``, ``JException``, and ``JObject``.  Use of ``isinstance()`` and
  ``issubclass`` now supported. The only challenge was keeping box types working.

* **(remove)** Functions that return a string now return a ``java.lang.String``
  rather than converting to Python.  Thus when chaining elements together in
  Java will get the full benefit matching types.  The previous auto convert
  has been removed.

* **(enhance)** ``java.lang.String`` now has much more complete set of Python 
  operations.  String conversions are now cached, so the penalty of 
  converting is kept to a minimum.

Wrappers
~~~~~~~~~

* **(internal)** Rewrote the ``JWrapper`` module from scratch to reflect the use i
  of ``JPValue``.  Renamed ``_jwrapper`` to ``_jtypes``.  The concept of wrappers 
  has now been lost internally.  All objects and primitives are just values.

* **(enhance)** Created import module containing all of the symbols needed for 
  creating types in jpype so that we can support a limited import statement 
  ``from jpype.types import *``

* **(enhance)** ``JString`` contructor now returns a ``java.lang.String`` object.  
  Removed ``JStringWrapper`` as ``java.lang.String`` serves its purpose.

* **(enhance)** ``JObject`` now returns an object with the Java type as a functional
  object rather than a dead end wrapper.  This does allow some redundant 
  things such as converting a Python class wrapper into a class 
  ``JObject(java.lang.String) == java.lang.String.class_`` but otherwise seems 
  good.

* **(enhance)** 'JObject' and 'JString' accept 0 arguments to generate a generic 
  object and empty string.

* Tried to be more consistent about returning errors that are valid in Python.

   - Too many or two few arguments to a function will throw a ``TypeError``
   - Value conversion out of range will throw ``OverFlowError``
   - Value conversions that are the right type but invalid value will 
     give ``ValueError`` (char from string too long)
   - Type conversions that cannot be completed should give ``TypeError``.
   - Errors setting attributes should give ``AttributeError`` such as 
     trying to set a final field or trying to get an instance field from a 
     static object.
   - Arrays access should produce ``IndexError`` on bad range.
     (it would be nice if these also mapped to Java errors and the corresponding
     errors in Java were derived from the Python error so that we can properly
     look for ArrayIndexOutOfBoundsException (derived from IndexException).  But
     that is too heavy to attempt now.)

* **(enhance)** ``JArray``, ``JException`` and ``JObject`` report as JavaClass when 
  using issubclass.

* **(enhance)** Short cut for just adding a base class as a customizer.


Internal
~~~~~~~~~
* **(internal)** Changes corresponding to the ``__init__`` rework to match revised 
  ``PyJP*`` classes.

* **(internal)** Changes corresponding to the capsule removal.

* **(internal)** Remove ``SPECIAL_CONSTRUCTOR_KEY`` as everything that uses it can 
  recognize a PyJPValue as indicating they are receiving an existing Java 
  resource as input.  All special handling required to construct objects from 
  within C++ layer were thus eliminated. 

* **(internal)** Removed almost all required resources from Python needing to be 
  register in ``_jpype`` with the exception of getClassMethod.  

* **(internal)** Java class customizers did not need to be deferred until after 
  the JVM is initialized.  Pushing them into the dictionary immediately 
  fixes issues in which a customizer was not applied to classes during 
  early bootstrapping.  This eliminates a large number of the need for 
  calling initialize on each jpype module in ``_core``.

* **(internal)** ``JArrayClass`` and ``JClass`` are the same for purposes of 
  Customizers and class tree.

* **(internal)** Customizer code and dictionary moved to ``_jcustomizer`` so that i
  it can be shared between Object and Array classes.

* **(internal)** Converted ``JavaClass`` to more Python like "try first, eat an 
  exception if it fails" philosophy to increase robustness to failure.  This 
  eliminates the problems when a new base class is introduced with a 
  customizer without setting up a meta class.

* **internal/enhance** Broke connections between boxed types and wrappers.  
  User supplied wrappers can implements specified "<type>Value" method.  
  Wrapper types now have similar methods to boxed types with appropriate 
  range checks.

* **(internal)** All ``$Static`` meta classes have been eliminated.  There is now 
  only one tree of classes.  A single meta class ``JClass`` serves as the type 
  for all classes.

Bugs
~~~~~~~

* **(bug fix)** Fixed bug in ``jpype.imports`` in which it would not install its 
  hooks if loaded afer the jvm was started.

* **(bug fix)** Fixed bug in JBoxed type wrappers in Python which would lead 
  ``java.lang.Double`` and ``java.lang.Float`` to have an integer value when 
  boxed was corrected.

* **(bug fix)** Fixed bug in ``JObject`` that was preventing classes from being 
  wrapped as objects.  Verified a number of test cases in the test suite.

* **(bug fix)** Reenabled the throw from Java test during proxy.  The issue was 
  that jpype was releasing resources before it could transfer control
  a ``PyErr_Clear`` removed the reference and thus our throwable was invalid.  
  It was dastardly to find, but the fix was moving a statement one line up.


Documentation changes
~~~~~~~~~~~~~~~~~~~~~~

* Documentation of major class methods have been added as well as marker 
  whereever the underlying assumptions are not reasonably transparent.

* Action items for further work have been marked as FIXME for now.


Incomplete
----------

These tasks had to be pushed over post 0.7 release.

* Finish specialization of ``JPArray`` classes for ``byte[]`` and ``char[]``

* Deal with fast array conversions misuse of types. ``int[]<=>float[]``

* Direct bridge methods for ``char[]`` are currently bypassing the unicode 
  translation layer.  It is unclear what Java does with extended unicode 
  when dealing with ``char[]``.  

* Add a system to register a translation customizer so that we do not need to
  modify C++ code to add new simple translations like Python date to Java 
  Instant.  These would be installed into the PyJPClass during class 
  wrapper customization. We will need to make sure each class has a Python
  type wrapper cached in ensureTypeCache so we are guaranteed to find
  the conversion.

* Add tests for Exception.args