Introduction to libnjb hacking
by Linus Walleij
------------------------------

This file contains some tips on how to get started if you want to
change stuff in libnjb, and some technical details we have found.

First, join the developer list at SourceForge, for details see
the project page at:

  http://www.sourceforge.net/projects/libnjb/


Architecture and portability:
-----------------------------

Libnjb uses the libusb which in turn use the kernel of the host
operating system:

              libnjb
                |
              libusb
                |
              kernel

When you're trying to fix a problem, first try to locate in which
layer of this sandwich the problem is located. This improves libnjb,
libusb and the kernel of your operating system at the same time,
and as other projects use libusb you help a *lot* of people when
you find bugs in libusb. See their homepage:

  http://www.sourceforge.net/projects/libusb/

Libnjb+libusb+kernel has been verified to work with:

  * Linux 2.4 and Linux 2.6 - issues remain with some USB 2.0
    cards. (Kernel problems, not in our code.)
  * FreeBSD
  * MacOS X
  * Microsoft Windows (Win2K)

Windows uses MinGW32 and libusb-win32, which is still in the works. 
See: http://www.mingw.org and http://libusb-win32.sourceforge.net/


The devices:
------------

Libnjb developers have a special view of the different jukeboxes,
based on their characteristics on the USB bus.

* Nomad Jukebox "NJB1": USB 1.1, protocol 1. This device has it's
  own protocol, not shared by any other jukeboxes. This device
  runs the OaSis operating system developed at the Department of
  Computer Science and Engineering in Kharagpur, India. There 
  exists an IEEE paper on this OS that you can find if you do a
  web search for "oasis operating system". The main CPU is a 
  Cirrus Logic EP7212.

The "series 3" devices are so called because the first device in
this class was the Nomad Jukebox 3 "NJB3", which actually appeared
on the market before the Nomad Jukebox 2 "NJB2". All the series 3
devices share the same protocol, with minor differences between the
USB 1.1 and USB 2.0 devices. (USB 2.0 means "high speed device" 
here.)

Series 3 devices:

* Nomad Jukebox 2 "NJB2": USB 2.0, protocol 3
* Nomad Jukebox 3 "NJB3": USB 1.1 + FireWire, protocol 3
* Nomad Jukeboz Zen "NJB Zen": USB 1.1 + FireWire, protocol 3
* Nomad Jukebox Zen 2.0 "NJB Zen 2.0": USB 2.0, protocol 3
* Nomad Jukebox Zen NX "NJB Zen NX": USB 2.0, protocol 3
* Nomad Jukebox Zen Xtra "NJB Zen Xtra": USB 2.0, protocol 3
* Dell Digital Jukebox "DELL DJ": USB 2.0, protocol 3
* Nomad Jukebox Zen Touch "NJB Zen Touch": USB 2.0, protocol 3

We don't know much about the circuitry or operating systems in 
these jukeboxes.

Chart:

                      Protocol    USB 1.1   USB 2.0  FireWire
-------------------------------------------------------------
NJB1                  1           X
NJB2                  3                     X               |
NJB3                  3           X                  X      |
NJB Zen               3           X                  X      |
NJB Zen 2.0           3                     X               |
NJB NX                3                     X               | series 3
NJB Xtra              3                     X               |
Dell DJ               3                     X               |
NJB Zen Touch         3                     X               |


The Dell DJ also differs from the series 3 jukeboxes in that it
only has the equalizer presets for the EAX processor installed.
None of the environments, spatialization, time-scale, or 
smartvolume stuff found in other series 3 jukeboxes is available.



Suggested Projects:
-------------------

* Fully analyze and document the protocol.

* Implement a small HTTP server for our "samples" that can be used
  for controlling the jukebox using only a web browser.

* Create a plugin for the GNOME Rhythmbox program in the same spirit
  as the iPod plugin.

* Create a plugin for the MacOS X iTunes program.
* Filesystems integration, see below.


On writing NJB filesystem-like components:
------------------------------------------

Users of Unix-like operating systems in particular have a nack to
turn everything into filesystems. Naturally, many people believe that
the best way of accessing the jukeboxes would be through a filesystem
component, which is indeed also the way that USB mass storage-enabled
devices like the Nomad MuVo and other peripherals like digital 
cameras do it.

The problem with the Nomad Jukeboxes is that they are actually not
filesystems: internally, the jukeboxes are databases. The Apple iPod
is engineered very much the same way. Thus, a "file" in a nomad
jukebox consists of a database entry with several metadata 
components, such as artist, title, original filename, and the actual
MP3/WAV/WMA-file is just a binary large object (BLOB) in this 
database entry. Such a database structure is not easily reflected
as a filesystem: when reading files off the device this is easy
enough, but when *writing* files, all relevant parts of the metadata
has to be added to the database. This means that a filesystem 
component that is capable of writing to the jukebox would have to
include something like an ID3 tag parser to extract metadata from
the file and insert this into the database.

The main reason why libnjb is a userspace library and not a kernel
filesystem is, however, that this makes libnjb portable, so that it
can be used on Linux, BSD, MacOS X and other targets. Kernel modules
are very kernel specific. It might be possible to "wrap" libnjb into
a kernel module, but no-one has yet tried this. Also, it brings the
advantage that a crash in libnjb does not crash the entire kernel 
and thus the entire operating system. Only a so-called microkernel
where filesystems are user processes could survive such an event, 
and neither e.g. Linux nor BSD is a microkernel.

There are however filesystems available at a higher level of 
abstraction: the desktop projects GNOME and KDE both carry their
own database-like filesystem interfaces. (The original BeOS also
had such a filesystem.) The GNOME interface is known as GNOME
VFS (Virtual File System, not to be confused with the Linux 
kernel VFS) and in KDE it is known as "kioslaves", the KDE
IO-Slaves. Even Microsoft Windows will in the near future incorporate
a system known as WinFS exhibiting much the same qualities. These 
are very much fit for the kind of database structure used by the 
jukeboxes.

Shaun Jackman has implemented a KDE KIO::slave for libnjb, named
kionjb, available at Sourceforge:

http://sourceforge.net/projects/kionjb/

Pedro Ayala Gomariz and David A. Knight, libnjb developers, has 
started a project named "njbstack" that aims at integrating libnjb 
with the GNOME VFS. This GNOME VFS interface is available in a test 
version from the project page for Neutrino at:

http://sourceforge.net/projects/neutrino/

Projects in this spirit are probably the best way to use libnjb
in desktop systems.


General USB characteristics:
----------------------------

All jukeboxes identify themselves as "custom" devices.
Apart from the compulsory device control endpoints, NJB1 uses 
endpoint 2 for both IN and OUT bulk transactions. The series 3
devices use endpoint 1 for OUT transactions and endpoint 2 for 
IN transactions.

The NJB1 has one configuration only. The USB 2.0 devices usually 
have  two configurations: high-speed (default) and full-speed 
(second configuration). When two configurations are present, they
are named "MEDIA" and both have one interface only, named "PDE1". 
The Zen Touch has one unnamed configuration and one "PDE1" 
interface.

This "PDE1" interface is in turn has the two bulk endpoints. 

If you want to investigate the USB interface of your device, you
can use the command "lsusb -v" under Linux.


USB Device ID:
--------------

Find the USB Device ID for a new device, in Linux by looking in 
the file /proc/bus/usb/devices. Report this device ID to:

* The libnjb development team. (Through the mailing list
  located at SourceForge.)

* The device database at http://www.linux-usb.org/usb.ids
  (email at top of that file).

* The "usb.lst" device database used by Debian "discover-data" package
  http://packages.debian.org/unstable/libs/discover-data.html
  http://packages.qa.debian.org/d/discover-data.html

(More places?)


FireWire (IEEE 1394)
--------------------

Libnjb doesn't currently include support for FireWire. The reasons
are simple:

* None of the active developers have a FireWire device. Only the
  Nomad Jukebox 3 and the first version of Nomad Jukebox Zen
  have FireWire.

* We don't have any traces of how the FireWire traffic actually 
  look. It would be good for us to know, whether we start to 
  support it or not, so traces are welcomed. Notice that such
  traces are hard to create: the only way we know of involves
  using hardware protocol analyzers.

As you see, this is a nice task for an eager developer to take on.
libraw1394 (http://sourceforge.net/projects/libraw1394) seem like
a good place to start for providing 1394 support to libnjb.


USB Traces:
-----------

When developing libnjb it may be necessary to compare the 
functionality of libnjb with that of programs developed for
Microsoft Windows.

Most Linux developers that want to create a driver for something
running on Windows will try to intercept and decode USB traffic.
There exist several professional and expensive tools to do this,
both hardware and software. If you have such tools at your hands,
then it is good for you, and you'll probably be doing a very good
job at reverse-engineering the jukebox drivers.

Moste people do not have such delicate devices and will have to
resort to using software only. USB Snoopy is such a software tool
for Windows, that will try to intercept and log USB traffic to/from
any USB device:

http://www.wingmanteam.com/usbsnoopy/

Try to perform the parts of the behaviour that you want to capture
and make a log. If it is implemented in libnjb but fails, compare
the log to the result of running libnjb "sample" tools (found in
the "sample" directory of this library) with the debug flag -D7
(e.g ./handshake -D7) which will produce output similar to what
USB snoopy produces.
  

Device addition / adding new commands:
--------------------------------------

When you add a new device to libnjb (provided you know the
vendor and Device ID), you do this by:

 * Adding the device in the "nomad.usermap" file
 * Adding the appropriate definitons in "src/njbusb.h"
   (don't forget the macros below the definitions)
 * Adding detection code to "src/base.c"
 * Adding any device-specific code in "src/base.c",
   "src/procedure.c", "src/protocol3.h", "src/usb_io.c" 
   and the like.


Error codes:
------------

In the series 3 devices, after almost every jukebox operation a 16-bit 
number is returned, as a status code. Usually this code will be 0x0000 
which means "OK".

The other possible values cannot be said to have been deeply 
explored, and are simply just regarded as "errors" though some of
them may actually be informative. Normally libnjb will just write 
out their occurance and leave it at that, doing nothing. If you 
run into a certain new error code, try to figure out:

* The number (given)
* When it occurs, and thus what it means.
* How it is handled by the PlayCenter software, if it can be provoked
  there.
* How we should implement code to handle the error.


Unimplemented commands:
-----------------------

The following commands are so far unimplemented in libnjb:

Nomad Jukebox 1 (src/protocol.c):

* Firmware upgrade - if you want to experiment with this you
  are either insane or have a box of NJB:s to waste. It is
  possible to capture a firmware upgrade with a USB protocol
  analyzer and "replay it", but we dare not try.

In Nomad Jukebox 2, 3, Zen, Zen USB 2.0, NX, Xtra and the
Dell Digital Jukebox, all known as the "series 3" devices 
(src/protocol3.c):

* Things related to the njb3_get_keys() command, enabling
  DRM:ed WMA files (see below).
* Smartvolume metadata addition to transfered files.
* Firmware upgrade (see above).


Smartvolume:
------------

Smartvolume exist on the series 3-devices only, except for the
Dell Digital Jukebox which don't have it. The inner workings of 
this option is not understood, and it is not mentioned in any
Creative manuals either.

The Creative software scans the file before transfer and after
transferring the track append a large metadata chunk of 0x1b6
bytes to the file. This metadata contain the smartvolume 
parameters for the file. The format of this metadata is not
understood, and as a consequence, we do not support smartvolume
management. (Few use it anyway.)


WMA encryption keys:
--------------------

At one time there was a discussion on the list about the
mysterious "AR00PL00SG00" command that has appeared on the newer
(series 3) devices. The "AR00", "PL00" and "SG00" are keys
for one 64-bit number each, totalling 192 bits. Newer devices, 
e.g. the Nomad Zen NX, has an "LG00" parameter as well, totalling
256 bits of information.

It is suspected that these keys may have something to do with
the Windows Media Audio (WMA) digital rights management (DRM)
system, which will most likely encrypt the files using the
Device ID and a previous seed from the jukebox. (So that the
keys seen contain the seed.) A lot of reading from the keys 
have been noticed when transfering DRM WMA files, and the key
values change after transfering such files to the device.

The libnjb will currently read in and parse these keys, but 
won't do anything with them. It is possible to retrieve the
keys from the API if you know how to encrypt the files using 
them.


Mysterious commands:
--------------------

In earlier firmware a command known as 0010 0001 was sent after
each successful MP3 file transfer. It consisted of the second 
32-bit part of the "AR00" key, the first word would not be sent, 
but was set to zero. God knows why. If you have a good 
guess, then please tell us. In newer software from Creative, 
this command is no longer sent, and omitting it has proven to be
perfectly safe.

Such a parameter will typically look like this

 0010 0001			Command
 0000 0000			Zero?
 000a				Length of this command
 0014				Metadata "0014"
 0000 0000 			32-bit number always 0x00000000
 0007 a3bd			32-bit number, latter AR00 key
 0000				Terminator


SDMI compliant device ID:
-------------------------

The 16 byte (128 bit) long device ID of the Nomad Jukeboxes 
were introduced to be SDMI compliant. The specification for
SDMI compliant portable devices says that a device ID shall be
atleast 32 bits if assigned by an authority, and atleast 128
bits if assigned randomly. I do not know if Creative actually
assign device ID:s at "random", but they behave as if they do.

Not that the Jukebox is SDMI compliant. It isn't. Nothing is,
really, because SDMI failed. Microsoft, however, probably 
use parts of the ideas from SDMI in their DRM technology for 
WMA, so the SDMI device ID is used alongside the keys from the
"AR00PL00SG00" command when encrypting WMA files before 
transfer to a device such as a Nomad Jukebox.