README for debtags 1.5.2
========================

Functionality
-------------

The library finally builds some API documentation using doxygen.  Not much of
the source code has useful comments yets, but since doxygen support is in
place, this situation is going to improve rapidly.

Usage examples of libdebtags can be found in the tests, in the source of the
``debtags`` tool and in the documentation.  More code using libdebtags1 can be
found in ``debtags-edit``.

The ``debtags`` tool is especially suited as an example because most of its
complexity resides in the library itself and it uses it in a very
straightforward way.

I'd still like to discuss design issues with interested people, and until when
the library will be used by a wider range of applications and will prove to be
sound enough for general use, the API is subject to changes.

Please get in touch with me if you start using libdebtags in some projects:
I would like to involve library users in design discussions and API changes.

debtags is the commandline maintenance and query tool for Debtags.  It can be
used to download up-to-date tag database and vocabulary, to query them, to edit
the categorization and to submit the changes to the central repository.


What is in this release
-----------------------

debtags and libdebtags1 have been merged in a single source package to make
maintainance easier.


Compiling from the subversion repository
----------------------------------------

Additionally to the build-dependencies, you need to install ``libtool`` and
``automake1.9`::

  apt-get install libtool automake1.9
  ./autogen.sh
  ./configure
  make

The program uses GNU Libtool.  While this is very nice, it can be a pain when
trying to debug a program. For that reason, compilation of shared libraries can
be turned off by specifying the ``--disable-shared`` option to ``configure``.


Resources
---------

  Website:

    http://debtags.alioth.debian.org

  Mailing lists:
  
    debtags-devel@lists.alioth.debian.org
    http://lists.alioth.debian.org/mailman/listinfo/debtags-devel
      Development and usage discussions
  
    debtags-commits@lists.alioth.debian.org
    http://lists.alioth.debian.org/mailman/listinfo/debtags-commmits
      Postings of subversion logs

  Subversion repository:

    http://svn.debian.org/viewcvs/debtags/libdebtags
      Browse online
    svn://svn.debian.org/debtags/libdebtags
      Read-only access
    svn+ssh://alioth.debian.org/svn/debtags/libdebtags
      Write access

  APT repository:

    deb http://debtags.alioth.debian.org/debian unstable main


Further stages of the development
---------------------------------

Development will now proceed by:

 * Slowly getting rid of the Package class, which should be handled by better
   libraries such as libapt-front
 * Making the library more generic, with more use of templates
 * Trying to find better data structure and algorithms to improve the
   efficiency of the various features provided by libdebtags1
 * Implementing more and improved statistics in ``debtags``


TODO-list items completed so far
--------------------------------

These are the TODO-list items completed so far::

--- Done in 1.4
 + Ported to libapt-front
 + Split again from libdebtags1 to avoid circular dependencies

--- Done in 1.3.2
 + Install .cc files with template code together with the headers
 + On update, don't try to rename files not present in
   /var/cache/debtags/partial like file:// links.  That used to crash debtags
   update

--- Done in 1.3.1
 + Better packaging of the documentation

--- Done in 1.3
 + Merged debtags and libdebtags1 source packages
   Big thanks to Benjamin Mesing for doing a good deal of the work.

--- Done in libdebtags1 version 1.0.8
 + Also ignore /etc/tagvoc.d

--- Done in 1.2.1
 + Recompiled with new library to fix 'update' feature
 + New bash-completion from Emanuele

--- Done in 1.2
 + Added 'search' option
 + Updated manpage
 + Perfected debian postinst handling

(older completed items are archived in the DONE file)



TODO-list items being worked on
-------------------------------

These are the TODO-list items currently being worked on::

 - Smart querying:
    1. full text search
    2. get the n (=3?) most recurrent tags (possibly multiplying their
       cardinality in the result by the tag score, to add a user-defined
       weighting)
    3. Add all the items whose tagset contains the n most recurrent tags
    4. Allow to AND or AND NOT for tags, possibly allowing the user to choose
       among the most recurrent tags
   Resulting query:
    ((Full text UNION "similar" items) UNION and items) INTERSECTION and not items

 - Allow to have ~/.debtags/vocabulary merged in
 
 - Try to get rid of the Singleton Environment

 - Remove throw declarations altogether except when the function obviously
	isn't throwing and won't be throwing anything.

 - perl bindings crash at shutdown: it looks like #186104: check what it was,
	and check how it has been corrected
	 - Does not seem to be the case:
	   TDBReadonlyDiskIndex's destructor reports garbage in the size of
		newItems, even if nothing has been ending up on it (that is,
		~/.debtags/patch has been moved aside).

 - Remove all PackageDB-related functions: libapt-front should be used for
	that.  PackageDB itself probably needs to stay for the APTFilter in Update.

 - Impossible now to detect out-of-space errors because TagcollSerializer isn't
	checking the return code of fprintf (changing it would break libtagcoll
	binary compatibility)
	(fixed in libtagcoll, should handle it appropriately in libdebtags)

 - Sending mail to central database aborts if the mailserver fails

 - See if mor standard c++ features can be used

   use of functors (``operator()``)

   The Debtags::Matcher is a good candidate for this one, having the same
   implementation like the match function. With this the code in
   TagSet::getFiltered could be implemented using a the copy_if() algorithm.
   Uhh... the above is hard to read. I mean the operator() could be added to
   the Debtags::Matcher() - where it could be used in std algorithms.
 - todoreport should mention that the number of packages has been limited

 - specials: given a facet A, for all other facets Bn generate a list of
   packages that have A and not Bn

 - generate one todoreport per maintainer

 - debtags related does not work
   ("debtags -d 14 related debtags" doesn't output anything

 - Statistics and automated checks:
    - Experiment with theory:
       - there should never be two packages with the same tagset: if it happens, a
         new facet may need to be created
       - Make a debtags function to detect when it happens

 - In grep expressions, allow to use and, or or not (hiddenly replacing them
   with &&, ||, !), now that tags always have :: in them

 - Code
    - In debtags update, print the names of packages that are not in the debtags
      archive
    - On debtags update, filter out tag data for packages not existing in the
      system
    - On debtags update, output a list of the packages not present in the tag collection
    - Usare "foreign" nell'inizializzazione di automake, in modo che non richieda il changelog
      (e cancellare il changelog) (anche negli altri pacchetti)
    - Make a CommandlineParserWithCommands supporting command-specific switches:
      addCmdSwitch("command", <same args as add()>)
      addCmdSwitch("", ..) adds a switch that gets interpreted when no valid
        command is found
      The option parsing should then scan until it finds the first non-option
        argument and see if it's a command, then re-scan the commandline with the
        good set of switches in effect.  There's a problem in this with arguments
        to short switches, like '-o file': `file' could be interpreted as a
        command and not as an option to `-o'.  Could be solved by having all
        switches be considered when scanning for the command, or (better) by
        scanning once for each command until a valid match is found

 - debtags install: Allow to call aptitude instead of apt-get

 - [erich] I had the idea that people can "adopt" a keyword, looking for
   volunteers to review a section regularly.


Future TODO-list items
----------------------

These are the TODO-list items that are to be addressed in the future::

 - Try wrapping with http://www.boost.org/libs/python/doc/index.html

 - Make a small LRU cache of Facets and Tags in Vocabulary, so that some
	TagImpl and FacetImpl are reused

 - Update doesn't really merge different tag sources
   (need more info) (seems to work for me: where does this come from?)

 - Implement failover strategies in case the vocabulary (or other files) are
	not present

 - Create new tags / facets
 - Save a vocabulary patch

 - Make a more efficient collection container than InputMerger for the
	operations performed by libdebtags

 - port foundation libraries to libboost when possible
 - Merging different tag sources
    - Scenario: I want to edit tags when I have multiple tag sources, submitting
      appropriate, separate tag patches to the tag sources which are in charge of
      the tags that changed.
       - Including a local, user defined, tag hierarchy (preference::interesting,
         preference::useless, preference::tried...)
    - Add origin to facets and save a different patch for every origin
    - [me]
      It could be interesting to outsource the first tag layout of such parts
      to the relative communities: for example, we could ask the gnome people
      to provide the right tag vocabulary for gnome-related qualities.
      I guess they may be happy to do it (and to be kept in consideration by
      the Debian people), and surely they know what are right qualities in
      their world.
 - Add Extends: apt to control file (like guessnet does with ifupdown)
 - Allow to manipulate a local vocabulary (~/.debtags/vocabulary ?)

 - Integrate with APT
    - See if it's possible to hook debtags update after apt-get update
    - Patch apt-cache to exec debtags for some things (when debtags is installed)
      [zufus]

 - When adding new tags to the vocabulary, announce them to debian-devel
   (no faster than weekly)
 - Maintain an upgrade checklist for the vocabulary.  cvs diff can help
   generating it

 - Since we're somehow similar to semantic web, have a look at what it is, how
   it works and how crossing that path can be useful

 - Navigazione: l'utente induce nella tagged collection una struttura a lui
   specifica, avvicinando categorie e allontanandone altre.  Questo può anche
   essere mappato bi o tridimensionalmente, con animazione.

 - Erich:
   This to improve in synaptic-debtags:
    - make the tree less deep, don't make subfolders if only < 10 packages are
      left etc.
    - show tag descriptions.
    - handle "virtual" tags in the tree, such as "ui", which basically is a
      union of "ui::gtk", "ui::qt", "ui::ncurses" etc. (virtual tags: tags
      where all packages are in a subgroup)

   Things to improve with debtags in general:
    - more tagging. Too many packages are still untagged
    - inconsistent tagging. New tags were added, so many tagged packages are
      incompletely tagged. For example many applications don't have a user
      interface specified.
    - inconsistent tags. some features have tags, others don't.
    - structure is becoming to deep IMHO. but if you want to keep the
      number-of-results low you need such a deep structure.

 - [Javier Fernández-Sanguino Peña]
   Have a look at TFIDF systems (implemented in bow) to use them to find out tags.
   The suggestion comes from Javier Fernández-Sanguino Peña <jfs@computer.org>,
   who's happy to help with problems understanding TFIDF
 - Ask for someone in the i18n team to join in, to help with the design of the
   language dependent files with tag names and descriptions.



Discarded TODO-list items
-------------------------

These TODO-list items have been discarded::

 = Note to the vocabulary task force: "mail,net" is a single tag, "mail, net"
   are two tags
   (this is an old note that doesn't make much sense anymore)
 = Note to the vocabulary task force: rub "debtags check" from time to time 
   (this is an old note that doesn't make much sense anymore)
 = [chlunde] Don't crash when giving data about a non-existing package: it
   might happen, such as when using the full database on a system without
   non-free.  Don't show the application, instead.
   [chlunde is working on it]
   (old note, fixed since long time on debtags-edit)
 = Consider implementing version information in Package, allowing more
	Package-s with the same name but with different versions in PackageGroups
	(can do by changing Package's comparison operators)
	 - Consider then doing dependency computations, like
	   PackageGroup PackageGroup::getDepends();
	   PackageGroup PackageGroup::getRecommends();
	   PackageGroup PackageGroup::getSuggests();
	   PackageGroup PackageGroup::getConflicts();
		...
 = Allow more flexibility in Package representations
	 - Make PackageImpl somehow more public, so that people can subclass it and
		write Package-like classes with more informations (as you can't subclass
		Package, being it a smart pointer)
	 - Make PackageSet a template, to which you can pass Package or something else
	 - Rename Package into BasicPackage
	 - Make a Package virtual interface

 = Bring in dependency info from synaptic (rpackage.h)
   Subversion: https://oops.kerneljanitors.org/repos/synaptic/trunk
	PackageSet Package::getDepends()
	PackageSet Package::getSuggests()
	PackageSet Package::getRecommends()
	PackageSet PackageSet::getDepends()
	PackageSet PackageSet::getSuggests()
	PackageSet PackageSet::getRecommends()
	(no: leave that to libapt-front)
 = Environment: accept the filter as const
 = Implement filters, searches and consumers also for MaintainerSet
 = Find out how to efficiently search a std::set<Package> given the package
	name, though: #include<algorithm> could come useful, hopefully, as it would
	come useful to see what operators are supported by the set iterator (it
	would be wonderful if it had tree operations...)
 = See how it is possible to be a base for libcapture
 = Integrate Maintainer info with Echelon data (from Debian LDAP database)
 = Do popcon researches as well

(older discarded items are archived in the DONE file)


Miscellaneous notes
-------------------

Some analysis techniques:

Consistency checks on facets that should go together::

  # Show all packages which have a tag in uitoolkit but not a tag in interface
  # (same can be repeated for hardware and hwtech, devel and langdevel)
  debtags facetcoll | grep uitoolkit | grep -v interface | cut -d: -f1 | sort | uniq

  # Count the number of facets
  debtags facetcoll | tagcoll reverse | wc -l

  # Get a list of toplevel facets in a facet hierarchy
  debtags facetcoll | tagcoll hierarchy | cut -f2 -d/ | cut -f1 -d: | sort | uniq


Interesting links:

 * http://www.itconsult.it/knowledge/articoli/detArticoli.asp?ID=18



Author
------

Enrico Zini <enrico@debian.org>