File: README

package info (click to toggle)
ffindex 0.9.9.9-2
  • links: PTS, VCS
  • area: main
  • in suites: buster
  • size: 488 kB
  • sloc: ansic: 2,382; makefile: 163; perl: 86; python: 56; sh: 21; ruby: 6
file content (93 lines) | stat: -rw-r--r-- 3,077 bytes parent folder | download | duplicates (4)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
* Copyright

FFindex was written by Andreas Hauser <Andreas.Hauser@LMU.de>.
Please add your name here if you distribute modified versions.

FFindex is provided under the Create Commons license "Attribution-ShareAlike 4.0",
which basically captures the spirit of the Gnu Public License (GPL).

See:
http://creativecommons.org/licenses/by-sa/4.0/

* Thanks

Thanks to Laszlo Kajan for creating and maintaining Debian packages
and many suggestions to improve the build and user experience.


* Overview

FFindex is a very simple index/database for huge amounts of small files. The
files are stored concatenated in one big data file, seperated by '\0'. A second
file contains a plain text index, giving name, offset and length of of the
small files. The lookup is currently done with a binary search on an array made
from the index file.


* Installation
 
$ cd src
$ make
$ make test

If you have MPI and want to compile ffindex_apply_mpi:
$ make HAVE_MPI=1

On OS X use for the first make line:
$ make -f Makefile.osx

# Please use a sensible value for INSTALL_DIR, e.g. /usr/local or /opt/ffindex
# or $HOME/ffindex instead of "..".
$ make install INSTALL_DIR=.. 

and with MPI:

$ make install INSTALL_DIR=.. HAVE_MPI=1


* Usage

Please note that before querying or unlinking entries a ffindex must be
sorted, although you can add to it without. So either specify -s with
ffindex_build or sorted later with ffindex_modify -s.
Also the length of the entry names is restricted. See the usage output of
the ffindex_build program.

Setup environment:
$ export PATH="$INSTALL_DIR/bin:$PATH"
$ export LD_LIBRARY_PATH="$INSTALL_DIR/lib:$LD_LIBRARY_PATH"
On OS X set DYLD_LIBRARY_PATH instead of LD_LIBRARY_PATH.

Build index from files in test/data and test/data2.
$ ffindex_build -s /tmp/test.data /tmp/test.ffindex test/data test/data2

Retrieve three entries:
$ ffindex_get  /tmp/test.data /tmp/test.ffindex a b foo

Unlink (Remove reference from index) an entry:
$ ffindex_modify -u /tmp/test.ffindex b

Retrieve three entries, "b" should now be missing:
$ ffindex_get /tmp/test.data /tmp/test.ffindex a b foo

Convert a Fasta file to ffindex, entry names are incerental IDs starting from 1:
$ ffindex_from_fasta -s fasta.ffdata fasta.ffindex NC_007779.ffn

Get first entry by name:
$ ffindex_get fasta.ffdata fasta.ffindex 1

Get first and third entry by entry index, this a little faster:
$ ffindex_get fasta.ffdata fasta.ffindex -n 1 3

Count the characters including header in each entry:
$ ffindex_apply fasta.ffdata fasta.ffindex wc -c

Count the number of characters in each sequence, without the header:
$ ffindex_apply fasta.ffdata fasta.ffindex perl -ne '$x += length unless(/^>/); END{print "$x\n"}'

Parallel version for counting the characters including header in each entry:
$ mpirun -np 4 ffindex_apply_mpi fasta.ffdata fasta.ffindex -- wc -c

Parallel version for counting the characters including header in each entry and
saving the output to a new ffindex:
$ mpirun -np 4 ffindex_apply_mpi fasta.ffdata fasta.ffindex -i out-wc.ffindex -d out-wc.ffdata -- wc -c