File: swarm.1

package info (click to toggle)
swarm-cluster 3.1.5%2Bdfsg-2
  • links: PTS, VCS
  • area: main
  • in suites: trixie
  • size: 1,500 kB
  • sloc: cpp: 7,517; python: 200; makefile: 128; sh: 13
file content (827 lines) | stat: -rw-r--r-- 35,231 bytes parent folder | download
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
.\" ============================================================================
.TH swarm 1 "March 31, 2024" "version 3.1.5" "USER COMMANDS"
.\" ============================================================================
.SH NAME
swarm \(em find clusters of nearly-identical nucleotide amplicons
.\" ============================================================================
.\" swarm version and help
.SH SYNOPSIS
.SY swarm
.B \-h|v
.YS
.PP
.\" swarm default usage
High-precision clustering:
.SY swarm
.RI [ filename ]
.YS
.\" swarm d = 1
.SY swarm
.RB [ \-d " 1" ]
.OP \-nrz
.OP \-a int
.OP \-i filename
.OP \-j filename
.OP \-l filename
.OP \-o filename
.OP \-s filename
.OP \-t int
.OP \-u filename
.OP \-w filename
.RI [ filename ]
.YS
.\" swarm fastidious
.SY swarm
.RB [ \-d " 1" ]
.B \-f
.OP \-nrz
.OP \-a int
.OP \-b int
.OP \-c|y int
.OP \-i filename
.OP \-j filename
.OP \-l filename
.OP \-o filename
.OP \-s filename
.OP \-t int
.OP \-u filename
.OP \-w filename
.RI [ filename ]
.YS
.PP
.\" swarm d > 1
Conservative clustering:
.SY swarm
.RB \-d " 2+"
.OP \-nrxz
.OP \-a int
.OP \-e int
.OP \-g int
.OP \-i filename
.OP \-l filename
.OP \-m int
.OP \-o filename
.OP \-p int
.OP \-s filename
.OP \-t int
.OP \-u filename
.OP \-w filename
.RI [ filename ]
.YS
.PP
.\" swarm d = 0
Dereplication (merge strictly identical sequences):
.SY swarm
.RB \-d " 0"
.OP \-rz
.OP \-a int
.OP \-i filename
.OP \-l filename
.OP \-o filename
.OP \-s filename
.OP \-u filename
.OP \-w filename
.RI [ filename ]
.YS
.\" ============================================================================
.SH DESCRIPTION
Environmental or clinical molecular studies generate large volumes of
amplicons (e.g., 16S or 18S SSU-rRNA sequences) that need to be
grouped into clusters. Traditional clustering methods are based on
greedy, input-order dependent algorithms, with arbitrary selection of
cluster centroids and cluster limits (often 97%-similarity). To
address that problem, we developed \fBswarm\fR, a fast and robust
method that recursively groups amplicons with \fId\fR or less
differences (i.e. substitutions, insertions or deletions). \fBswarm\fR
produces natural and stable clusters centered on local peaks of
abundance, mostly free from input-order dependency induced by centroid
selection.
.PP
Exact clustering is impractical on large data sets when using a naïve
all-vs-all approach (more precisely a 2-combination without
repetitions), as it implies unrealistic numbers of pairwise
comparisons. \fBswarm\fR is based on a maximum number of differences
\fId\fR between two amplicons, and focuses only on very close local
relationships. For \fId\fR = 1, the default value, \fBswarm\fR uses an
algorithm of linear complexity that generates all possible single
mutations and performs exact-string matching by comparing
hash-values. For \fId\fR = 2 or greater, \fBswarm\fR uses an algorithm
of quadratic complexity that performs pairwise string comparisons. An
efficient \fIk\fR-mer-based filtering and an astute use of comparisons
results obtained during the clustering process allows \fBswarm\fR to
avoid most of the amplicon comparisons needed in a naïve approach. To
speed up the remaining amplicon comparisons, \fBswarm\fR implements an
extremely fast Needleman-Wunsch algorithm making use of the Streaming
SIMD Extensions (SSE2) of x86-64 CPUs, NEON instructions of ARM64
CPUs, or Altivec/VMX instructions of POWER8 CPUs. If SSE2 instructions
are not available, \fBswarm\fR exits with an error message.
.PP
\fBswarm\fR can read nucleotide amplicons in fasta format from a
normal file or from the standard input (using a pipe or a
redirection). The amplicon \fIheader\fR is defined as the string
comprised between the '>' symbol and the first space or the end of the
line, whichever comes first. Each header must end with an \fIabundance
annotation\fR representing the amplicon copy number and defined as '_'
followed by a positive integer. See option \-z for input data using
usearch/vsearch's abundance annotation format
(';size=\fIinteger\fR[;]'). Once stripped from the abundance
annotation, the remaining part of the header is call the
\fIlabel\fR. In summary, using regular expression patterns:
.PP
.ce 1
\f[CR]>header[[:blank:]]\f[]   and   \f[CR]header = label_[1-9][0-9]*$\f[]
.ce 0
.PP
Abundance annotations play a crucial role in the clustering process,
and swarm exits with an error message if that information is not
available. As \fBswarm\fR outputs lists of amplicon labels, amplicon
labels must be unique to avoid any ambiguity; swarm exits with an
error message if labels are not unique. The amplicon sequence is
defined as a string of [ACGT] or [ACGU] symbols (case insensitive, 'U'
is replaced with 'T' internally), starting after the end of the header
line and ending before the next header line or the file end;
\fBswarm\fR silently removes newline symbols ('\\n' or '\\r') and
exits with an error message if any other symbol is present. Accepted
sequence lengths range from 1 nucleotide to 67 million
nucleotides. Please note that processing 67-Mb sequences requires at
least 32 gigabytes of memory. Lastly, if sequences are not all unique,
i.e. were not properly dereplicated, swarm will exit with an error
message.
.PP
Clusters are written to output files (specified with \-i, \-o, \-s and
\-u) by decreasing abundance of their seed sequences, and then by
alphabetical order of seed sequence labels. An exception to that is
the \-w (\-\-seeds) output, which is sorted by decreasing \fIcluster
abundance\fR (sum of abundances of all sequences in the cluster), and
then by alphabetical order of seed sequence labels. This is
particularly useful for post-clustering steps, such as \fIde novo\fR
chimera detection, that require clusters to be sorted by decreasing
abundances.
.\" ----------------------------------------------------------------------------
.SS General options
.TP 9
.B \-h\fP,\fB\ \-\-help
display this help and exit successfully.
.TP
.BI \-t\fP,\fB\ \-\-threads\~ "positive integer"
number of computation threads to use. Values between 1 and 512 are
accepted, but we recommend to use a number of threads lesser or equal
to the number of available CPU cores. Default number of threads is 1.
.TP
.B \-v\fP,\fB\ \-\-version
output version information and exit successfully.
.TP
.B \-\-
delimit the option list. Later arguments, if any, are treated as
operands even if they begin with '\-'. For example, 'swarm \-\-
\-file.fasta' reads from the file '\-file.fasta'.
.\" This is a POSIX requirement for all utilities
.\" (see POSIX chapter 12.02, guideline 10).
.LP
.\" ----------------------------------------------------------------------------
.SS Clustering options
.TP 9
.BI \-d\fP,\fB\ \-\-differences\~ "zero or positive integer"
maximum number of differences allowed between two amplicons, meaning
that two amplicons will be grouped if they have \fIinteger\fR (or
less) differences. This is \fBswarm\fR's most important parameter. The
number of differences is calculated as the number of mismatches
(substitutions, insertions or deletions) between the two amplicons
once the optimal pairwise global alignment has been found
(see 'pairwise alignment advanced options' to influence that step).

Any \fIinteger\fR from 0 to 255 can be used, but high \fId\fR values
will decrease the taxonomical resolution of \fBswarm\fR
results. Commonly used \fId\fR values are 1, 2 or 3, rarely
higher. When using \fId\fR = 0, \fBswarm\fR will output results
corresponding to a strict dereplication of the dataset, i.e. merging
identical amplicons. Warning, whatever the \fId\fR value, \fBswarm\fR
requires fasta entries to present abundance values. Default number of
differences \fId\fR is 1.
.TP
.B \-n\fP,\fB\ \-\-no\-otu\-breaking
when working with \fId\fR = 1, deactivate the built-in cluster
refinement (not recommended). Amplicon abundance values are used to
identify transitions among in-contact clusters and to separate them,
yielding higher-resolution clustering results. That option prevents
that separation, and in practice, allows the creation of a link
between amplicons A and B, even if the abundance of B is higher than
the abundance of A.
.LP
.\" ----------------------------------------------------------------------------
.SS Fastidious options
.TP 9
.BI \-b\fP,\fB\ \-\-boundary\~ "positive integer"
when using the option \-\-fastidious (\-f), define the minimum
abundance of what should be considered a \fIlarge\fR cluster. By
default, a cluster with a total abundance of 3 or more is considered
large. Conversely, a cluster is small if it has a total abundance of 2
or less, meaning that it is composed of either one amplicon of
abundance 2, or two amplicons of abundance 1, or one amplicon of
abundance 1. Any positive value greater than 1 can be specified. Using
higher boundary values can reduce the number of clusters (up to a
point), and will reduce the taxonomical resolution of \fBswarm\fR
results. It will also slightly increase computation time.
.TP
.BI \-c\fP,\fB\ \-\-ceiling\~ "positive integer"
when using the option \-\-fastidious (\-f), define \fBswarm\fR's
maximum memory footprint (in megabytes). \fBswarm\fR will adjust the
\-\-bloom\-bits (\-y) value of the Bloom filter to fit within the
specified amount of memory. Values accepted range from 40 to
1,073,741,824 megabytes. See the \-\-bloom\-bits (\-y) option for an
alternative way to control the memory footprint.
.TP
.B \-f\fP,\fB\ \-\-fastidious
when working with \fId\fR = 1, perform a second clustering pass to
reduce the number of small clusters (recommended option). During the
first clustering pass, an intermediate amplicon can be missing for
purely stochastic reasons, interrupting the aggregation process. The
fastidious option will create virtual amplicons, allowing to graft
small clusters upon larger ones. By default, a cluster is considered
large if it has a total abundance of 3 or more (see the \-\-boundary
option to modify that value).

To speed things up, \fBswarm\fR uses a Bloom filter to store
intermediate results. Warning, the second clustering pass can be 2 to
3 times slower than the first pass and requires much more memory to
store the virtual amplicons in Bloom filters. See the options
\-\-bloom\-bits (\-y) or \-\-ceiling (\-c) to control the memory
footprint of the Bloom filter.

The fastidious option modifies clustering results: the output files
produced by the options \-\-log (\-l), \-\-output\-file (\-o),
\-\-mothur (\-r), \-\-uclust\-file, and \-\-seeds (\-w) are updated to
reflect these modifications; the file \-\-statistics\-file (\-s) is
partially updated (columns 6 and 7 are not updated); the output file
\-\-internal\-structure (\-i) is partially updated (column 5 is not
updated for amplicons that belonged to the small cluster).
.TP
.BI \-y\fP,\fB\ \-\-bloom\-bits\~ "positive integer"
when using the option \-\-fastidious (\-f), define the size (in bits)
of each entry in the Bloom filter. That option allows to balance the
efficiency (i.e. speed) and the memory footprint of the Bloom
filter. Large values will make the Bloom filter more efficient but
will require more memory. Any value between 2 and 64 can be
used. Default value is 16. See the \-\-ceiling (\-c) option for an
alternative way to control the memory footprint.
.LP
.\" ----------------------------------------------------------------------------
.SS Input/output options
.TP 9
.BI \-a\fP,\fB\ \-\-append\-abundance\~ "positive integer"
set abundance value to use when some or all amplicons in the input
file lack abundance values (_\fIinteger\fR, or ;size=\fIinteger\fR;
when using \-z). Warning, it is not recommended to use \fBswarm\fR on
datasets where abundance values are all identical. We provide that
option as a courtesy to advanced users, please use it
carefully. \fBswarm\fR exits with an error message if abundance values
are missing and if this option is not used.
.TP
.BI \-i\fP,\fB\ \-\-internal\-structure \0filename
output all pairs of nearly-identical amplicons to \fIfilename\fR using
a five-column tab-delimited format:
.RS
.RS
.nr step 1 1
.IP \n[step]. 4
amplicon A label (header without abundance annotations).
.IP \n+[step].
amplicon B label (header without abundance annotations).
.IP \n+[step].
number of differences between amplicons A and B (\fIpositive
integer\fR).
.IP \n+[step].
cluster number (\fIpositive integer\fR). Clusters are numbered in
their order of delineation, starting from 1. All pairs of amplicons
belonging to the same cluster will receive the same number.
.IP \n+[step].
cummulated number of steps from the cluster seed to amplicon B
(\fIpositive integer\fR). When using the option \-\-fastidious (\-f),
the actual number of steps between grafted amplicons and the cluster
seed cannot be re-computed efficiently and is always set to 2 for the
amplicon pair linking the small cluster to the large
cluster. Cummulated number of steps in the small cluster (if any) are
left unchanged.
.RE
.RE
.TP
.BI \-j\fP,\fB\ \-\-network\-file \0filename
(advanced users) when working with \fId\fR = 1, dump raw amplicon
network to \fIfilename\fR using a two-column tab-delimited table of
headers with abundance annotations. Each line represents a connection
between two similar amplicons, from the most abundant to the lesser
abundant. When amplicons have the same abundance value, connections
are bi-directional and are represented on two lines: A to B, then B to
A.

In order to delineate clusters and to compute the equivalent of a
minimal spanning tree for each cluster (see option
\-\-internal\-structure), swarm first builds a network of similar
amplicons. This option is for advanced users who would like to explore
this raw network.
.TP
.BI \-l\fP,\fB\ \-\-log \0filename
output all messages to \fIfilename\fR instead of \fIstandard error\fR,
with the exception of error messages of course. That option is useful
in situations where writing to \fIstandard error\fR is problematic
(for example, with certain job schedulers).
.TP
.BI \-o\fP,\fB\ \-\-output\-file \0filename
output clustering results to \fIfilename\fR. Results consist of a list
of clusters, one cluster per line. A cluster is a list of amplicon
headers separated by spaces. That output format can be modified by the
option \-\-mothur (\-r). Default is to write to \fIstandard output\fR.
.TP
.B \-r\fP,\fB\ \-\-mothur
output clustering results in a format compatible with Mothur. That
option modifies \fBswarm\fR's default output format.
.TP
.BI \-s\fP,\fB\ \-\-statistics\-file \0filename
output statistics to \fIfilename\fR. The file is a tab-separated table
with one cluster per row and seven columns of information:
.RS
.RS
.nr step 1 1
.IP \n[step]. 4
number of unique amplicons in the cluster,
.IP \n+[step].
total abundance of amplicons in the cluster,
.IP \n+[step].
label of the initial seed (header without abundance annotations),
.IP \n+[step].
abundance of the initial seed,
.IP \n+[step].
number of amplicons with an abundance of 1 in the cluster,
.IP \n+[step].
maximum number of iterations before the cluster reached its natural limit,
.IP \n+[step].
cummulated number of steps along the path joining the seed and the
furthermost amplicon in the cluster. Please note that the actual
number of differences between the seed and the furthermost amplicon is
usually much smaller. When using the option \-\-fastidious (\-f),
grafted amplicons are not taken into account.
.RE
.RE
.TP
.BI \-u\fP,\fB\ \-\-uclust\-file \0filename
output clustering results in \fIfilename\fR using a tab-separated
uclust-like format with 10 columns and 3 different type of entries (S,
H or C). That option does not modify \fBswarm\fR's default output
format. Each fasta sequence in the input file can be either a cluster
centroid (S) or a hit (H) assigned to a cluster. Cluster records (C)
summarize information for each cluster (number of hits, centroid
header). Column content varies with the type of entry (S, H or C):
.RS
.RS
.nr step 1 1
.IP \n[step]. 4
Record type: S, H, or C.
.IP \n+[step].
Cluster number (zero-based).
.IP \n+[step].
Centroid length (S), query length (H), or number of hits (C).
.IP \n+[step].
Percentage of similarity with the centroid sequence (H), or set to '*'
(S, C).
.IP \n+[step].
Match orientation + or - (H), or set to '*' (S, C).
.IP \n+[step].
Not used, always set to '*' (S, C) or to zero (H).
.IP \n+[step].
Not used, always set to '*' (S, C) or to zero (H).
.IP \n+[step].
set to '*' (S, C) or, for H, compact representation of the pairwise
alignment using the CIGAR format (Compact Idiosyncratic Gapped
Alignment Report): M (match), D (deletion) and I (insertion). The
equal sign '=' indicates that the query is identical to the centroid
sequence.
.IP \n+[step].
Header of the query sequence (H), or of the centroid sequence (S, C).
.IP \n+[step].
Header of the centroid sequence (H), or set to '*' (S, C).
.RE
.RE
.TP
.BI \-w\fP,\fB\ \-\-seeds \0filename
output cluster representative sequences to \fIfilename\fR in fasta
format. The abundance value of each cluster representative is the sum
of the abundances of all the amplicons in the cluster. Fasta headers
are formated as follows: '>label_\fIinteger\fR',
or '>label;size=\fIinteger\fR;' if the \-z option is used, and
sequences are uppercased. Sequences are sorted by decreasing
abundance, and then by alphabetical order of sequence labels.
.TP
.B \-z\fP,\fB\ \-\-usearch\-abundance
accept amplicon abundance values in usearch/vsearch's style
(>label;size=\fIinteger\fR[;]). That option influences the abundance
annotation style used in swarm's \fIstandard output\fR (\-o), as well
as the output of options \-r, \-u and \-w.
.LP
.\" ----------------------------------------------------------------------------
.SS Pairwise alignment advanced options
when using \fId\fR > 1, \fBswarm\fR recognizes advanced command-line
options modifying the pairwise global alignment scoring parameters:
.RS
.TP 9
.BI \-m\fP,\fB\ \-\-match\-reward\~ "positive integer"
Default reward for a nucleotide match is 5.
.TP
.BI \-p\fP,\fB\ \-\-mismatch\-penalty\~ "positive integer"
Default penalty for a nucleotide mismatch is 4.
.TP
.BI \-g\fP,\fB\ \-\-gap\-opening\-penalty\~ "positive integer"
Default gap opening penalty is 12.
.TP
.BI \-e\fP,\fB\ \-\-gap\-extension\-penalty\~ "positive integer"
Default gap extension penalty is 4.
.TP
.B \-x\fP,\fB\ \-\-disable\-sse3
On the x86-64 CPU architecture, disable SSE3 and later
instructions. This option is meant for developers, not for regular
users.
.LP
.RE
As \fBswarm\fR focuses on close relationships (e.g., \fId\fR = 2 or
3), clustering results are resilient to pairwise alignment model
parameters modifications. When clustering using a higher \fId\fR
value, modifying model parameters has a stronger impact.
.\" classic parameters are +5/-4/-12/-1
.\" ============================================================================
.SH EXAMPLES
.PP
Clusterize the compressed data set \fImyfile.fasta\fR using the finest
resolution possible (1 difference by default, built-in breaking,
fastidious option) using 4 computation threads. Clusters are written
to the file \fImyfile.swarms\fR, and cluster representatives are
written to \fImyfile.representatives.fasta\fR:
.EX
.RS
zcat myfile.fasta.gz | \\
    swarm \\
        \-t 4 \\
        \-f \\
        \-w myfile.representatives.fasta \\
        \-o myfile.swarms
.RE
.EE
.\" ============================================================================
.\" .SH LIMITATIONS
.\" List known limitations or bugs.
.\" ============================================================================
.SH AUTHORS
Concept by Frédéric Mahé, implementation by Torbjørn Rognes.
.\" ============================================================================
.SH CITATION
Mahé F, Rognes T, Quince C, de Vargas C, Dunthorn M. (2014) Swarm:
robust and fast clustering method for amplicon-based studies.
\fIPeerJ\fR 2:e593
.UR https://doi.org/10.7717/peerj.593
.UE .
.PP
Mahé F, Rognes T, Quince C, de Vargas C, Dunthorn M. (2015) Swarm v2:
highly-scalable and high-resolution amplicon clustering.  \fIPeerJ\fR
3:e1420
.UR https://doi.org/10.7717/peerj.1420
.UE .
.PP
Mahé F, Czech L, Stamatakis A, Quince C, de Vargas C, Dunthorn M, Rognes T. (2021)
Swarm v3: towards tera-scale amplicon clustering.  \fIBioinformatics\fR
.UR https://doi.org/10.1093/bioinformatics/btab493
.UE .
.\" ============================================================================
.SH REPORTING BUGS
Submit suggestions and bug-reports at
.UR https://github.com/torognes/swarm/issues
.UE ,
send a pull request at
.UR https://github.com/torognes/swarm/pulls
.UE ,
or compose a friendly or curmudgeonly e-mail to
.MT frederic.mahe@cirad.fr
Frédéric Mahé
.ME
and
.MT torognes@ifi.uio.no
Torbjørn Rognes
.ME .
.\" ============================================================================
.SH AVAILABILITY
Source code and binaries available at
.UR https://github.com/torognes/swarm
.UE .
.\" ============================================================================
.SH COPYRIGHT
Copyright (C) 2012-2024 Frédéric Mahé & Torbjørn Rognes
.PP
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU Affero General Public License as
published by the Free Software Foundation, either version 3 of the
License, or any later version.
.PP
This program is distributed in the hope that it will be useful, but
WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
Affero General Public License for more details.
.PP
You should have received a copy of the GNU Affero General Public
License along with this program.  If not, see
.UR https://www.gnu.org/licenses/
.UE .
.PP
.\" ============================================================================
.SH SEE ALSO
\fBswipe\fR, an extremely fast Smith-Waterman database search tool by
Torbjørn Rognes (available at
.UR https://github.com/torognes/swipe
.UE ).
.PP
\fBvsearch\fR, an open-source re-implementation of the classic uclust
clustering method (by Robert C. Edgar), along with other amplicon
filtering and searching tools. \fBvsearch\fR is implemented by
Torbjørn Rognes and documented by Frédéric Mahé, and is available at
.UR https://github.com/torognes/vsearch
.UE .
.PP
.\" ============================================================================
.SH VERSION HISTORY
New features and important modifications of \fBswarm\fR (short lived
or minor bug releases are not mentioned):
.RS
.TP
.BR v3.1.5\~ "released March 31, 2024"
Version 3.1.5 changes the minimal value for the ceiling option from 8
megabytes to 40 megabytes, and fixes four minor bugs. Warning, peak
RSS memory increased by 5 to 10% when \fId\fR >= 2. Version 3.1.5
improves documentation (now covering option \-\-network_file), adds
more compilation checks and eliminates 50 compilation warnings with
GCC 13, GCC 14 and clang 19, as well as 1,677 static analysis
warnings.
.TP
.BR v3.1.4\~ "released September 20, 2023"
Version 3.1.4 fixes a minor bug. It eliminates compilation warnings
with GCC 13 and clang 18, as well as 1,040 static analysis
warnings. The maximal number of threads swarm can run is now 512,
instead of 256. Compilation with runtime checks (`-DNDEBUG`) is now
the default. When d > 1, overall memory allocations remain unchanged,
but peak RSS memory increased by 6 to 10%, due to a change in the
timing of memory deallocations. Peak RSS memory is expected to regress
to its prior levels as refactoring continues.
.TP
.BR v3.1.3\~ "released December 5, 2022"
Version 3.1.3 fixes a regression introduced in version 3.1.1 (memory
over-allocation when d > 1). It also fixes a minor off-by-one error
when allocating memory for a Bloom filter, compilation warnings with
GCC 12 and clang 13, as well as static analysis
warnings. Documentation was improved, as well as our test suite
(swarm-tests).
.TP
.BR v3.1.2\~ "released November 10, 2022"
Fix a bug with fastidious mode introduced in version 3.1.1, that could
cause Swarm to crash. Probably due to allocating too much memory.
.TP
.BR v3.1.1\~ "released September 29, 2022"
Version 3.1.1 eliminates a risk of segmentation fault with extremely
long sequence headers. Documentation and error messages have been
improved, and code cleaning continued.
.TP
.BR v3.1.0\~ "released March 1, 2021"
Version 3.1.0 includes a fix for a bug in the 16-bit SIMD alignment
code that was exposed with a combination of d>1, long sequences, and
very high gap penalties. The code has also been been cleaned up,
tested and improved substantially, and it is now fully C++11
compliant. Support for macOS on Apple Silicon (ARM64) has been added.
.TP
.BR v3.0.0\~ "released October 24, 2019"
Version 3.0.0 introduces a faster algorithm for \fId\fR = 1, and a
reduced memory footprint. Swarm has been ported to Windows x86-64,
GNU/Linux ARM 64, and GNU/Linux POWER8. Internal code has been
modernized, hardened, and thoroughly tested. Strict dereplication of
input sequences is now mandatory. The \-\-seeds option (\-w) now
outputs results sorted by decreasing abundance, and then by
alphabetical order of sequence labels.
.TP
.BR v2.2.2\~ "released December 12, 2017"
Version 2.2.2 fixes a bug that would cause swarm to wait forever in
very rare cases when multiple threads were used.
.TP
.BR v2.2.1\~ "released October 27, 2017"
Version 2.2.1 fixes a memory allocation bug for \fId\fR = 1 and
duplicated sequences.
.TP
.BR v2.2.0\~ "released October 17, 2017"
Version 2.2.0 fixes several problems and improves usability. Corrected
output to structure and uclust files when using fastidious
mode. Corrected abundance output in some cases. Added check for
duplicated sequences and fixed check for duplicated sequence
IDs. Checks for empty sequences. Sorts sequences by additional fields
to improve stability. Improves compatibility with compilers and
operating systems.  Outputs sequences in upper case. Allows 64-bit
abundances. Shows message when waiting for input from stdin. Improves
error messages and warnings. Improves checking of command line
options. Fixes remaining errors reported by test suite. Updates
documentation.
.TP
.BR v2.1.13\~ "released March 8, 2017"
Version 2.1.13 removes a bug with the progress bar when writing seeds.
.TP
.BR v2.1.12\~ "released January 16, 2017"
Version 2.1.12 removes a debugging message.
.TP
.BR v2.1.11\~ "released January 16, 2017"
Version 2.1.11 fixes two bugs related to the SIMD implementation of
alignment that might result in incorrect alignments and scores.  The
bug only applies when \fId\fR > 1.
.TP
.BR v2.1.10\~ "released December 22, 2016"
Version 2.1.10 fixes two bugs related to gap penalties of alignments.
The first bug may lead to wrong aligments and similarity percentages
reported in UCLUST (.uc) files. The second bug makes swarm use a
slightly higher gap extension penalty than specified. The default gap
extension penalty used have actually been 4.5 instead of 4.
.TP
.BR v2.1.9\~ "released July 6, 2016"
Version 2.1.9 fixes errors when compiling with GCC version 6.
.TP
.BR v2.1.8\~ "released March 11, 2016"
Version 2.1.8 fixes a rare bug triggered when clustering extremely
short undereplicated sequences. Also, alignment parameters are not
shown when \fId\fR = 1.
.TP
.BR v2.1.7\~ "released February 24, 2016"
Version 2.1.7 fixes a bug in the output of seeds with the \-w option
when \fId\fR > 1 that was not properly fixed in version 2.1.6. It also
handles ascii character #13 (CR) in FASTA files better. Swarm will now
exit with status 0 if the \-h or the \-v option is specified. The help
text and some error messages have been improved.
.TP
.BR v2.1.6\~ "released December 14, 2015"
Version 2.1.6 fixes problems with older compilers that do not have the
x86intrin.h header file. It also fixes a bug in the output of seeds
with the \-w option when \fId\fR > 1.
.TP
.BR v2.1.5\~ "released September 8, 2015"
Version 2.1.5 fixes minor bugs.
.TP
.BR v2.1.4\~ "released September 4, 2015"
Version 2.1.4 fixes minor bugs in the swarm algorithm used for \fId\fR
= 1.
.TP
.BR v2.1.3\~ "released August 28, 2015"
Version 2.1.3 adds checks of numeric option arguments.
.TP
.BR v2.1.1\~ "released March 31, 2015"
Version 2.1.1 fixes a bug with the fastidious option that caused it to
ignore some connections between large and small clusters.
.TP
.BR v2.1.0\~ "released March 24, 2015"
Version 2.1.0 marks the first official release of swarm v2.
.TP
.BR v2.0.7\~ "released March 18, 2015"
Version 2.0.7 writes abundance information in usearch style when using
options \-w (\-\-seeds) in combination with \-z
(\-\-usearch\-abundance).
.TP
.BR v2.0.6\~ "released March 13, 2015"
Version 2.0.6 fixes a minor bug.
.TP
.BR v2.0.5\~ "released March 13, 2015"
Version 2.0.5 improves the implementation of the fastidious option and
adds options to control memory usage of the Bloom filter (\-y and
\-c).  In addition, an option (\-w) allows to output cluster
representatives sequences with updated abundances (sum of all
abundances inside each cluster). This version also enables \fBswarm\fR
to run with \fId\fR = 0.
.TP
.BR v2.0.4\~ "released March 6, 2015"
Version 2.0.4 includes a fully parallelised implementation of the
fastidious option.
.TP
.BR v2.0.3\~ "released March 4, 2015"
Version 2.0.3 includes a working implementation of the fastidious
option, but only the initial clustering is parallelized.
.TP
.BR v2.0.2\~ "released February 26, 2015"
Version 2.0.2 fixes SSSE3 problems.
.TP
.BR v2.0.1\~ "released February 26, 2015"
Version 2.0.1 is a development version that contains a partial
implementation of the fastidious option, but it is not usable yet.
.TP
.BR v2.0.0\~ "released December 3, 2014"
Version 2.0.0 is faster and easier to use, providing new output
options (\-\-internal\-structure and \-\-log), new control options
(\-\-boundary, \-\-fastidious, \-\-no\-otu\-breaking), and built-in
cluster refinement (no need to use the python script anymore). When
using default parameters, a novel and considerably faster algorithmic
approach is used, guaranteeing \fBswarm\fR's scalability.
.TP
.BR v1.2.21\~ "released February 26, 2015"
Version 1.2.21 is supposed to fix some problems related to the use of
the SSSE3 CPU instructions which are not always available.
.TP
.BR v1.2.20\~ "released November 6, 2014"
Version 1.2.20 presents a production-ready version of the alternative
algorithm (option \-a), with optional built-in cluster breaking
(option \-n). That alternative algorithmic approach (usable only with
\fId\fR = 1) is considerably faster than currently used clustering
algorithms, and can deal with datasets of 100 million unique amplicons
or more in a few hours. Of course, results are rigourously identical
to the results previously produced with swarm. That release also
introduces new options to control swarm output (options \-i and \-l).
.TP
.BR v1.2.19\~ "released October 3, 2014"
Version 1.2.19 fixes a problem related to abundance information when
the sequence label includes multiple underscore characters.
.TP
.BR v1.2.18\~ "released September 29, 2014"
Version 1.2.18 reenables the possibility of reading sequences from
\fIstdin\fR if no file name is specified on the command line. It also
fixes a bug related to CPU features detection.
.TP
.BR v1.2.17\~ "released September 28, 2014"
Version 1.2.17 fixes a memory allocation bug introduced in version
1.2.15.
.TP
.BR v1.2.16\~ "released September 27, 2014"
Version 1.2.16 fixes a bug in the abundance sort introduced in version
1.2.15.
.TP
.BR v1.2.15\~ "released September 27, 2014"
Version 1.2.15 sorts the input sequences in order of decreasing
abundance unless they are detected to be sorted already. When using
the alternative algorithm for \fId\fR = 1 it also sorts all subseeds
in order of decreasing abundance.
.TP
.BR v1.2.14\~ "released September 27, 2014"
Version 1.2.14 fixes a bug in the output with the \-\-swarm_breaker
option (\-b) when using the alternative algorithm (\-a).
.TP
.BR v1.2.12\~ "released August 18, 2014"
Version 1.2.12 introduces an option \-\-alternative\-algorithm to use
an extremely fast, experimental clustering algorithm for the special
case \fId\fR = 1. Multithreading scalability of the default algorithm
has been noticeably improved.
.TP
.BR v1.2.10\~ "released August 8, 2014"
Version 1.2.10 allows amplicon abundances to be specified using the
usearch style in the sequence header (e.g. '>id;size=1') when the \-z
option is chosen.
.TP
.BR v1.2.8\~ "released August 5, 2014"
Version 1.2.8 fixes an error with the gap extension penalty. Previous
versions used a gap penalty twice as large as intended. That bug
correction induces small changes in clustering results.
.TP
.BR v1.2.6\~ "released May 23, 2014"
Version 1.2.6 introduces an option \-\-mothur to output clustering
results in a format compatible with the microbial ecology community
analysis software suite Mothur (
.UR https://www.mothur.org/
.UE ).
.TP
.BR v1.2.5\~ "released April 11, 2014"
Version 1.2.5 removes the need for a POPCNT hardware instruction to be
present. \fBswarm\fR now automatically checks whether POPCNT is
available and uses a slightly slower software implementation if
not. Only basic SSE2 instructions are now required to run \fBswarm\fR.
.TP
.BR v1.2.4\~ "released January 30, 2014"
Version 1.2.4 introduces an option \-\-break\-swarms to output all
pairs of amplicons with \fId\fR differences to \fIstandard
error\fR. That option is used by the companion script
`swarm_breaker.py` to refine \fBswarm\fR results. The syntax of the
inline assembly code is changed for compatibility with more compilers.
.TP
.BR v1.2\~ "released May 16, 2013"
Version 1.2 greatly improves speed by using alignment-free comparisons
of amplicons based on \fIk\fR-mer word content. For each amplicon, the
presence-absence of all possible 5-mers is computed and recorded in a
1024-bits vector. Vector comparisons are extremely fast and
drastically reduce the number of costly pairwise alignments performed
by \fBswarm\fR. While remaining exact, \fBswarm\fR 1.2 can be more
than 100-times faster than \fBswarm\fR 1.1, when using a single thread
with a large set of sequences. The minor version 1.1.1, published just
before, adds compatibility with Apple computers, and corrects an issue
in the pairwise global alignment step that could lead to sub-optimal
alignments.
.TP
.BR v1.1\~ "released February 26, 2013"
Version 1.1 introduces two new important options: the possibility to
output clustering results using the uclust output format, and the
possibility to output detailed statistics on each cluster. \fBswarm\fR
1.1 is also faster: new filterings based on pairwise amplicon sequence
lengths and composition comparisons reduce the number of pairwise
alignments needed and speed up the clustering.
.TP
.BR v1.0\~ "released November 10, 2012"
First public release.
.LP
.\" ============================================================================
.\" NOTES
.\" visualize and output to pdf
.\" man -l swarm.1
.\" man -t <(sed -e 's/\\-/-/g' ./swarm.1) | ps2pdf -sPAPERSIZE=a4 - > swarm_manual.pdf
.\"
.\" INSTALL (sysadmin)
.\" gzip -c swarm.1 > swarm.1.gz
.\" mv swarm.1.gz /usr/share/man/man1/