1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
1001
1002
1003
1004
1005
1006
1007
1008
1009
1010
1011
1012
1013
1014
1015
1016
1017
1018
1019
1020
1021
1022
1023
1024
1025
1026
1027
1028
1029
1030
1031
1032
1033
1034
1035
1036
1037
1038
1039
1040
1041
1042
1043
1044
1045
1046
1047
1048
1049
1050
1051
1052
1053
1054
1055
1056
1057
1058
1059
1060
1061
1062
1063
1064
1065
1066
1067
1068
1069
1070
1071
1072
1073
1074
1075
1076
1077
1078
1079
1080
1081
1082
1083
1084
1085
1086
1087
1088
1089
1090
1091
1092
1093
1094
1095
1096
1097
1098
1099
1100
1101
1102
1103
1104
1105
1106
1107
1108
1109
1110
1111
1112
1113
1114
1115
1116
1117
1118
1119
1120
1121
1122
1123
1124
1125
1126
1127
1128
1129
1130
1131
1132
1133
1134
1135
1136
1137
1138
1139
1140
1141
1142
1143
1144
1145
1146
1147
1148
1149
1150
1151
1152
1153
1154
1155
1156
1157
1158
1159
1160
1161
1162
1163
1164
1165
1166
1167
1168
1169
1170
1171
1172
1173
1174
1175
1176
1177
1178
1179
1180
1181
1182
1183
1184
1185
1186
1187
1188
1189
1190
1191
1192
1193
1194
1195
1196
1197
1198
1199
1200
1201
1202
1203
1204
1205
1206
1207
1208
1209
1210
1211
1212
1213
1214
1215
1216
1217
1218
1219
1220
1221
1222
1223
1224
1225
1226
1227
1228
1229
1230
1231
1232
1233
1234
1235
1236
1237
1238
1239
1240
1241
1242
1243
1244
1245
1246
1247
1248
1249
1250
1251
1252
1253
1254
1255
1256
1257
1258
1259
1260
1261
1262
1263
1264
1265
1266
1267
1268
1269
1270
1271
1272
1273
1274
1275
1276
1277
1278
1279
1280
1281
1282
1283
1284
1285
1286
1287
1288
1289
1290
1291
1292
1293
1294
1295
1296
1297
1298
1299
1300
1301
1302
1303
1304
1305
1306
1307
1308
1309
1310
1311
1312
1313
1314
1315
1316
1317
1318
1319
1320
1321
1322
1323
1324
1325
1326
1327
1328
1329
1330
1331
1332
1333
1334
1335
1336
1337
1338
1339
1340
1341
1342
1343
1344
1345
1346
1347
1348
1349
1350
1351
1352
1353
1354
1355
1356
1357
1358
1359
1360
1361
1362
1363
1364
1365
1366
1367
1368
1369
1370
1371
1372
1373
1374
1375
1376
1377
1378
1379
1380
1381
1382
1383
1384
1385
1386
1387
1388
1389
1390
1391
1392
1393
1394
1395
1396
1397
1398
1399
1400
1401
1402
1403
1404
1405
1406
1407
1408
1409
1410
1411
1412
1413
1414
1415
1416
1417
1418
1419
1420
1421
1422
1423
1424
1425
1426
1427
1428
1429
1430
1431
1432
1433
1434
1435
1436
1437
1438
1439
1440
1441
1442
1443
1444
1445
1446
1447
1448
1449
1450
1451
1452
1453
1454
1455
1456
1457
1458
1459
1460
1461
1462
1463
1464
1465
1466
1467
1468
1469
1470
1471
1472
1473
1474
1475
1476
1477
1478
1479
1480
1481
1482
1483
1484
1485
1486
1487
1488
1489
1490
1491
1492
1493
1494
1495
1496
1497
1498
1499
1500
1501
1502
1503
1504
1505
1506
1507
1508
1509
1510
1511
1512
1513
1514
1515
1516
1517
1518
1519
1520
1521
1522
1523
1524
1525
1526
1527
1528
1529
1530
1531
1532
1533
1534
1535
1536
1537
1538
1539
1540
1541
1542
1543
1544
1545
1546
1547
1548
1549
1550
1551
1552
1553
1554
1555
1556
1557
1558
1559
1560
1561
1562
1563
1564
1565
1566
1567
1568
1569
1570
1571
1572
1573
1574
1575
1576
1577
1578
1579
1580
1581
1582
1583
1584
1585
1586
1587
1588
1589
1590
1591
1592
1593
1594
1595
1596
1597
1598
1599
1600
1601
1602
1603
1604
1605
1606
1607
1608
1609
1610
1611
1612
1613
1614
1615
1616
1617
1618
1619
1620
1621
1622
1623
1624
1625
1626
1627
1628
1629
1630
1631
1632
1633
1634
1635
1636
1637
1638
1639
1640
1641
1642
1643
1644
1645
1646
1647
1648
1649
1650
1651
1652
1653
1654
1655
1656
1657
1658
1659
1660
1661
1662
1663
1664
1665
1666
1667
1668
1669
1670
1671
1672
1673
1674
1675
1676
1677
1678
1679
1680
1681
1682
1683
1684
1685
1686
1687
1688
1689
1690
1691
1692
1693
1694
1695
1696
1697
1698
1699
1700
1701
1702
1703
1704
1705
1706
1707
1708
1709
1710
1711
1712
1713
1714
1715
1716
1717
1718
1719
1720
1721
1722
1723
1724
1725
1726
1727
1728
1729
1730
1731
1732
1733
1734
1735
1736
1737
1738
1739
1740
1741
1742
1743
1744
1745
1746
1747
1748
1749
1750
1751
1752
1753
1754
1755
1756
1757
1758
1759
1760
1761
1762
1763
1764
1765
1766
1767
1768
1769
1770
1771
1772
1773
1774
1775
|
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
<html>
<head>
<title>SWISH-Enhanced: CHANGES - List of revisions </title>
<link href="./style.css" rel=stylesheet type="text/css" title="refstyle">
</head>
<body>
<h1 class="banner">
<a href="http://swish-e.org"><img border=0 src="images/swish.gif" alt="Swish-E Logo"></a><br>
<img src="images/swishbanner1.gif"><br>
<img src="images/dotrule1.gif"><br>
CHANGES - List of revisions
</h1>
<hr>
<p>
<div class="navbar">
<a href="./INSTALL.html">Prev</a> |
<a href="./index.html">Contents</a> |
<a href="./SWISH-CONFIG.html">Next</a>
</div>
<p>
<div class="toc">
<A NAME="toc"></A>
<P><B>Table of Contents:</B></P>
<UL>
<LI><A HREF="#Revision_History">Revision History</A>
<UL>
<LI><A HREF="#Version_2_4_3_December_9_2004">Version 2.4.3 December 9, 2004</A>
<LI><A HREF="#Version_2_4_3_pr1_Wed_Dec_1_09_52_50_PST_2004">Version 2.4.3-pr1 - Wed Dec 1 09:52:50 PST 2004</A>
<LI><A HREF="#Version_2_4_2_March_09_2004">Version 2.4.2 - March 09, 2004</A>
<LI><A HREF="#Version_2_4_1_December_17_2003">Version 2.4.1 - December 17, 2003</A>
<LI><A HREF="#Version_2_4_0_October_27_2003">Version 2.4.0 - October 27, 2003</A>
<LI><A HREF="#Version_2_4_0_Release_Candidate_4_September_26_2003">Version 2.4.0 (Release Candidate 4) September 26, 2003</A>
<LI><A HREF="#Version_2_4_0_Release_Candidate_3_September_11_2003">Version 2.4.0 (Release Candidate 3) September 11, 2003</A>
<LI><A HREF="#Version_2_4_0_Release_Candidate_2_September_10_2003">Version 2.4.0 (Release Candidate 2) September 10, 2003</A>
<LI><A HREF="#Version_2_4_0_Release_Candidate_1_May_21_2003">Version 2.4.0 (Release Candidate 1) May 21, 2003</A>
<LI><A HREF="#Version_2_2_3_December_11_2002">Version 2.2.3 - December 11, 2002</A>
<LI><A HREF="#Version_2_2_2_November_14_2002">Version 2.2.2 - November 14, 2002</A>
<LI><A HREF="#Version_2_2_1_September_26_2002">Version 2.2.1 - September 26, 2002</A>
<LI><A HREF="#Version_2_2_September_18_2002">Version 2.2 - September 18, 2002</A>
<LI><A HREF="#Version_2_2rc1_August_29_2002">Version 2.2rc1 - August 29, 2002</A>
</UL>
</UL>
</div>
[ <B><FONT SIZE=-1><A HREF="#toc">TOC</A></FONT></B> ]
<HR>
<P>
<H1><A NAME="Revision_History">Revision History</A></H1>
<P>
This document contains list of bug fixes and feature additions to Swish-e.
<P>
[ <B><FONT SIZE=-1><A HREF="#toc">TOC</A></FONT></B> ]
<HR>
<H2><A NAME="Version_2_4_3_December_9_2004">Version 2.4.3 December 9, 2004</A></H2>
<DL>
<P><DT><STRONG><A NAME="item_Improved">Improved error messsages when using incremental indexing</A></STRONG><DD>
<P>
There was a bit of confusion on how to use incremental indexing (still
experimental) so added better logic for error messages.
<P>
Also fixed a logic error when setting the incremental update mode. Caught
by Paul Loner.
</DL>
<P>
[ <B><FONT SIZE=-1><A HREF="#toc">TOC</A></FONT></B> ]
<HR>
<H2><A NAME="Version_2_4_3_pr1_Wed_Dec_1_09_52_50_PST_2004">Version 2.4.3-pr1 - Wed Dec 1 09:52:50 PST 2004</A></H2>
<DL>
<P><DT><STRONG><A NAME="item__Fixed_">"Fixed" libxml2's change in UTF8Toisolat1() return value</A></STRONG><DD>
<P>
Bernhard Weisshuhn supplied a patch to parser.c for checking the return
value of <CODE>UTF8Toisolat1().</CODE> Seems that libxml2 now returns the
number of characters converted instead of zero for success.
<P>
<table>
<tr>
<td bgcolor="#eeeeee" width="1">
</td>
<td>
<pre> <A HREF="http://bugzilla.gnome.org/show_bug.cgi?id=153937">http://bugzilla.gnome.org/show_bug.cgi?id=153937</A></pre>
</td>
</tr>
</table>
<P><DT><STRONG><A NAME="item_Added">Added swish-config and pkg-config</A></STRONG><DD>
<P>
Swish now provides a swish-config script and config file for the pkg-config
utility. These tools help when building programs that link with the swish-e
library.
<P>
The SWISH::API Makefile.PL program uses swish-config to locate the
installation directory of swish-e. This should make building SWISH::API
easier when swish-e is installed in a non-standard location.
<P><DT><STRONG><A NAME="item_Fixed">Fixed rank bias in merge</A></STRONG><DD>
<P>
Peter van Dijk noticed that MetaNamesRank settings were not being copied to
the output index when merging.
<P><DT><STRONG>Added SwishFuzzy function</STRONG><DD>
<P>
SwishFuzzy function (SWISH::API::Fuzzy) lets you stem a word without first
searching. This might be helpful for playing with queries prior to the
search.
<P><DT><STRONG>Fixed translate character table</STRONG><DD>
<P>
Michael Levy found an error in the table used to translate 8859-1 to
ascii7. Luckily, it was an upper case translation and the table is only
used on lower case characters.
<P><DT><STRONG><A NAME="item_MetaNamesRank">MetaNamesRank documentation</A></STRONG><DD>
<P>
Changed the 'not yet implemented' caveat to 'implemented but experimental'.
<P><DT><STRONG>Added Continuation option to config processing</STRONG><DD>
<P>
You can now use continuation lines in the config file:
<P>
<table>
<tr>
<td bgcolor="#eeeeee" width="1">
</td>
<td>
<pre> IgnoreWords \
the \
am \
is \
are \
was</pre>
</td>
</tr>
</table>
<P>
There may not be any characters following the backslash.
<P><DT><STRONG>Fixed Buzzwords (and other word lists entered in the config)</STRONG><DD>
<P>
Words entered in config were not converted to lower case before storing in
the index.
<P><DT><STRONG>Fixed metaname mapping problem in Merge</STRONG><DD>
<P>
Peter Karman found an error when merging indexes where the source indexes
had the same metanames, but listed in a different order in their config
files. Words would then be indexed under the wrong metaID number in the
output index.
<P><DT><STRONG><A NAME="item_SWISH">SWISH::Filters and spider.pl updates</A></STRONG><DD>
<P>
The web spider <EM>spider.pl</EM> was updated to work better with SWISH::Filter by default and also make it
easier to use the spider default along with a spider config file. See
spider.pl for details.
<P>
SWISH::Filter was updated. The way filters are created has changed. If you
created your own filters you will need to update them. Take a look at
SWISH::Filter and the filters included in the distribution.
<P><DT><STRONG><A NAME="item_Updates">Updates to Documentation</A></STRONG><DD>
<P>
Richard Morin submitted formatting and punctuation dates to the README and
INSTALL docs.
<P><DT><STRONG>Added -R option to support IDF word weighting in ranking. (karman)</STRONG><DD>
<P>
Added Inverse Document Frequency calculation to the <CODE>getrank()</CODE>
routine. This will allow the relative frequency of a word in relationship
to other words in the query to impact the ranking of documents.
<P>
Example: if 'foo' is present twice as often as 'bar' in the collection as a
whole, a search for 'foo bar' will weight documents with 'bar' more heavily
(i.e., higher rank) than those with 'foo'.
<P>
The impact is greatest when OR'ing words in a query rather than AND'ing
them (which is the default).
<P>
Also added Rank discussion to the FAQ.
<P><DT><STRONG>Updates to the example scripts</STRONG><DD>
<P>
Updated PhraseHighlight.pm as suggested by Bill Schell for an optimization
when all words in a document are highlighted.
<P>
Updated search.cgi and PhraseHighlight.pm to use the internal stemmers via
the SWISH::API module as suggested by Jonas Wolf.
<P><DT><STRONG><A NAME="item_Leak">Leak when using C library</A></STRONG><DD>
<P>
David Windmueller found a memory leak when calling multiple searches on a
swish handle. The problem was swish loading the pre-sorted property index
on every search, even after the table had been loaded into memory.
<P><DT><STRONG><A NAME="item_Swish">Swish.cgi now kills swish-e on time out</A></STRONG><DD>
<P>
The example script <EM>swish.cgi</EM> uses an alarm (on platforms that support alarm) to abort processing after
some number of seconds, but it was not killing the child process, swish-e.
Bill Schell submitted a patch to kill the child when the alarm triggers.
<P><DT><STRONG><A NAME="item_The">The template search.tt was renamed to swish.tt</A></STRONG><DD>
<P>
The template was renamed because it's used by <EM>swish.cgi</EM>, not by
<EM>search.cgi</EM>, which was confusing.
<P><DT><STRONG>Updates to the search.cgi</STRONG><DD>
<P>
The example script <EM>search.cgi</EM> was updated to work better with mod_perl and to use external template files
and style sheets.
<P><DT><STRONG><A NAME="item_New">New MS Word Filter</A></STRONG><DD>
<P>
James Job provided the SWISH::Filter::Doc2html filter that uses the wvWare
(http://wvware.sourceforge.net/) program for filtering MS Word documents.
If both catdoc and wvWare are installed then wvWare will be used.
<P>
wvWare is reported to do a good job at converting MS Word docs to HTML. In
a few tests it did work well, but other cases it failed to generate correct
output. It was also much, much slower than catdoc. I tested with wvWare
0.7.3 on Debian Linux. Testing with both is recommended.
<P><DT><STRONG><A NAME="item_Change">Change in way symbolic links are followed</A></STRONG><DD>
<P>
John-Marc Chandonia pointed out that if a symlink is skipped by FileRules,
then the actual file/directory is marked as "already seen" and
cannot be indexed by other links or directly.
<P>
Now, files and directories are not marked "already seen" until
after passing FileRules (i.e after a file is actually indexed or a
directory is processed).
<P><DT><STRONG><A NAME="item_Could">Could not set SwishSetSort() more than once</A></STRONG><DD>
<P>
David Windmueller found a problem when trying to set the sort order more
than once on an existing search object. Memory was not correctly reset
after clearing the previous sort values.
<P><DT><STRONG><A NAME="item_Access">Access MetaNames and PropertyNames from API</A></STRONG><DD>
<P>
Patch provided by Jamie Herre to access the MetaNames and PropertyNames via
the C API and to test via the testlib program. Swish::API also updated to
access this data.
<P><DT><STRONG><A NAME="item_SwishResultPropertyULong">SwishResultPropertyULong() bug fixed</A></STRONG><DD>
<P>
David Windmueller reported that <CODE>SwishResultPropertyULong()</CODE> was
returning ULONG_MAX on all calls. This was fixed.
<P><DT><STRONG><A NAME="item_Null">Null written to wrong location in file.c</A></STRONG><DD>
<P>
Bill Schell with the help of valgrind found a null written past the end of
a buffer in file.c in the code that supports the old parsers. This resulted
in a segfault while indexing a large set of XML documents.
<P><DT><STRONG>Fixed problem when indexing very large files</STRONG><DD>
<P>
Steve Harris reported a problem when indexing a very large document that
caused an integer overflow. Jos Ruiz updated to used unsigned integers.
<P><DT><STRONG><A NAME="item_Bump">Bump word position on block tags with HTML2 parser</A></STRONG><DD>
<P>
Peter Karman pointed out the the libxml2 HTML parser was allowing phrase
matches across block level html elements. Swish now bumps the word position
on these elements.
</DL>
<P>
[ <B><FONT SIZE=-1><A HREF="#toc">TOC</A></FONT></B> ]
<HR>
<H2><A NAME="Version_2_4_2_March_09_2004">Version 2.4.2 - March 09, 2004</A></H2>
<UL>
<P><LI><STRONG><A NAME="item_UseStemming">UseStemming didn't take no for an answer</A></STRONG>
<P>
UseStemming was coded as an alias for FuzzyIndexingMode when Snowball was
compiled in (the default), but "no" doesn't always mean no when
the Norwegian stemmer is available.
<P><LI><STRONG><A NAME="item_Fixed">Fixed problem building incremental version</A></STRONG>
<P>
Fixed compile problem with building incremental indexing mode. This is an
experimental option with swish-e to allow adding files to an index. See
configure --help for build option. Incremental indexes are not compatible
with standard indexes.
<P><LI><STRONG><A NAME="item_Updated">Updated build instructions in INSTALL</A></STRONG>
<P>
Added a few comments about use of CPPFLAGS and LDFLAGS.
<P><LI><STRONG><A NAME="item_Updated">Updated the index_hypermail.pl</A></STRONG>
<P>
Updated to work with latest version of hypermail (pre-2.1.9).
<P><LI><STRONG><A NAME="item_Time">Time zone in ResultPropertyStr()</A></STRONG>
<P>
Format string for generating date did not include the time zone in
location. Add strftime format string to config.h
<P><LI><STRONG><A NAME="item_Undefined">Undefined and Blank Properties and (NULL)</A></STRONG>
<P>
Fixed a few problems with printing properties:
<P>
1) Using -p and -x showed different results if a bad property value was
given:
<P>
<table>
<tr>
<td bgcolor="#eeeeee" width="1">
</td>
<td>
<pre> $ swish-e -w not dkdk -p badname -H0
err: Unknown Display property name "badname"
.
$ swish-e -w not dkdk -x '<badname>\n' -H0
(NULL)</pre>
</td>
</tr>
</table>
<P>
Now both return an error.
<P>
2) Fixed bug where using a "fmt" string with -x output generated
(bad) output if the result did not have the specified property.
<P>
<table>
<tr>
<td bgcolor="#eeeeee" width="1">
</td>
<td>
<pre> $ swish-e -w not dkdk -x '<somedate>\n' -H0 # undefined value</pre>
</td>
</tr>
</table>
<P>
<table>
<tr>
<td bgcolor="#eeeeee" width="1">
</td>
<td>
<pre> $ swish-e -w not dkdk -x '<somedate fmt="%Y %B %d">\n' -H0
%Y %B 1075353525</pre>
</td>
</tr>
</table>
<P>
Now nothing is printed if the property does not exist.
<P>
3) Updated SWISH::API to <CODE>croak()</CODE> on invalid property names,
and to return undefined values for missing properties.
<P>
4) Updated swish.cgi and search.cgi to not generate warnings on undefined
values return as properties. Note that swish.cgi will now die on undefined
properties. Previously would just display (NULL).
<P><LI><STRONG><A NAME="item_Fixed">Fixed segfault when generating warnings while parsing</A></STRONG>
<P>
Parser.c was incorrectly calling <CODE>warning()</CODE> incorrectly. And
-Wall was not catching this!
<P><LI><STRONG><A NAME="item_Added">Added check for internal property names.</A></STRONG>
<P>
Parser was not checking for use of Swish-e reserved property names.
<P>
<table>
<tr>
<td bgcolor="#eeeeee" width="1">
</td>
<td>
<pre> <swishrank>foo</swishrank></pre>
</td>
</tr>
</table>
<P>
This will now generate a warning.
</UL>
<P>
[ <B><FONT SIZE=-1><A HREF="#toc">TOC</A></FONT></B> ]
<HR>
<H2><A NAME="Version_2_4_1_December_17_2003">Version 2.4.1 - December 17, 2003</A></H2>
<UL>
<P><LI><STRONG><A NAME="item_Added">Added new example CGI script</A></STRONG>
<P>
search.cgi is a new skeleton CGI script that uses SWISH::API for searching.
It is installed in the same location as swish.cgi.
<P><LI><STRONG><A NAME="item_Add">Add Fuzzy access to C and Perl interfaces</A></STRONG>
<P>
Added a number of functions to the C API (and SWISH::API) to access the
stemmer used when indexing a given index.
<P><LI><STRONG><A NAME="item_Commas">Commas in numbers</A></STRONG>
<P>
Added commas to summary display at end of indexing.
<P><LI><STRONG><A NAME="item_Insert">Insert whitespace between tags</A></STRONG>
<P>
Parser.c was updated to flush the text buffer before and after every
(non-inline HTML) tag.
<P>
The problem was that:
<P>
<table>
<tr>
<td bgcolor="#eeeeee" width="1">
</td>
<td>
<pre> foo<tag>bar</tag>baz</pre>
</td>
</tr>
</table>
<P>
would index as a single word "foobarbaz".
<P><LI><STRONG><A NAME="item_DirTree">DirTree.pl</A></STRONG>
<P>
DirTree.pl was updated to work with SWISH::Filter and to work on Windows.
DirTree.pl is a program to fetch files from the file system and works with
the -S prog input method.
<P><LI><STRONG><A NAME="item_Problem">Problem with --enable-incremental option</A></STRONG>
<P>
Fixed configure script to build incremental option. Note that this is still
experimental. But testers are welcome.
<P><LI><STRONG><A NAME="item_headers">headers.c bug</A></STRONG>
<P>
Mark Fletcher with the help of valgrind found a bug in headers.c function
SwishIndexHeaderNames used by the C API.
<P><LI><STRONG><A NAME="item_Clarify">Clarify documentation regarding search order</A></STRONG>
<P>
At the prompting of Doralyn Rossmann updated SEARCH.pod to try and make the
explanation of searching clearer, and to fix an error in the description of
nested searches.
</UL>
<P>
[ <B><FONT SIZE=-1><A HREF="#toc">TOC</A></FONT></B> ]
<HR>
<H2><A NAME="Version_2_4_0_October_27_2003">Version 2.4.0 - October 27, 2003</A></H2>
<UL>
<P><LI><STRONG><A NAME="item_Note">Note: Different Index Format</A></STRONG>
<P>
Swish-e version 2.4.0 has a different index file format from previous
versions of Swish-e. Upgrading will <STRONG>require</STRONG> reindexing -- version 2.4.0 cannot read indexes created with previous
versions.
</UL>
<P>
[ <B><FONT SIZE=-1><A HREF="#toc">TOC</A></FONT></B> ]
<HR>
<H2><A NAME="Version_2_4_0_Release_Candidate_4_September_26_2003">Version 2.4.0 (Release Candidate 4) September 26, 2003</A></H2>
<UL>
<P><LI><STRONG><A NAME="item_robots">robots.txt not closed correctly</A></STRONG>
<P>
When using -S http method robots.txt was not closed and that caused the
(last) .contents file to not be unlinked under Windows. Windows seems to
think filenames are related to files.
<P><LI><STRONG><A NAME="item_SWISH">SWISH::Filter and locating programs on Windows</A></STRONG>
<P>
SWISH::Filter now scans <CODE>$libexecdir</CODE> in addition to the PATH
for programs (such at catdoc and pdftotext), and also checks for programs
by adding the extensions ".exe" and ".bat" to the
program name.
<P><LI><STRONG><A NAME="item_Install">Install sample templates</A></STRONG>
<P>
The sample templates included with swish.cgi are now installed in
<CODE>$pkgdatadir</CODE> (typically /usr/local/share/swish-e).
</UL>
<P>
[ <B><FONT SIZE=-1><A HREF="#toc">TOC</A></FONT></B> ]
<HR>
<H2><A NAME="Version_2_4_0_Release_Candidate_3_September_11_2003">Version 2.4.0 (Release Candidate 3) September 11, 2003</A></H2>
<UL>
<P><LI><STRONG><A NAME="item_Fix">Fix parser bug meta=(foo*)</A></STRONG>
<P>
Fixed bug in query parser caused in rc2's (pr2) attempt to catch wildcards
errors.
</UL>
<P>
[ <B><FONT SIZE=-1><A HREF="#toc">TOC</A></FONT></B> ]
<HR>
<H2><A NAME="Version_2_4_0_Release_Candidate_2_September_10_2003">Version 2.4.0 (Release Candidate 2) September 10, 2003</A></H2>
<UL>
<P><LI><STRONG><A NAME="item_Indexing">Indexing HTML title</A></STRONG>
<P>
Fixed a problem when these were used in combination:
<P>
<table>
<tr>
<td bgcolor="#eeeeee" width="1">
</td>
<td>
<pre> MetaNames swishtitle
MetaNameAlias swishtitle title</pre>
</td>
</tr>
</table>
<P>
That failed to correctly reset the metaname stack and indexed text under
the wrong metaID.
<P><LI><STRONG><A NAME="item_Single">Single Wildcards</A></STRONG>
<P>
Due to the way the query parser "works" a search of
<P>
<table>
<tr>
<td bgcolor="#eeeeee" width="1">
</td>
<td>
<pre> "foo *"</pre>
</td>
</tr>
</table>
<P>
would result in a search of "foo*". Now that results in:
<P>
<table>
<tr>
<td bgcolor="#eeeeee" width="1">
</td>
<td>
<pre> err: Single wildcard not allowed as word </pre>
</td>
</tr>
</table>
<P><LI><STRONG><A NAME="item_Fixed">Fixed search parsing bug</A></STRONG>
<P>
Brad Miele reported that the word "andes" was not being found. It
was being stemmed to "and" when was then considered an operator.
[moseley]
<P><LI><STRONG><A NAME="item_Add">Add new directive PropertyNamesSortKeyLength</A></STRONG>
<P>
PropertyNamesSortKeyLength sets the sort key length to use when sorting
string properties. The default is 100 characters. There was a hard-coded
100 char limit before, but that was a problem where people were not
building from source (Windows). The value of this is questionable -- it's
intended to limit how much memory is used when sorting while indexing and
searching. [moseley]
<P><LI><STRONG><A NAME="item_Fixed">Fixed sorting issues with multiple indexes and reverse sorting</A></STRONG>
<P>
Reworked much of the sorting code. Still to do is setting the character
sort order. [moseley]
<P><LI><STRONG><A NAME="item_Fixed">Fixed minor memory leak</A></STRONG>
<P>
Fixed leak of not releasing memory of index file name and swish_handle
destroy, and fixed SwishStemWord to default to the Stemmer_en. [moseley]
<P>
Fixed libtest.c example program that was not cleaning up memory after an
error condition.
<P><LI><STRONG><A NAME="item_Replaced">Replaced Swish-e's Porter Stemmer with Snowball</A></STRONG>
<P>
Swish-e now has support for Snowball stemmers
(http://snowball.tartarus.org/). The stemmers are enabled for an index with
FuzzyIndexingMode Stemming_* where "*" can be:
<P>
<table>
<tr>
<td bgcolor="#eeeeee" width="1">
</td>
<td>
<pre> de, dk, en1, en2, es, fi, fr, it, nl, no, pt, ru, se</pre>
</td>
</tr>
</table>
<P>
In addition, UseStemming yes or FuzzyIndexingMode Stemming_en will use the
old stemmer.
</UL>
<P>
[ <B><FONT SIZE=-1><A HREF="#toc">TOC</A></FONT></B> ]
<HR>
<H2><A NAME="Version_2_4_0_Release_Candidate_1_May_21_2003">Version 2.4.0 (Release Candidate 1) May 21, 2003</A></H2>
<UL>
<P><LI><STRONG><A NAME="item_Security">Security Fix: swish.cgi</A></STRONG>
<P>
The swish.cgi script was not correctly escaping HTML when searching by the
right combination of metanames and highlighting module. This could lead to
cross-site scripting if indexing un-trusted documents. [moseley]
<P><LI><STRONG><A NAME="item_Added">Added Support for building a Debian Package</A></STRONG>
<P>
To build as a .deb unpack the distribution and chdir then run
<P>
<table>
<tr>
<td bgcolor="#eeeeee" width="1">
</td>
<td>
<pre> $ fakeroot debian/build binary</pre>
</td>
</tr>
</table>
<P>
Then install the generated .deb file with dpkg -i
<P><LI><STRONG><A NAME="item_Use">Use SWISH::Filter by default with spider.pl</A></STRONG>
<P>
spider.pl is installed in the libexecdir directory as well as the
SWISH::Filter modules. PDF, MS Word, MP3, and XML documents will be indexed
automatically if the required helper applications (e.g. catdoc, pdftotext)
or scripts (e.g. MP3::Tag) are installed.
<P>
Swish also knows about libexecdir, so you you specify a relative path with
-S prog swish-e will look for the program in libexecdir. This is mostly for
spider.pl so indexing only requires:
<P>
<table>
<tr>
<td bgcolor="#eeeeee" width="1">
</td>
<td>
<pre> IndexDir spider.pl
SwishProgParameters default <A HREF="http://localhost/index.html">http://localhost/index.html</A></pre>
</td>
</tr>
</table>
<P>
And swish-e will find spider.pl and SWISH::Filter will be used to convert
docs.
<P><LI><STRONG><A NAME="item_Fixed">Fixed Document-Type bug</A></STRONG>
<P>
Document-Type was not being reset after set input from a -S prog program
causing the wrong parser to be used. [moseley]
<P><LI><STRONG><A NAME="item_New">New Directive: PropertyNamesNoStripChars</A></STRONG>
<P>
Swish replaces all series of low ASCII chars with a single space character.
This option instructs swish to store all chars in the property. [moseley]
<P><LI><STRONG><A NAME="item_Change">Change HTTP access defaults</A></STRONG>
<P>
Defaults used with -S http access method were changed.
<P>
Delay was reduced from one minute between start of each request to five
seconds between requests.
<P>
MaxDepth was changed from five to zero, meaning there is no limit to depth
indexed by default. [moseley]
<P><LI><STRONG><A NAME="item_swishspider">swishspider location and SpiderDirectory</A></STRONG>
<P>
The swishspider program is now installed in $prefix/lib/swish-e by default.
This can be changed by the --libexecdir option to configure.
<P>
The SpiderDirectory option now defaults to the value of libexecdir instead
of the current directory. [moseley]
<P><LI><STRONG><A NAME="item_Added">Added libtool and automake support</A></STRONG>
<P>
Replaces the build system with Autotools. Now builds libswish-e as a shared
library on systems that support shared libraries. The swish-e binary links
against this shared library. Can also build outside the source tree on
platforms with GNU make. [moseley]
<P><LI><STRONG><A NAME="item_Updates">Updates to installation</A></STRONG>
<P>
Running "make install" now installs additional files. Files
include the swish-e binary, the libswish-e search library, swish-e.h
header, documentation files, the swishspider program, and Perl modules used
for the example swish.cgi search script. Directories will be created if
they do not already exist. Installation directories can be specified at
build time.
<P><LI><STRONG><A NAME="item_Fixed">Fixed bug when searching at end of inverted index</A></STRONG>
<P>
Swish was not correctly detecting the end of the inverted index when
searching a wildcard word that was past the last word in the index. Caught
by Frank Heasley. [moseley]
<P><LI><STRONG><A NAME="item_Increase">Increase sort key length from 50 to 100 characters</A></STRONG>
<P>
The setting MAX_SORT_STRING_LEN in <EM>src/config.h</EM> sets the max length used when sorting in swish-e. You may reduce this
number to save memory while sorting, or increase it if you have very long
properties to sort.
<P><LI><STRONG><A NAME="item_Remove">Remove &quot; entity from -p output</A></STRONG>
<P>
The -p option to print properties was escaping double quotes in properties
with the &quot; entity. -x does not do that, so inconsistent. -p no
longer converts double quotes. The user should pick a good delimiter with
-d or preferably use the -x method for generating output.
<P><LI><STRONG><A NAME="item_XML">XML parser and Windows</A></STRONG>
<P>
The XML parser was being passed the incorrect buffer length when used on
Windows platform causing the parser to abort with an error.
<P><LI><STRONG><A NAME="item_Version">Version Numbering</A></STRONG>
<P>
SWISH-E versions starting with 2.3.4 use kernel version numbering. Versions
are in the form: Major.Minor.Build. Odd minor versions are development.
Even minor versions are releases. 2.3.4 would be a development version.
2.4.0 would be a release version. 2.3.20 would be the 20th build of 2.3.
<P><LI><STRONG><A NAME="item_Added">Added RPM support</A></STRONG>
<P>
RPMs can be built with:
<P>
<table>
<tr>
<td bgcolor="#eeeeee" width="1">
</td>
<td>
<pre> ./configure
make dist</pre>
</td>
</tr>
</table>
<P>
Copy the resulting tarball to RPM's SOURCES directory and then run as a
superuser:
<P>
<table>
<tr>
<td bgcolor="#eeeeee" width="1">
</td>
<td>
<pre> rpmbuild -ba rpm/swish-e.spec</pre>
</td>
</tr>
</table>
<P>
You should have swish-e packages in your RPMS/<arch> directory.
[augur]
<P><LI><STRONG><A NAME="item_Changed">Changed default perl binary location</A></STRONG>
<P>
Most perl scripts provided with SWISH-E now use /usr/bin/perl by default.
Note that some scripts are generated at build time, so those will look in
the path for the location of the perl binary.
<P><LI><STRONG><A NAME="item_New">New Feature: MetaNamesRank</A></STRONG>
<P>
MetaNamesRank can be used to adjust the ranking for words based on the
word's MetaName.
<P><LI><STRONG><A NAME="item_New">New Swish Library API and Perl Module</A></STRONG>
<P>
The Swish-e C library interface was rewritten to provide better memory
management and better separation of data. Most indexing related code has
been removed from the library. A new header file is provided for the API:
swish-e.h.
<P>
The Perl module SWISHE was replaced with the SWISH::API module in the
Swish-e distribution.
<P>
<STRONG>Previous versions of the SWISHE module will not work with this version of Swish-e.</STRONG>
<P>
If you are using the SWISHE module from a previous version of Swish then
you must either rewrite your code to use the new SWISH::API module (highly
recommended) or use the replacement SWISHE module. The replacement SWISHE
module is a thin interface to the SWISH::API module. It can be downloaded
from
<P>
<table>
<tr>
<td bgcolor="#eeeeee" width="1">
</td>
<td>
<pre> <A HREF="http://swish-e.org/Download/old/SWISHE-0.03.tar.gz">http://swish-e.org/Download/old/SWISHE-0.03.tar.gz</A></pre>
</td>
</tr>
</table>
<P><LI><STRONG><A NAME="item_NoContents">NoContents not working with libxml2 parser</A></STRONG>
<P>
Corrected problem when using NoContents with binary files and the HTML2
parser.
<P>
Trying to index image file names with:
<P>
<table>
<tr>
<td bgcolor="#eeeeee" width="1">
</td>
<td>
<pre> IndexOnly .gif .jpeg
NoContents .gif .jpeg</pre>
</td>
</tr>
</table>
<P>
failed to index the path names because the default parser (HTML2 when
libxml2 is linked with swish-e) was not finding any text in the binary
files. [moseley]
<P><LI><STRONG><A NAME="item_Updates">Updates to swish.cgi</A></STRONG>
<P>
The example/swish.cgi script can now use the SWISH::API module for
searching an index. Combined with mod_perl this module can improve search
performance considerably.
<P>
The Perl modules used with the swish.cgi script have all been moved into
the SWISH::* namespace. Hence, files in the <EM>modules</EM> directory were moved into the <EM>modules::SWISH</EM> directory.
</UL>
<P>
[ <B><FONT SIZE=-1><A HREF="#toc">TOC</A></FONT></B> ]
<HR>
<H2><A NAME="Version_2_2_3_December_11_2002">Version 2.2.3 - December 11, 2002</A></H2>
<P>
Multiple -L options were ORing instead of ANDing. Catch by Patrick Mouret.
[moseley]
<P>
[ <B><FONT SIZE=-1><A HREF="#toc">TOC</A></FONT></B> ]
<HR>
<H2><A NAME="Version_2_2_2_November_14_2002">Version 2.2.2 - November 14, 2002</A></H2>
<P>
Pass non- text/* files onto indexing code IF there is a FileFilter
associated with the *extension* of the URL. Fixes the problem of not being
able to index, say, pdf files by using the FileFilter configuation option.
<P>
Fixed bug where nulls were stripped when using FileFilter with -S prog.
Catch by Greg Fenton. [moseley]
<P>
[ <B><FONT SIZE=-1><A HREF="#toc">TOC</A></FONT></B> ]
<HR>
<H2><A NAME="Version_2_2_1_September_26_2002">Version 2.2.1 - September 26, 2002</A></H2>
<UL>
<P><LI><STRONG><A NAME="item_NoContents">NoContents with -S prog</A></STRONG>
<P>
Failed to use the correct default parser when using the No-Contents header
and libxml2 linked in. [moseley]
<P><LI><STRONG><A NAME="item_Add">Add tests for IRIX and sparc machines</A></STRONG>
<P>
8-byte alignment in mem_zones is is required for these machine [moseley]
<P><LI><STRONG><A NAME="item_Fixed">Fixed code when removing files</A></STRONG>
<P>
Was not correctly removing words from index when parser aborted [jmruiz]
<P><LI><STRONG><A NAME="item_Merge">Merge segfault</A></STRONG>
<P>
Fixed segfault caused by trying to print null dates while merging duplicate
files. [moseley]
<P><LI><STRONG><A NAME="item_Documentation">Documentation patches</A></STRONG>
<P>
Spelling corrections to the SWISH-CONFIG pod page [Steve Eckert]
<P><LI><STRONG><A NAME="item_Configure">Configure corrections</A></STRONG>
<P>
Fixed a zlib test error that used "==" in a test [Steve Eckert]
<P><LI><STRONG><A NAME="item_Updates">Updates to VMS build</A></STRONG>
<P>
The VMS build was updated [Jean-Franois PIRONNE]
<P><LI><STRONG><A NAME="item_MANIFEST">MANIFEST corrections</A></STRONG>
<P>
Added missing filters and vms build file into MANIFEST [moseley]
</UL>
<P>
[ <B><FONT SIZE=-1><A HREF="#toc">TOC</A></FONT></B> ]
<HR>
<H2><A NAME="Version_2_2_September_18_2002">Version 2.2 - September 18, 2002</A></H2>
<UL>
<P><LI><STRONG><A NAME="item_Default">Default parser</A></STRONG>
<P>
Swish-e will now use the HTML2 (libxml2) parser by default if libxml2 is
installed and DefaultContents or IndexContents is not used.
<P><LI><STRONG><A NAME="item_Selecting">Selecting parsers</A></STRONG>
<P>
Allow HTML*, XML*, and TXT* to automatically select the libxml2-based
parsers if libxml2 is linked with Swish-e, otherwise fallback to the
built-in parsers.
<P><LI><STRONG><A NAME="item_SwishSpider">SwishSpider and Filters</A></STRONG>
<P>
Filters (FileFilter directive) did not work correctly when spidering with
the -S http method. A new filter system was developed and now filtering of
documents (e.g. pdf->html or MSWord->text) is handled by the
src/SwishSpider program.
<P>
When indexing with the -S http method only documents of content-type
"text/*" are indexed. Other documents must be converted to text
by using the filter system.
<P><LI><STRONG><A NAME="item_Buffer">Buffer overflow in xml.c</A></STRONG>
<P>
Fixed bug in xml.c reported by Rodney Barnett when very long words were
indexed. [moseley]
<P><LI><STRONG><A NAME="item_configure">configure script updates</A></STRONG>
<P>
Updated from _WIN32 checks to feature checks using autoconf [moseley,
norris]
<P><LI><STRONG><A NAME="item_updates">updates to run on Alpha (Linux 2.4 (Debian 3.0))</A></STRONG>
<P>
Fixed a cast error when calling zlib, and the calls to read/write a packed
longs to disk. [jmruiz, moseley]
<P><LI><STRONG><A NAME="item_COALESCE_BUFFER_MAX_SIZE">COALESCE_BUFFER_MAX_SIZE</A></STRONG>
<P>
Some people were seeing the following error:
<P>
<table>
<tr>
<td bgcolor="#eeeeee" width="1">
</td>
<td>
<pre> err: Buffer too short in coalesce_word_locations.
Increase COALESCE_BUFFER_MAX_SIZE in config.h and rebuild.</pre>
</td>
</tr>
</table>
<P>
This was due to indexing binary data or files with very large number of
words. The best solution is to not index binary data or files with a very
large number of words.
<P>
Swish-e will now automatically reallocate the buffer as needed. [jmruiz]
</UL>
<P>
[ <B><FONT SIZE=-1><A HREF="#toc">TOC</A></FONT></B> ]
<HR>
<H2><A NAME="Version_2_2rc1_August_29_2002">Version 2.2rc1 - August 29, 2002</A></H2>
<P>
Many large changes were made internally in the code, some for performance
reasons, some for feature changes and additions, and some to prepare for
new features in later versions of Swish-e.
<UL>
<P><LI><STRONG><A NAME="item_Documentation">Documentation!</A></STRONG>
<P>
Documentation is now included in the source distribution as .pod (perldoc)
files, and as HTML files. In addition, the distribution can now generate
PDF, postscript, and unix man pages from the source .pod files. See <A HREF="././README.html">README</A> for more information.
<P><LI><STRONG><A NAME="item_Indexing">Indexing and searching speed</A></STRONG>
<P>
The indexing process has been imporoved. Depending on a number of factors,
you may see a significant improvement in indexing speed, especially if
upgrading from version 1.x.
<P>
Searching speed has also been improved. Properties are not loaded until
results are displayed, and properties are pre-sorted during indexing to
speed up sorting results by properties while searching.
<P><LI><STRONG><A NAME="item_Properties">Properties are written to a sepearte file</A></STRONG>
<P>
Swish-e now stores document properties in a separate file. This means there
are now two files that make up a Swish-e index. The default files are <CODE>index.swish-e</CODE> and <CODE>index.swish-e.prop</CODE>.
<P>
This change frees memory while indexing, allowing larger collections to be
indexed in memory.
<P><LI><STRONG><A NAME="item_Internal">Internal data stored as Properties</A></STRONG>
<P>
Pre 2.2 some internal data was stored in fixed locations within the index,
namely the file name, file size, and title. 2.2 introduced new internal
data such as the last modified date, and document summaries. This data is
considered <EM>meta data</EM> since it is data about a document.
<P>
Instead of adding new data to the internal structure of the index file, it
was decided to use the MetaNames and PropertyNames feature of Swish-e to
store this meta information. This allows for new meta data to be added at a
later time (e.g. Content-type), and provides an easy and customizable way
to print results with the <CODE>-p</CODE> switch and the new <CODE>-x</CODE> switch. In addition, search results can now be sorted and limited by
properties.
<P>
For example, to sort by the rank and title:
<P>
<table>
<tr>
<td bgcolor="#eeeeee" width="1">
</td>
<td>
<pre> swish-e -w foo -s swishrank desc swishtitle asc</pre>
</td>
</tr>
</table>
<P><LI><STRONG><A NAME="item_The">The header display has been slightly reorganized.</A></STRONG>
<P>
If you are parsing output headers in a program then you may need to adjust
your code. There's a new switch <-H> to control the level of header
output when searching.
<P><LI><STRONG><A NAME="item_Results">Results are now combined when searching more than one index.</A></STRONG>
<P>
Swish-e now merges (and sorts) the results from multiple indexes when using <CODE>-f</CODE> to specify more than one index. This change effects the way maxhits (<CODE>-m</CODE>) works. Here's a summary of the way it works for the different versions.
<P>
<table>
<tr>
<td bgcolor="#eeeeee" width="1">
</td>
<td>
<pre> 1.3.2 - MaxHits returns first N results starting from the first index.
e.g. maxhits=20; 15 hits Index1, 40 hits Index2
All 15 from Index1 plus first five from Index2 = 20 hits.</pre>
</td>
</tr>
</table>
<P>
<table>
<tr>
<td bgcolor="#eeeeee" width="1">
</td>
<td>
<pre> 2.0.0 - MaxHits returns first N results from each index.
e.g. Maxhits=20; 15 hits Index1, 40 hits Index2
All 15 from Index1 plus 15 from Index2.</pre>
</td>
</tr>
</table>
<P>
<table>
<tr>
<td bgcolor="#eeeeee" width="1">
</td>
<td>
<pre> 2.2.0 - Results are merged and first N results are returned.
e.g. Maxhits=20; 15 hits Index1, 40 hits Index2
Results are merged from each index and sorted
(rank is the default sort) and only the first
20 are returned.</pre>
</td>
</tr>
</table>
<P><LI><STRONG><A NAME="item_New">New prog document source indexing method</A></STRONG>
<P>
You can now use -S prog to use an external program to supply documents to
Swish-e. This external program can be used to spider web servers, index
databases, or to convert any type of document into html, xml, or text, so
it can be indexed by Swish-e. Examples are given in the
<CODE>prog-bin</CODE> directory.
<P><LI><STRONG><A NAME="item_The">The indexing parser was rewritten to be more logical.</A></STRONG>
<P>
TranslateCharacters now is done before WordCharacters is checked. For
example,
<P>
<table>
<tr>
<td bgcolor="#eeeeee" width="1">
</td>
<td>
<pre> WordCharacters abcdefghijklmnopqrstuvwxyz
TranslateCharacters n</pre>
</td>
</tr>
</table>
<P>
Now <CODE>El Nio</CODE> will be indexed as El Nino (el and nino), even though <CODE></CODE>
is not listed in WordCharacters.
<P>
Previously, stopwords were checked after stemming and soundex conversions,
as well as most of the other word checks (WordCharacters, min/max length
and so on). This meant that the stopword list probably didn't work as
expected when using stemming.
<P><LI><STRONG><A NAME="item_The">The search parser was rewritten to be more logical</A></STRONG>
<P>
The search parser was rewritten to correct a number of logic errors.
Swish-e did not differentiate between meta names, Swish-e operators and
search words when parsing the query. This meant, for example, that
metanames might be broken up by the WordCharacters setting, and that they
could be stemmed.
<P>
Swish-e operator characters <CODE>"*()=</CODE> can now be searched by escaping with a backslash. For example:
<P>
<table>
<tr>
<td bgcolor="#eeeeee" width="1">
</td>
<td>
<pre> ./swish-e -w 'this\=odd\)word'</pre>
</td>
</tr>
</table>
<P>
will end up searching for the word <CODE>this=odd)word</CODE>. To search for a backslash character preceed it with a backslash.
<P>
Currently, searching for:
<P>
<table>
<tr>
<td bgcolor="#eeeeee" width="1">
</td>
<td>
<pre> ./swish-e -w 'this\*'</pre>
</td>
</tr>
</table>
<P>
is the same as a wildcard search. This may be fixed in the future.
<P>
Searching for buzzwords with those characters will still require
backslashing. This also may change to allow some un-escaped operator
characters, but some will always need to be escaped (e.g. the double-quote
phrase character).
<P><LI><STRONG><A NAME="item_Quotes">Quotes and Backslash escapes in strings</A></STRONG>
<P>
A bug was fixed in the <CODE>parse_line()</CODE> function (in <EM>string.c</EM>) where backslashes were not escaping the next character. <CODE>parse_line()</CODE> is used to parse a string of text into tokens (words). Normally splitting
is done at whitespace. You may use quotes (single or double) to define a
string (that might include whitespace) as a single parameter. The backslash
can also be used to escape the following character when *within* quotes
(e.g. to escape an embedded quote character).
<P>
<table>
<tr>
<td bgcolor="#eeeeee" width="1">
</td>
<td>
<pre> ReplaceRules append "foo bar" <- define "foo bar" as a single word
ReplaceRules append "foo\"bar" <- escape the quotes
ReplaceRules append 'foo"bar' <- same thing</pre>
</td>
</tr>
</table>
<P><LI><STRONG><A NAME="item_Example">Example user.config file removed.</A></STRONG>
<P>
Previous versions of Swish-e included a configuration file called
<CODE>user.config</CODE> which contained examples of all directives. This has been replaced by a
series of example configuration files located in the
<CODE>conf</CODE> directory. The configuration directives are now described in
<A HREF="././SWISH-CONFIG.html">SWISH-CONFIG</A>.
<P><LI><STRONG><A NAME="item_Ports">Ports to Win32 and VMS</A></STRONG>
<P>
David Norris has included the files required to build Swish-e under
Windows. See <CODE>src/win32</CODE>. A self-extracting Windows version is available from the Download page of
the swish-e.org web site.
<P>
Jean-Franois Pironne has provided the files required to build Swish-e
under OpenVMS. See <CODE>src/vms</CODE> for more information.
<P><LI><STRONG><A NAME="item_String">String properties are concatenated</A></STRONG>
<P>
Multiple <EM>string</EM> properties of the same name in a document are now concatenated into one
property. A space character is added between the strings if needed. A
warning will be generated if multiple numeric or date properties are found
in the same document, and the additional properties will be ignored.
<P>
Previously, properties of the same name were added to the index, but could
not be retrieved.
<P>
To do: remove the <CODE>next</CODE> pointer, and allow user-defined character to place between properties.
<P><LI><STRONG><A NAME="item_regex">regex type added to ReplaceRules</A></STRONG>
<P>
A more general purpose pattern replacement syntax.
<P><LI><STRONG><A NAME="item_New">New Parsers</A></STRONG>
<P>
Swish-e's XML parser was replaced with James Clark's expat XML parser
library.
<P>
Swish-e can now use Daniel Veillard's libxml2 library for parsing HTML and
XML. This requires installation of the library before building Swish-e. See
the <A HREF="././INSTALL.html">INSTALL</A> document for information. libxml2 is not required, but is strongly
recommended for parsing HTML over Swish-e's internal HTML parser, and
provides more features for both HTML and XML parsing.
<P><LI><STRONG><A NAME="item_Support">Support for zlib</A></STRONG>
<P>
Swish-e can be compiled with zlib. This is useful for compressing large
properties. Building Swish-e with zlib is stronly recommended if you use
its <CODE>StoreDescription</CODE> feature.
<P><LI><STRONG><A NAME="item_LST">LST type of document no longer supported</A></STRONG>
<P>
LST allowed indexing of files that contained multiple documents.
<P><LI><STRONG><A NAME="item_Temporary">Temporary files</A></STRONG>
<P>
To improve security Swish-e now uses the <CODE>mkstemp(3)</CODE> function to create temporary files. Temporary files are used while indexing
only. This may result in some portability issues, but the security issues
were overriding.
<P>
(Currently this does not apply to the -S http indexing method.)
<P>
<CODE>mkstemp</CODE> opens the temporary with O_EXCL|O_CREAT flags. This prevents overwriting
existing files. In addition, the name of the file created is a lot harder
to guess by attackers. The temporary file is created with only owner
permissions.
<P>
Please report any portability issues on the Swish-e discussion list.
<P><LI><STRONG><A NAME="item_Temporary">Temporary file locations</A></STRONG>
<P>
Swish-e now uses the environment variables <CODE>TMPDIR</CODE>, <CODE>TMP</CODE>, and
<CODE>TEMP</CODE> (in that order) to decide where to write temporary files. The configuration
setting of <A HREF="././SWISH-CONFIG.html#item_TmpDir">TmpDir</A> will be used if none of the environment variables are set. Swish-e uses the
current directory otherwise; there is no default temporary directory.
<P>
Since the environment variables override the configuration settings, a
warning will be issued if you set <A HREF="././SWISH-CONFIG.html#item_TmpDir">TmpDir</A>
in the configuration file and there's also an environment variable set.
<P>
Temporary files begin with the letters "swtmp" (which can be
changed in
<EM>config.h</EM>), followed by two or more letters that indicate the type of temporary
file, and some random characters to complete the file name. If indexing is
aborted for some reason you may find these temporary files left behind.
<P><LI><STRONG><A NAME="item_New">New Fuzzy indexing method Double Metaphone</A></STRONG>
<P>
Based on Lawrence Philips' Metaphone algorithm, add two new methods of
creating a fuzzy index (in addition to Stemming and Soundex).
</UL>
<P>
Changes to Configuration File Directives. Please see
<A HREF="././SWISH-CONFIG.html">SWISH-CONFIG</A> for more info.
<UL>
<P><LI><STRONG><A NAME="item_New">New directives: IndexContents and DefaultContents</A></STRONG>
<P>
The IndexContents directive assigns internal Swish-e document parsers to
files based on their file type. The DefaultContents directive assigns a
parser to be used on file that are not assigned a parser with
IndexContents.
<P><LI><STRONG><A NAME="item_New">New directive: UndefinedMetaTags [error|ignore|index|auto]</A></STRONG>
<P>
This describes what to do when a meta tag is found in a document that is
not listed in the MetaNames directive.
<P><LI><STRONG><A NAME="item_New">New directive: IgnoreTags</A></STRONG>
<P>
Will ignore text with the listed tags.
<P><LI><STRONG><A NAME="item_New">New directive: SwishProgParameters *list of words*</A></STRONG>
<P>
Passes words listed to the external Swish-e program when running with
<CODE>-S prog</CODE> document source method.
<P><LI><STRONG><A NAME="item_New">New directive: ConvertHTMLEntities [yes|no]</A></STRONG>
<P>
Controls parsing and conversion of HTML entities.
<P><LI><STRONG><A NAME="item_New">New directive: DontBumpPositionOnMetaTags</A></STRONG>
<P>
The word position is now bumped when a new metatag is found -- this is to
prevent phrases from matching across meta tags. This directive will disable
this behavior for the listed tags.
<P>
This directive works for HTML and XML documents.
<P><LI><STRONG><A NAME="item_Changed">Changed directive: IndexComments</A></STRONG>
<P>
This has been changed such that comments are not indexed by default.
<P><LI><STRONG><A NAME="item_Changed">Changed directive: IgnoreWords</A></STRONG>
<P>
The builtin list of stopwords has been removed. Use of the SwishDefault
word will generate a warning, and no stop words will be used. You must now
specify a list of stopwords, or specify a file of stopwords.
<P>
A sample file <CODE>stopwords.txt</CODE> has been included in the <EM>conf/stopwords</EM>
directory of the distribution, and can be used by the directive:
<P>
<table>
<tr>
<td bgcolor="#eeeeee" width="1">
</td>
<td>
<pre> IgnoreWords File: /path/to/stopwords.txt</pre>
</td>
</tr>
</table>
<P><LI><STRONG><A NAME="item_Change">Change of the default for IgnoreTotalWordCountWhenRanking</A></STRONG>
<P>
The default is now "yes".
<P><LI><STRONG><A NAME="item_New">New directive: Buzzwords</A></STRONG>
<P>
Buzzwords are words that should be indexed as-is, without checking for
stopwords, word length, WordCharacters, or any other of the word limiting
features. This allows indexing of things like <CODE>C++</CODE> when "+" is not listed in WordCharacters.
<P>
Currenly, IgnoreFirstChar and IgnoreLastChar will be stripped before
processing Buzzwords.
<P>
In the future we may use separate IgnoreFirst/Last settings for buzzwords
since, for example, you may wish to index all <CODE>+</CODE> within Swish-e words, but strip <CODE>+</CODE> from the start/end of Swish-e words, but not from the buzzword <CODE>C++</CODE>.
<P><LI><STRONG><A NAME="item_New">New directives: PropertyNamesNumeric PropertyNamesDate</A></STRONG>
<P>
Before Swish-e 2.2 all user-defined document properties were stored in the
index as strings. PropertyNamesNumeric and PropertyNamesDate tell it that a
property should be stored in binary format. This allows for correct sorting
of numeric properties.
<P>
Currenly, only integers can be stored, such as a unix timestamp. (Swish-e
uses <CODE>strtoul</CODE> to convert the number to an unsigned long internally.)
<P>
PropertyNamesDate only indicates to Swish-e that a number is a unix
timestamp, and to display the property as a formatted time when printing
results. Swish does not currently parse date strings; you must provide a
unix timestamp.
<P><LI><STRONG><A NAME="item_New">New directive: MetaNameAlias</A></STRONG>
<P>
You may now create alias names for MetaNames. This allow you to map or
group multiple names to the same MetaName.
<P><LI><STRONG><A NAME="item_New">New directive: PropertyNameAlias</A></STRONG>
<P>
Creates aliases for a PropertyName.
<P><LI><STRONG><A NAME="item_New">New directive: PropertyNamesMaxLength</A></STRONG>
<P>
Sets the max length of a text property.
<P><LI><STRONG><A NAME="item_New">New directive: HTMLLinksMetaName</A></STRONG>
<P>
Defines a metaname to use for indexing href links in HTML documents.
Available only with libxml2 parser.
<P><LI><STRONG><A NAME="item_New">New directive: ImageLinksMetaName</A></STRONG>
<P>
Defines a metaname to use for indexing src links in <img> tags. Allow
you to search image pathnames within HTML pages. Available only with
libxml2 parser.
<P><LI><STRONG><A NAME="item_New">New directive: IndexAltTagMetaName</A></STRONG>
<P>
Allows indexing of image ALT tags. Only available when using the libxml2
parser.
<P><LI><STRONG><A NAME="item_New">New directive: AbsoluteLinks</A></STRONG>
<P>
Attempts to convert relative links indexed with HTMLLinksMetaName and
ImageLinksMetaName to absolute links. Available only with libxml2 parser.
<P><LI><STRONG><A NAME="item_New">New directive: ExtractPath</A></STRONG>
<P>
Allows you to use a regular expression to extract out part of the path of
each file and index it with a meta name. For example, this allows searches
to be limited to parts of your file tree.
<P><LI><STRONG><A NAME="item_New">New directive: FileMatch</A></STRONG>
<P>
FileMatch is similar to FileRules. Where FileRules is used to exclude files
and directoires, FileMatch is used to <EM>include</EM> files.
<P><LI><STRONG><A NAME="item_New">New directive: PreSortedIndex</A></STRONG>
<P>
Controls which properties are pre-sorted while indexing. All properties are
sorted by default.
<P><LI><STRONG><A NAME="item_New">New directive: ParserWarnLevel</A></STRONG>
<P>
Sets the level of warning printed when using libxml2.
<P><LI><STRONG><A NAME="item_New">New directive: obeyRobotsNoIndex [yes|NO]</A></STRONG>
<P>
When using libxml2 to parse HTML, Swish-e will skip files marked as
NOINDEX.
<P>
<table>
<tr>
<td bgcolor="#eeeeee" width="1">
</td>
<td>
<pre> <meta name="robots" content="noindex"></pre>
</td>
</tr>
</table>
<P>
Also, comments may be used within HTML and XML source docs to block
sections of content from indexing:
<P>
<table>
<tr>
<td bgcolor="#eeeeee" width="1">
</td>
<td>
<pre> <!-- SwishCommand noindex -->
<!-- SwishCommand index --></pre>
</td>
</tr>
</table>
<P>
and/or these may be used also:
<P>
<table>
<tr>
<td bgcolor="#eeeeee" width="1">
</td>
<td>
<pre> <!-- noindex -->
<!-- index --></pre>
</td>
</tr>
</table>
<P><LI><STRONG><A NAME="item_New">New directive: UndefinedXMLAttributes</A></STRONG>
<P>
This describes how the content of XML attributes should be indexed, if at
all. This is similar to UndefinedMetaTags, but is only for XML attributes
and when parsed by libxml2. The default is to not index XML attributes.
<P><LI><STRONG><A NAME="item_New">New directive: XMLClassAttributes</A></STRONG>
<P>
XMLClassAttributes can specify a list of attribute names whose content is
combined with the element name to form metanames.
<P><LI><STRONG><A NAME="item_New">New directive: PropCompressionLevel [0-9]</A></STRONG>
<P>
If compiled with zlib, Swish-e uses this setting to control the level of
compression applied to properties. Properties must be long enough (defined
in config.h) to be compressed. Useful for StoreDescription.
<P><LI><STRONG><A NAME="item_Experimental">Experimental directive: IgnoreNumberChars</A></STRONG>
<P>
Defines a set of characters. If a word is made of of *only* those
characters the word will not be indexed.
<P><LI><STRONG><A NAME="item_New">New directive: FuzzyIndexingMode</A></STRONG>
<P>
This configuration directive is used to define the type of
"fuzzy" index to create. Currently the options are:
<P>
<table>
<tr>
<td bgcolor="#eeeeee" width="1">
</td>
<td>
<pre> None
Stemming
Soundex
Metaphone
DoubleMetaphone</pre>
</td>
</tr>
</table>
</UL>
<P>
Changes to command line arguments. See <A HREF="././SWISH-RUN.html">SWISH-RUN</A> for documentation on these switches.
<UL>
<P><LI><STRONG><A NAME="item_New">New command line argument -H</A></STRONG>
<P>
Controls the level (verbosity) of header information printed with search
results.
<P><LI><STRONG><A NAME="item_New">New command line argument -x</A></STRONG>
<P>
Provides additional header output and allows for a <EM>format string</EM>
to describe what data to print.
<P><LI><STRONG><A NAME="item_New">New command line argument -k</A></STRONG>
<P>
Prints words stored in the Swish-e index.
<P><LI><STRONG><A NAME="item_New">New command line argument -N</A></STRONG>
<P>
Provides a way to do incremental indexing by comparing last modification
dates. You pass <CODE>-N</CODE> a path to a file and only files newer than the last modified date of that
file will be indexed.
<P><LI><STRONG><A NAME="item_Removed">Removed command line argument -D</A></STRONG>
<P>
<CODE>-D</CODE> no longer dumps the index file data. Use <CODE>-T</CODE> instead.
<P><LI><STRONG><A NAME="item_New">New command line argument -T</A></STRONG>
<P>
<CODE>-T</CODE> is used for debugging indexing and searching.
<P><LI><STRONG><A NAME="item_Enhanced">Enhanced command line argument -d</A></STRONG>
<P>
Now <CODE>-d</CODE> can accept some back-slashed characters to be used as output separators.
<P><LI><STRONG><A NAME="item_Enhanced">Enhanced command line argument -P</A></STRONG>
<P>
Now -P sets the phrase delimiter character in searches.
<P><LI><STRONG><A NAME="item_New">New command line argument -L</A></STRONG>
<P>
Swish-e 2.2 contains an <STRONG>experimental</STRONG> feature to limit results by a range of property values. This behavior of
this feature may change in the future.
<P><LI><STRONG><A NAME="item_Modified">Modified command line argument -v</A></STRONG>
<P>
Now the argument <CODE>-v 0</CODE> results in *no* output unless there is an error. This is a bit more handy
when indexing with cron.
</UL>
[ <B><FONT SIZE=-1><A HREF="#toc">TOC</A></FONT></B> ]
<HR>
<p>
<div class="navbar">
<a href="./INSTALL.html">Prev</a> |
<a href="./index.html">Contents</a> |
<a href="./SWISH-CONFIG.html">Next</a>
</div>
<p>
<P ALIGN="CENTER">
<IMG ALT="" WIDTH="470" HEIGHT="10" SRC="images/dotrule1.gif"></P>
<P ALIGN="CENTER">
<div class="footer">
<BR>SWISH-E is distributed with <B>no warranty</B> under the terms of the
<A HREF="http://www.fsf.org/copyleft/gpl.html">GNU Public License</A>,<BR>
Free Software Foundation, Inc.,
59 Temple Place - Suite 330, Boston, MA 02111-1307, USA<BR>
Public questions may be posted to
the <A HREF="http://swish-e.org/Discussion/">SWISH-E Discussion</A>.
</div>
</body>
</html>
|
