b59fc7d7 (HEAD -> master, origin/master) When checking for empty strings, use 'not' instead of 'empty()'.
fa88ed53 Correction to handling of exclusionary list member indexing
3671de18 Remove dead code
bf3bb0e5 Update README.rst
485429d7 double the buffer size
d6e4b94c workaround underflow? verbosity+
5426d918 some lint stuff
cea87758 Remove non-ICU code paths as a consequence of issue #501
27b1eadb Preserve sigma set when converting to and from OL formats
a0099ea5 (origin/pmatch-arc-compression) Merge branch 'empty_pmatch_alphabet'
dc158fe2 Replace "use-character-classes" option with "unicode-character-classes"
30069af7 Merge branch 'pmatch-unicode-character-classes'
db931052 Blank alphabets in RTN transducers
ff935594 Don't waste time parsing redundant alphabets
0aa7d314 (origin/pmatch-unicode-character-classes) Add caching mechanism to avoid calling expensive unicode functions
ba777b01 Ignore empty lists
82f5e042 Merge branch 'pmatch-unicode-character-classes' of https://github.com/hfst/hfst into pmatch-unicode-character-classes
d31667f9 Fix copypaste errors
65380472 don't escape ] in xerox mode
47a9ee20 Correctly indent warnings about nonsubtractable symbols
e9b82181 Warn about subtractions that probably break due to metasymbols
c55ff9a4 Add new unicode special symbols to is_special_symbol(string)
63b12e04 Added runtime ICU versions of more character classes
97540c3c An implementation & use of character class UnicodeAlpha
50ee623c Improve warning about nested context conditions
08716576 Improvements to reporting of function call evaluation in verbose mode
3de9f4ef Improvements to verbose reporting of compilation steps
dff92692 An implementation & use of character class UnicodeAlpha
82c3b87e Merge branch 'master' of https://github.com/hfst/hfst
cecdaec1 Added a potential new symbol, UnicodeAlpha
9b1dbccd skip sub-readings of partial unks
9432431c use ?? as unk sign too
fe9a89b2 force ICU, fixes #501
8f4a23a5 oops debug pritns
4de5a50d logic fail
7cb2eca7 blah
dcafd64d warn before reweighting unweighted argh
848956c0 hack to restore composition middletape
3b238265 Drop leftover checks for C++11 headers
db1da471 SVN is obsolete, refer to GIT instead
f5df65e4 Fix memory leak in TransducerAlphabet: use virtual destructor.
cfb5dbc8 Use empty method for std::string.
fcb7eb91 Drop support for TR1 standard assuming that C++14 is minimal requirement.
65296280 Use range-based loops in HFST transducer. (#550)
8bea3e68 Drop read_prop_line because it is a duplicate of add_prop_line.
432c2016 Also run action for PRs
bda85b7b Cleanup XFST compiler (#548)
abc9de56 Use unique_ptr for input/ouput stream memory allocation (closes #547)
e2e9c95b Fix memory leak in hfst-lexc-compiler
fd5008c7 Fix compiler warning: return constant string.
9eba5b60 Fix memory leak in FST type formatting.
417b9494 Fix memory leak in hfst-strings2fst tool.
3511d26a Fix memory leak in XFST substitution operation.
84bf68b1 Fix memory leak in hfst-summarize tool.
a0c8654e Fix memory leak in SFST transducer.
e34c7e66 Fix memory leaks when transducer is redefined in XFST compiler.
1916181b Fix memory leak in print DOT and modernize for-loop.
4e9363e9 Fix memory leaks in PC-KIMMO print.
e467c8c3 Fix implicit fall-through in hfst-optimized-lookup
f09ddc48 Fix memory leaks in unit tests. (#533)
fa2ca7f4 Drop legacy wrappers for foma and lexc. (#532)
eaf901eb Use upstream Foma; Use C++17, or C++11 with bundled OpenFST; Use SSE where available
66f047d1 Fix segfault in hfst-summarize: skip first input symbols if FST is empty.
ad6854dd Newer OpenFST don't `using namespace std`
76b00004 Fix segfault when Foma sigmas are merged. (#530)
e4b5bbc5 Update external links for sfst, foma and flex.
734ae775 Use upstream OpenFST.
3977c5c1 OFTC
3486e40c Fix segfault when disjunct is applied to hude transducer.
727c1387 Fix segfault in Foma flag elimination.
1c47bbce Add lex/yacc sources to dist (see issue #524)
76dc60fd Fix compiler warnings for SFST error reporting functions.
d174a7a9 Fix memory leak when ranges are released
7317f7e8 (tag: v3.15.5) Merge branch 'master' of https://github.com/hfst/hfst
70c48438 Release 3.15.5.
fd7f7770 allow relative paths in source command
f68fb11f Drop string copy function because it leads to stack corruption and is not used.
1de31ce6 Fix memory leaks when input/output streams are not released.
25becf42 Fix memory leak when parallel rule mismatch error is detected.
f8d464dd Fix memory leak in command line arguments parsing code of hfst-xfst.
cede7aa7 Fix memory leak in are_equivalent method of FomaTransducer.
afb3cfa6 Fix alloc-dealloc-mismatch (malloc vs operator delete) for function name variable.
49050967 Fix memory leak in foma transducer: avoid double allocation of sigma.
63caf314 Fix memory leak in XFST and XRE compilers undefine method.
badaa605 Fix link typos
4e73272c Fix memory leaks in LexcCompiler detected by address sanitizer.
c4349fa4 Avoid memory leak in XfstCompiler::write_stack: free outstream after usage.
2a454e3c Avoid segfault when regexp parsing fails in lexicon.
fef85e4f Snip flex version warning as all supported platforms have new enough flex (closes #508); Python 3.x
7b8eb265 Merge branch 'master' of https://github.com/hfst/hfst
6d967ad5 Modify tracking of entry points to allow for RTNs in Captures()
a2741510 truncate long names, fixes #488
2a51e9dd Initialize input_mark_symbol (works around https://github.com/divvun/libdivvun/issues/43)
3b91edb0 hfst-tokenise: skip modifier letters in is_cg_tag
1eafb28f (tag: v3.15.4) Intended license was LGPLv3+ so formalize that, and explicitly mention that the SFST-derived files are used under LGPLv3+ agreement (closes #494)
1a29cf31 Change https://kitwiki.csc.fi/twiki/bin/view/KitWiki -> https://github.com/hfst/hfst/wiki (fixes #452)
4a10952e Allow hfst-xfst to source from absolute paths
f7c208b4 Properly open input files for hfst-expand-equivalences (#495)
be76d456 Release 3.15.4; Import Foma patches with license change to Apache-2.0 and minimal other fixes; Allow regeneration of Foma files so the generated files can be omitted; Explicit license file for SFST
6128b1c8 (tag: v3.15.3) accoring -> according
94b48cb5 Release 3.15.3
ed49cfde Merge pull request #456 from hfst/440-lemma-vs-tag-vs-non-composed-single-arc
5b231fce (origin/440-lemma-vs-tag-vs-non-composed-single-arc) ${ICU_CPPFLAGS} to AM_CPPFLAGS
437e9eb3 Use ICU if possible to find characterBoundary in is_cg_tag
c78084b3 escape all \ for cg output fixes #491
fd35fa6e whitespace only
8bc6c247 bump reqd automake in autogensh
4e2e8be7 Merge pull request #489 from mr-martian/expand-help-string
143098ce I promise I can count
321bc2d4 don't follow cycles in example
0d8e021e clearer help text for hfst-expand
5dadce48 try newer automake travis
b01f4220 git tries to modify line-endings in binaries
6a4eeaf7 sedded the object references
4748ae87 add subdir-objects fixes #486
b7fef31f Merge pull request #471 from ftyers/free-insert
a40be44f theres some bug in eliminate flags that overflows
1d19c46d Priority union (#480)
eabf0408 Realign (#468)
4d6b15ad Parse wordbound blanks as normal blanks in hfst-proc (#478)
a140e286 Travis macOS Homebrew just doesn't work
452b7289 Some files should keep crlf
342e98a8 Fix Travis; Add Github Action; Force text and eol=lf
088e48ac Fix Travis; Add Github Action; Force text and eol=lf
c1a4887a Fst2txt union (#475)
b2129ff2 improve handling of input based on hfst-substitute
323ea043 insert freely manpage
01c5f907 add insert-freely
265bc37a some fixes to hfst-proc (#465)
f7e1636d Add clarifying line to help message
78bcf55f Merge pull request #438 from mr-martian/master
a761da1a hfst-proc: escape apertium reserved chars in lemma
505aac3d (tag: v3.15.2) Release 3.15.2.
57dd2e18 Fix off-by-one bug in checking captures
95ec8680 Change Travis: ccache fails for Python bindings
dee52b33 Put Python bindings under autotools via --enable-python-bindings
356ddd30 (tag: v3.15.1) Fix Travis
b456e6e7 Release 3.15.1; Python 2 -> 3; Trusty -> Xenial
f1dc19aa Handle all-space-forms in hfst-tokenize, cf. #443
0a796763 fix typo in HfstBasicTransducer.write_xfst()
6a318906 HFST does not need nightly
12a16951 EL7 has ICU 50
5a25971d icu-config is gone in newer distros
99e606b9 doh, switched upper/lower here
7dbd2df3 test non-latin1 hfst-proc uppercasing
1c8aa88a travis: test icu config too
7d1f88f9 we need $(ICU_LIBS) here on mac
c1a18f60 go back to icu-config since .pc isn't in old debians
3fa86a80 Add $(ICU_LIBS) to LDADD to make it compile
5f0c4895 (origin/icu) no GLIB_CFLAGS/ICU_CFLAGS in hfst.pc
0250e35c use pkg-config instead of icu-config (deprecated)
490efdd5 implement icu handler in hfst-proc and hfst-pmatch2fst
7ad603a5 real configure.ac check for icu
16b1b381 Unicode support for pmatch casing functions, if available
de22d584 Include glib headers if we have them to support unicode
024c111d Fixes https://github.com/hfst/hfst/issues/429
c2f13c6e Adjust spacing of help message to better fit in 80 columns
ec14ee3b Fixes to help strings as per KP-2152
a3911243 Fix handling of DefInsed nets when precomputing allowed initial symbols
368f15b9 Sort PROPtags so they appear in consistent order
61413f63 Add tests
d05a7b73 New version
506337c6 Fix bug where finnish-tokenize was printing analyses too
4f72bb20 Set PYTHONIOENCODING environment variable
4c9ebad1 New package structure
1b601689 Add throw specifiers to destructors, one inheriting the other
d26cd6bb hfst-tokenize: drop trailing zeroes (except first) off weights
f09fc9bd Also mention python 2.7.
ae56dbaa Update pypi README.
f06b5f34 Support python version 2 in tests.
b89d26a7 Fix issue #409.
c75708cc Use hfst_commandline module in all python command line tools.
72c97f5b Rewrite output stream processing.
58f31fa2 Use module hfst_commandline for option parsing.
b2d63868 Define functions for output stream handling as well as single input stream handling.
49d3b339 Fix free parameter handling.
89938963 Use hfst_commandline module in all binary tools.
91d12188 Make hfst-compose use hfst_commandline.
0cb784db Handle input streams in a separate module.
5be41fb1 Improve command line argument handling.
701048a4 Rewrite comamnd line option processing.
0e26ecff pmatch: Make sure @PMATCH_BACKTRACK@ is not treated as special symbol either
e98973ad Define encoding.
9abdc7c6 Fix some things noticed when compiling with clang with c++11 disabled.
dd4420af (tag: v3.15.0) Ready for release 3.15.0.
ad302837 Do not print warnings.
7ab98bee Small fixes noticed when compiling on windows.
a76e3c96 Ignore hfst-eliminate-flags
1c8e5215 pmatch: Make INPUT_MARK not special_symbols, so we can refer to it in pmscripts
432203dd When verbose, print what goes into Like() operations
e80aadeb as_string() for symbols
93f3280d Allow negative weights
f1bc2475 Merge branch 'master' of https://github.com/hfst/hfst
da7bcee8 Remove "." from internal punctuation & adjust url rule
2ae7765f Option to print weights in location mode
35557c0d Allow both escaped and unescaped zeros in multicharacters in all lexicon entries. Fixes issue #274.
640b91d1 Use c strings with printf.
34cbfc3b Allow both escaped and unescaped zeros in multicharacters in one-level lexicon entries. Partially fixes issue #274.
cc258f5d Make link to hfst-apertium-proc only if proc is enabled.
14309ef7 Add tool hfst-eliminate-flags.
d69404d0 Warn user if transducer file contains more than one transducer in hfst-optimized-lookup.
37bdeded Expand @include"filename" directives as a preprocessing stage
732d645e Also reweight state that is both initial and final in hfst-reweight. Fixes issue #368.
6d9c495f Don't delete the implementation's FdTable
3048e03a Print warnings and error messages also in earlier stages of twol compilation.
a7f27143 Do not let distributed files depend on configuration.
cbf1e941 Update man files and include all of them in dist.
444c8860 Remove nonexistent files from dist.
59e314dc Add option --execute-and-quit (-E) to hfst-xfst.
b0a2be55 Merge pull request #392 from frankier/fix-python-lookup-leak
6a7ac658 Fix memory leak in Python .lookup(...) method
99dc9ffd Add returns in non-void functions
22448e08 Option --no-readline should be enough.
a6c923b3 Document datatype Location and function locate and make them visible in the API.
245ae4d6 Add missing file pmatch_tokenize.cc.
d6258bcc Fix incorrect use of delete; Fix missing virtual dtor; Remove some (void)
a1eeeada Fix typo
630fd7b2 Merge branch 'pmatch-implode-explode-fixes'
1afaca88 Make Implode() and Explode() allow symbol expansion
ae00e6b5 Preserve empty lines; Flush
f1ff7731 Merge branch 'master' into pmatch-implode-explode-fixes
295b1264 Don't try to collect initial symbols if initial state is -1
87e74a61 Merge branch 'master' of https://github.com/hfst/hfst
01cfc9f9 Fix internal punctuation case
9073e218 Name transducers for langname instead of langcode in these scripts too
8df796e7 Correct handling of markup markers
a6e96d32 No need for an allocation there; Fix final read
bb8e4d44 Be more verbose in READMEs.
4de9548a Add urns to metashare articles.
856902d0 Make implode and explode accept more than just bare symbolsw
3184e2ad Return a list of strings by default in tokenize.
18f88cc1 Use langname instead of langcode in transducer names
5f694239 Merge branch 'master' of https://github.com/hfst/hfst
61713be8 Fix indentation
b4f726d4 Replace containment implementations with correct ones
f7985c78 Add keyword arguments to PmatchContainer.tokenize.
75a69b35 Quote array expansions, otherwise they break on spaces
1d0d0620 Use * to concatenate array to string
f3b5acb0 Quote grep pattern so shell won't interpret
859de389 Convert dos2unix line endings
af328c48 Force type
8b297fd6 Protect line-based XML/HTML tags (WIP issue #379)
fe4c2f69 Modify tokenization implemented via PmatchContainer.
ceb5d2b0 Merge branch 'master' of https://github.com/hfst/hfst
d44f3947 Don't free data pointer until after we've evaluated everything
80a2bcd1 Add package data, i.e. readme and licensing information for each language package.
1c463a0c Add metadata for each language package for download directory.
683a5f4e (tag: v3.14.0) Ready for release 3.14.0.
e824094a Update morphology tests so that they pass after small changes in pmatching and tokenization.
48e3c363 Move hacky foreign conjugation guessing to unknown token part
85ef0616 Don't tag numbers after all
bcde7b96 One more weight typo
c815db6c Fix same typo again and merge changes from skeleton
ddac854f Foreign word conjugations
eb8efa4c Annotate numerical expressions
e980a909 Adjust indentation
12315805 Fix typo
9b8e3bb6 Merge branch 'master' of https://github.com/hfst/hfst
4b215fca Use language-specific patches when provided
9365b6c9 Bugfixes related to internal punctuation
ce5a05a8 Some language-specific customisations to the tokenizer
c685c412 Implement Ins() arcs inside right contexts
de9cc378 Fix to xerox output mode with plain, unanalyzed tokens
c4056f82 Normalize unicode apostrophies
4f2a4cec Rename morphology directory accordingly.
fcc122b3 Get rid of *new in new morphology tests.
d5e87dbb Rename old morphology tests as *old.
904647ef Tentatively add tests for upcoming new morphology scripts.
e019c23a Merge branch 'master' of https://github.com/hfst/hfst
86f98c8b When uniqifying without regard for weights, make sure to sort by weights
857624ec Return a skip value if tool is not found.
47ba60b6 Update morphology tests: omit deprecated .sh extensions and omorfi-analyze/generate. Do not include python tests in tool test.
3ab93036 Add missing file.
ffd17976 Add missing files to dist.
1032bf75 Explicitly list all functionality tests in dist.
90e161a6 Implement a cautious form of first-input-symbol scanning
9730a4e8 Correct variable name
a450d58b When not printing weights and in --unique xerox mode, suppress weight differences
4b05205c --uniq does not require an arg!
85f305ec Check for tokenizer, not analyzer
27ec41d8 Remove duplicate analyses
4295c847 Improve punct-handling
95f49df0 Add paragraph-tokenizing newline behaviour to --xerox too
82165de2 Migrate to using hfst-tokenize --xerox instead of hfst-lookup
0bef94da Not generating analyzers anymore
00a9cf93 Reflect new --xerox functionality
d0945dfd Don't use the fancy Omorfi tokenizer
393407e6 Nuke old contents of installable dir before proceeding
9863b099 Redirect stdout from command, not stderr..
7b4042d0 Check that root path was set
251108f6 Add & parallelization to Finnish
b470ec4e Fix tests for existing programs
746c069b Wait for everything to finish before making replacements
6f7ca8e3 Make test expect result of previous commits
ff232a41 Utility script for building everything at once
c02ab2df Check for programs and give info if missing
02986612 Unify blankline behaviour of --finnpos and --tokenize
9c45fc95 Add function HfstTransducer::get_initial_input_symbols.
633fee2a Rewrite morphology tests.
3fc0c433 Add stubs for collecting initial symbols
d1191f9b Handle default arcs in pmatch & fix spelling of initial symbols variable
cb6e3c39 Fix definition of unknown_token
51b9c411 Simplify main morpho-tokenization
6d08f5e4 Remove messy first-possible-symbol detection code, read it from hopefully precompiled info instead
05382326 Add clarifying comment
13a14243 Get a bit more speed by pre-reserving space for the vectors
efeb7039 Implement reading vector data from .bin binary format (much faster)
98e6697f Ins() the morphology check
1fa87847 When we have a dictionary word, don't override it
f6c33a59 When returning from a RTN, don't overwrite the original local variables!
6413d710 Add a script for testing the new morphology packages.
38af4eef Handle weight cutoffs inside pmatch
f35c5e28 Fix help message
28d8ce1a Optimized-lookup does work with hfst3 headers, so let's support it
9b53cb33 Add universal tokenizer
253d3e65 Disjunct empty string so transducers "know" about sentence boundaries
9d517a61 Improve usage notes
8b56ed6a Merge branch 'master' of https://github.com/hfst/hfst
132ff958 Link to github wiki instead of kitwiki.
f3858d05 Note about weights in existing morphologies
381bad57 Change default to weighted
13a43f44 Add notes about existing morphologies
241153bf Correct name of temp morphology for building tokenizer
3d2ef763 Add option to build installable dir in-place in specified dir
548a1370 Add options for capital casing and punctuation including
0e162d8c More elaborate tokenizer based on omorfi-tokenize
1949af28 General README
526cf4e5 Punctuation list
0e9d26d0 Merge branch 'master' of https://github.com/hfst/hfst
59df066e Add default tokenizer
0a0ebe2a Tentatively add match_and_print to python api as 'pmatch_tokenize'.
10e65659 Add some unicode quotes and relax the "don't know what to do" case
fc23a214 Try to guess whether we're at a sentence boundary or ordinal period.
334551e5 Handle colon separated conjugations differently to avoid size explosion
338b725a Cut out numerical parts of OmorFi to fix some bugs in it.
cbda6710 Improve dash, hyphen & range handling
07e6f8de Assume that omorfi handles uppercasing correctly; add xml tags
54bd3c88 Merge branch 'master' of https://github.com/hfst/hfst
8232010c Add introspecion mechanism to PmatchObjects; use for optimization
e0193229 travis: include macos
74da1a4f on macos, when bash is called as /bin/sh, echo prints '\x22' as "
c9eb7cb6 Use input symbols in lookup. Add a test for cases such as looking up string 'foo' in transducer [f:0 o:0 o:foo].
c71da223 Use only input symbols for tokenization in apply down. Add tests for cases where this matters, e.g. [f:foo o:0 o:0].
5da96bad Add one more test for compile-replace.
b1010cc2 Align regexp start and end tags with epsilons in compile-replace for better alignment between oridinary symbols.
220f031d Allow arbitrary symbols (except whitespace) in strings given as input to apply up and appy down commands.
8e154aed Implement variable 'retokenize'.
2db4eadf Add precompiled tokenizers
ca6b412e Fix typo
54be1901 Renamed to analyze-words and generate-words, for clarity
1a4d52c5 Add README
65994b0d Add missing files.
75fd90c9 Add function HfstInputStream::read_all and support an iterable object of several transducers in HfstOutputStream::write. Also fix some indentations.
7865f323 handle escaped @ as part of form, keep escape in output
693fe064 Add an extra parameter int max_number to HfstBasicTransducer::lookup, the default being -1 (infinite). Also fix a template issue with function compile_first of python API.
ba5ae812 Merge branch 'master' of https://github.com/hfst/hfst
d476a8ee Update documentation link.
715eee48 More loop protection
c8d197c4 Endless loop protection
384ecbd9 hfst-proc: more fixes and tests for serial unicode ranges
eff43225 tests: use $TOOLDIR, things are not installed yet :)
3720c0c6 travis: make verboser check so we can see what failed
a3445f47 hfst-proc: seems like the test for serial unicode ranges was flipped
0dbbcdb0 Add preserving "plain" omorfi-based Finnish tokenizer.
339ccc77 If compile_sfst_file is given an empty string as filename, read from standard input.
ae77b0a3 Add option -H. Add symbol-harmonization to python tests.
6bdcea82 Specify exceptions thrown. Add functions to the interface. Replace quoted values 'True' and 'False' with True and False.
917801d1 Improvements to python command line tools.
040d2e9c Enable python tools.
9f3d164f Add hfst-strings2fst.py to tests.
43eab05c Add a skeleton for hfst-fst2strings.py.
d54e180e Add hfst-substitute.py.
4286e36d Add implementation for lexc compiler. It still fails with one test case.
bb15ded5 Implement hfst-fst2txt.py.
89a0ee27 Add compose intersect tool.
26d67f0a Add pmatch tools.
549e2254 Implement python regexp parser.
29aabb65 Add tool for pushing weights.
132ba1c4 Add new python tools.
1a4ae938 Add hfst-subtract.py tool.
eb520631 Fix parameter handling and return values.
377d2ebc Add a script for testing python command line tools. Add option --prolog to hfst-txt2fst.py tool.
c5e6cd58 Add hfst-tail python command line tool.
7b584ae6 Add new tools.
2050ec67 Add new python command line tools. Fix a typo in hfst-compose.py.
ec3d25ab Return a skip value for python tests.
82238887 Tools hfst-fst2strings and hfst-head.py implemented.
5ec5cc74 Fix tests.
c5ca695c Fix scripts.
34e42a84 Fix rest of variables in tests.
e8de5ef1 Fix variables in scripts.
445d9e66 Modify scripts. Add a new python command line tool.
aae549f9 Add python versions of the tests.
bcd208d1 Add a python command line tool hfst-format.py.
82513574 Use string instead of char pointers in VarMaps and SVarMaps.
c14bf89f Check if foldername is the empty string and do not add a '/' in that case.
b58f63ed Print an error message if input is in optimized lookup format.
12b55059 update man page about --giella-cg tag assumptions
701b0328 Yet more python tools.
442d6399 Add new (minimal) python versions of command line tools.
95dc1f28 Add option --python to rest of functionality tests.
f77e97f9 Add more --python options to functionality tests.
6c9aa62c Tentatively add option --python to functionality tests. If the option is used, the tests are run with python API, if available.
b17c2864 Add a simple python implementation for hfst-calculate. Enable it with option --python in calculate-functionality.sh.
fd8dfea7 Keep separate RTN stacks for each depth level
29e9155d New morphology scripts
0f768266 Fix some typos, including 'kvargs' -> 'kwargs'.
c6686094 correct home page url of HfstPushWeights
1e2c797b Merge pull request #365 from frankier/fix-print-dot-labels
b0d3218d don't tokenize_multichar as default in --giella-cg yet
c6331179 put tokenize_multichar in TokenizeSettings; default true for --giella-cg
a1894737 Update finnish morphology script.
c187d5ae (tag: v3.13.0) Ready for release 3.13.0.
f6bc1cde -W/--no-weights overrides earlier -w (or -w implied by -g) options
cfa49ab8 Ensure print_dot labels are valid utf-8
cb9916d5 Add missing files.
c92bcaf1 Add missing files to dist.
d98ffa7b Initial commit of finnish-tagtools scripts Includes a few ones from FinnPos modified for our purposes
ee4662b4 Merge branch 'master' of https://github.com/hfst/hfst
d6a247e7 Added a variable "xerox-composition", default to "on"
355aaf9c allow creating HfstInputStream's from std::istream's
8ff49aaa Use libreadline in hfst-pmatch when available
b5b7995b [].with(X = Y) feature in pmatch This provides support in the pmatch2fst compiler to define "global flags" with the syntax [your expression here].with(X = Y). This will set up a flag diacritic readable from anywhere within the brackets, even if Ins()ed.
0858e633 Support getting a list of operations involved with a particular feature. This also involves making define_diacritic() redefine the meaning of symbols if called multiple times on the same symbol number.
4d4471de tokenize tests should pass again
f81dcb53 tokenize --giella-cg: assume FST handles space around tags
b07c47b9 Merge branch 'master' of https://github.com/hfst/hfst
60e5c288 Check for success of the now-working test suite.
acf3c785 Cleanup of remainder of the pmatch test suite Many cases of adding set "need-separators off"; many cases of fixing mistaken test assumptions; one case of fixing code; one case of removing test that was already in pmatch2fst-functionality.sh; one case of removing entire test because it was trying to do something impossible; one case of relaxing test assumptions that should be possibly revisited. Multiple FIXMEs left but all the tests now pass.
2669274a Make both-sided OptCap() be actually both-sided. It still doesn't downcase the continuation of words, which is possibly a bug.
64a18742 tokenize --giella-cg: remove as_cg_tag, just need is_cg_tag now
9decc11c Modify check for list symbols. It was conflicting with left markers for Xerox rules (@LM@), causing a test to very surprisingly break (it didn't even seem to be using optimized-lookup format at all..)
4e4ba7c9 Add tests for Lst() and Exc()
7a0176ac Avoid keeping an extra copy of the tokenizing dictionary in memory (fixes https://github.com/hfst/hfst/issues/362)
8c10e1df Consider list symbols (@L..." and "@X...") to be special
e4dff091 hfst-tokenize --giella-cg, is_tag: check if symbol Multichar_symbol
3ec59daa tokenize --giella-cg: non-verbose as default
862ea0ec Add C and C++ dlls to dist.
d6531234 Fix misspelling of nrc for nlc
c005a4aa Merge branch 'master' of https://github.com/hfst/hfst
86dcec64 Fix runtime handling of contexts and compilation of negative contexts. This was broken when the processing state stack was changed to accommodate "RTN safety". The local state is now pushed into a stack when entering a context and pushed again when exiting it, and popped when backtracking out of those situations.
10967c48 set need-delimiters no for more tests that were broken by it
06cbb209 Made Xerox output of unknowns follow Xerox
97ecf446 Merge pull request #359 from kartikm/patch-1
23dcc545 Fixed typo
134d5a5a Works around issue #358
21b0e820 Actually support multiple different Capture() names
be9e214f Instead of names, use pointers to the calling transducers for rtn calls
34bf445c Don't track returns from Ins() arcs in the call stack Pushing returns from Ins() calls to the call stack were clobbering nested calls. This is unnecessary, because we have the necessary information about the correct return location from the stack depth variable anyway.
1358576a Use hfst-pmatch --newline in the tests so we don't see irrelevant blank lines
e6f7067b Decrement recursion counter when returning from main loop due to no input
528952fa A somewhat hacky fix to a bug introduced by changes to RTN processing pmatch is lacking recursion depth with each RTN entered. This is mostly a problem due to eventually running out of depth and truncating results. For now we reset depth at the start of each input, which makes sense anyway, but the leak should really be fixed.
c45f620a In locate mode, omit blank lines even if in blankline mode if we didn't print anything
74f3a6fd Remember captures for the duration of the entire input. Also avoid copying the captured vectors around, instead passing iterators to input
01c2fd76 Fix short form of end tagging also disable automatic disabling of need-separators when encountering a context
f2b6c8c0 Merge branch 'master' of https://github.com/hfst/hfst
64528f78 In blankline-separated mode, keep blanklines in output too
b0a17a28 Merge branch 'master' of https://github.com/hfst/hfst
31f0767d Include pre-swig-generated wrappers to source distribution. Update README.
de22111c Remove bashisms that was breaking tests that apparently are not run with bash
368acefd Fix tokenize backtrack regression, need to handle @PMATCH_BACKTRACK@ explicitly
da0ab47e Fix test which was probably written with a misunderstanding of replace rules
ed5ac3bb Reinstate and fix some tests by moving them from pmatch-tests.sh (where escaping done by the testing script breaks some tests) to pmatch2fst-functionality.sh
882ee743 Allow term complement of named transducers
26865255 Merge branch 'master' of https://github.com/hfst/hfst
215a4829 Support backreferences (Capture())
eea79b86 Update README.
e1d519c5 Link to exact place in page.
27d3e1c1 Update README.
2e935163 For installation instructions, link to PyPI pages instead of KitWiki.
b68f1461 Update README.
7a3618ea Make Like() and Unlike() much faster by not sorting the whole vocabulary (by doing an insertion sort of the n best words) and only calculating each comparison key once
85f8a8cb Fixed binding of freely insert operator in hfst-twolc.
83eea8b0 Remove unnecessary (debug?) prints.
e10be33e Actually use other than ascii characters in att file.
5102d2d0 Fix some typos in test.sh. Replace epsilon with empty string in fst2strings.py. Read input in utf-8 format in prolog2fst.py, if possible.
394284b4 Fix a typo.
b52f348b Test if hfst-xfst exists after all options are processed.
cdaa748c Implement an equivalent python script for all command line tools needed in tests.
c6dae7a6 When using python for tests, also replace part of command line tools with equivalent python scripts.
88a70d33 Fix a couple of typos.
ef656676 Update python package version number. Use .cpp extension.
7bc8e180 Update README.
778ebcac Get rid of unnecessary CommandLine in python api. Fix some typos.
1c345ef9 Error functions are void, not int.
b8edb3aa Import sys.version_info before using it...
6a2107cb Use extension cpp for all c++ files in pypi package. Also use the c++ implementation of foma backend by default.
43190464 A couple more fixes.
3f63a4e7 Fix some issues noticed by clang.
27c30eed Add a c++ version of foma backend for testing.
31c91c39 Add arguments to function declaration.
0b468456 Fix a couple more typos.
a117e056 Fix some typos.
76048509 Improve test script and add expected results for cases where xfst solution does not exist but the result from hfst script is checked.
c4592072 Add two more python implementations for fsmbook tests.
c75cbb8e Merge branch 'master' of https://github.com/hfst/hfst
61be4541 Add Unlike() operation, which could be better named also add option for whether information about cosine distances should be included in the result when compiling word vector operations
87cb7d6a Two more python tests...
42ba171c Add two more python tests.
221f8dcc Add more tests.
e25e2d46 Add python test for FinnishNumerals
7d917a49 Allow empty string as input for hfst.fst and hfst.fsa and interpret it as epsilon.
3842501e Use xfst instead of python for slow tests.
5962b3a6 Add one more python test.
7b3dccbf Add python test.
dd5ed1f7 Fix broken test assumption
a45a97d8 Prohibit EndTagging printable-empty lengths of input
77d5c7b2 Fix single-character transducers not getting automatic context separators
432d16c9 Verbose message about automatic separators
7bf07dd2 Make using Ins() -arcs safe Ins() arcs are no longer locally greedy, so there is no semantic difference between network inclusion with Ins() or by explicitly embedding. eg. define greedy "a+"; now the following work the same way: Ins(greedy) {abc} greedy {abc}
adc3b0b5 Use temp weights rather than subtractions like in pmatch
1378513d Fix test by having need-separators off
849d2973 Minimize automatic delimiter contexts
97a7b050 Merge pull request #352 from unhammer/tokenise-lib
2fcc6659 Merge branch 'master' into tokenise-lib
ef026f10 Add option --silent to tests.
c0996a8c Add .0000000000 to tests; ToDo: Trim trailing 0 after the dot, then trim the dot; ToDo: Investigate where std::fixed gets disabled
1fe2af94 Wrap compile_twolc_file inside a class as TwolcCompiler::compile.
dbd0d14f Revert "Removed rounding"
813f2d39 Add keyword arguments to compile_twolc_file.
786af199 Removed rounding
702425ee Add twolc files to pypi setup.
30e3c4f9 Fix a typo.
e8acc1a5 Flush and close the stream.
2b26a3b0 Add tests for 'compile_twolc_file'.
b27f9932 Remember to reset also the second parser.
7fb07f52 Add a possibility to reset twolc parsers between reading several twolc files.
251cef5e Add option --local-hfst to setup.py.
87a77e75 Add a tentative implementation of twolc compilation.
45676091 Make code clearer and add documentation.
87382726 Keep track of weights along context checking paths and unify weight handling
539592b7 Merge branch 'master' of https://github.com/hfst/hfst
31950dea Add beam mode to result filtering
532925d4 Make sure TOP has the right name if it's eg. loaded from a @bin""
dc9bf4e9 Make sure the HfstTransducer name and the container names match
5289eb35 Fixes issue 353. Unbelievably, the return value for a weight-returning get_weight() was bool, which after casting mostly worked until it ran into negative weights.
2bfb55c0 Fix problem with rtns that are referred but not really present
2dd77678 Improve help messages of scripts.
15484805 Add hfst-specific options --with-c++11 and --without-c++11 to setup.py.
f2854007 Use HFST_THROW macro instead of plain throw for HfstExceptions.
4be886d5 If USE_FOMA_EPSILON_REMOVAL is defined, use foma for epsilon removal in minimization for unweighted TropicalWeightTransducers to make it faster.
7b03fd00 Add an option --verbose to test.sh.
5463306b Fix the way unordered containers are used on windows with python2.7.
48de6fdd Give pythonpath as first optional argument for each test, except for stream tests that take it as a second optional argument. Also specify coding for each test.
dbb055f7 round weight to zero decimals, non-sci, in cg/giellacg vislcg3 doesn't (yet) accept floating point weights
8d389257 Use by default c++11 unordered_map and unordered_set, unless otherwise specified via definitions (INCLUDE_TR1_UNORDERED_MAP_AND_SET and USE_TR1_UNORDERED_MAP_AND_SET).
31cd16c3 Merge branch 'master' of https://github.com/hfst/hfst
8f17d4ec Add boolean CPP_STD_11=True to control c++11 support.
baec6872 Move tokenize to correct alphabetical place.
194b9cca Couple of fixes to vc 2008 compilation without c++11 support.
38ad5946 Fix issues noticed when compiling without c++11 support with VC 2008 on Windows.
89ddca2c Merge branch 'master' of https://github.com/hfst/hfst
8121be60 Fixed bug 341 (priority union)
318fdb58 Add an option --without-c++11 (defaults to 'no') to compile hfst without c++11 support.
d3a5c5da c++ standard is set in configure, do not set it here
ee84e29d make hfst-tokenise usable as a lib; include simple string→string fn
fbf8d49a Update documentation and setup.
9617edb0 Add a quick fix to missing 'strtof' in msvc 2012.
b22626ed Add a function cross_product that takes an iterable object of transducers. Document other similar functions.
b1fbf8bc Add simple help messages.
195ef733 Add a function 'compose' that takes a list of transducers. Comment out a debug print.
8cb5ee5a Skip calculate_funtionality.py test until it is fixed.
b9aa6186 Add a script that copies files needed for creating a distribution for windows.
72b14638 Add a script for generating cc and hh files from yacc/flex sources for windows. Update copy script for windows. Change flex token ECHO to ECHO_ to prevent collision with flex/yacc macro with the same name.
0b61a94a Make script faster.
06a6c9dd More fsmbook python scripts.
88cc5bf8 Add a simple --help message to test.sh. Add python versions of two tests.
36490188 Add new python scripts.
3ec8d35b If --python is requested in test.sh, use python API also for hfst scripts that have an equivalent python script.
0ebc87e6 Add options --python and --pythonpath to test.sh which enable compiling xfst files also via the python API.
7266f932 Add an example for HfstBasicTransducer.remove_transition.
f2a547fe clang requires libc++ and osx version >= 10.7.
9a60444b Add function HfstBasicTransducer.remove_final_weight and document it. Also improve documentation of HfstBasicTransducer.remove_transition and HfstBasicTransducer.add_transition.
a2dbe8eb Add function HfstBasicTransducer::remove_final_weight.
9950d90f Add option --restricted-mode (-R) to hfst-xfst. If it is requested, Write and read operations are allowed only in current directory (i.e. pathnames cannot contain '/' or '\') and system calls are disabled.
059b5b02 Add tests for hfst-xfst's 'negate' command.
cdab3f74 Add a function HfstTransducer::negate() which is equivalent to [?* - A] (where A is the transducer) with the exception that flag diacritics are treated as ordinary symbols. Use the function to implement 'negate' command of hfst-xfst.
4001b451 Allow complement/negation only for automata.
a741bdc5 (tag: v3.12.2) Ready for release 3.12.2.
56534b34 Add missing file.
cc2dea54 Merge branch 'master' of https://github.com/hfst/hfst
22383de8 Check more carefully which backends are actually enabled.
4cebb756 Actually remove commented code instead of making it visible.
c811b357 Remove commented code.
357ab0fe Remove commented code.
f9796b35 Update README.
60742c42 Add extra compile option -std=c++0x.
25ce84e8 Reapply changes in commit de59747. Require at least automake 1.12, unless building from pre-yacc-generated sources.
750a3cae Revert changes in commit de59747. Allow automake older than 1.12.
0ae7b1e9 Use std::unordered_map instead of hash_map.
0b3a248f Use unordered_map instead of hash_map.
8b4b1d4b Skip lexc wrapper test.
d11b00e3 Add a simple exception handling mechanism.
02e16cbd Fix a typo.
de597474 Stop supporting automake < 1.12. Warn that build will fail unless building with pre-flex-generated sources.
e776ecc1 Remove commented code. Remove unnecessary variables.
8b30936a Skip hfst-train-tagger and hfst-twolc-loc tools in version and help message testing.
ba28c397 Add missing file to dist.
b29e140c Remove lexc wrapper tests.
4033c44e Disable lexc and foma wrappers unless explicitly requested. Fix a typo in --enable-expand-equivalences.
50351323 Actually enable hfst-calculate and hfst-xfst by default as promised in commit 244b9b5...
31fc7c70 Add missing condition.
5191c886 Add option --with-openfst-log=lean to configure which supports only reading, writing and converting log openfst transducers.
30d3792a Actually check if --with-sfst=lean is configured when compiling sfst functions.
94eda73a Add configure option --with-sfst=lean which supports a limited set of sfst functions (reading, writing and converting between formats).
5df548e3 Add missing const to definition, remove second argument of HFST_THROW.
214ab27b Add function 'has_weights'. Make function 'get_profile_seconds' public.
a27f845c Add function 'get_profile_seconds' for profiling foma back-end.
1b64f33b Add function 'has_weights' and make function 'get_profile_seconds' static.
244b9b57 Remove duplicate of AC_ARG_ENABLE([xfst]). Enable hfst-calculate and hfst-xfst by default.
e47f0100 Remove commented code. Make sure that code compiles when openfst backend is disabled. Add comments to some functions.
d0a1e91c Use iosfwd header instead of iostream when possible. Try to include only header files that are actually needed.
389679db Clean temporary files generated by tests. Add some files to be ignored.
610572fd Merge branch 'master' of https://github.com/hfst/hfst
f66f396c Check for nested context conditions, ignoring the inner ones
d8db5525 Update man pages. Remove man pages of tools that are no longer supported. Update copyright year.
62da656e Disable hfst-twolc script and hfst-train-tagger tool unless explicitly enabled with --enable-twolc-script and --enable-train-tagger.
e369608c Add files to be ignored.
142bfca5 By default, implement hfst-twolc as a single program and disable the script (can be controlled via --enable-twolc-script).
98e08f62 Get rid of hfst-twolc-system tool. The script hfst-twolc will soon be replaced by a single program.
e30202c2 Update windows scripts according to recent changes in twolc processing.
d6760437 Update files ignored by git.
ae820a5b Revert to earlier version of htwolc-main.cc which calls twolc parsers directly instead of TwolcCompiler class. The latter approach sometimes causes a segmentation fault.
d96b441c Define warning and error streams in TwolcCompiler constructor.
9c0cca49 Throw an error instead of calling exit(1) when parsing input.
b820f11c Handle twolc error and warning streams.
e116a74c Add TwolcCompiler class and use it in htwolc program. Move CommandLine under libhfst/src/parsers.
9a190880 Move most twolc code from tools/src to libhfst/src/parsers.
0705bec9 if no --superblanks, newlines won't be in [], so need to print on unblanked newlines
13512574 --giella-cg: only treat superblanks if given --superblanks
efa0964f Update list of files ignored by git.
19fd1d1b Add an option to use htwolc in twolc tests. Disable it by default.
59cfdce9 Rename twolc files.
a6029ef4 Rename twolc parser and lexer files. Move functions under namespace hfst::twolcpre[1|2|3].
5844621d Actually exclude htwolc from installation.
7661b389 Add an alternative implementation of hfst-twolc which does all processing in a single program. Exclude it from installation until it is properly tested.
52d52573 only warn for backtrack-on-substrings-without-analyses if --verbose
3936fdbe Control output of htwolcpre1.
0cc7582e Change names of extern twolc variables and add functions to access some of them.
f9a69b21 Add missing new file hfst_sfst_extensions.cc.
d5baf5ad Separate twolc parsing and command line tools to their own files.
035b6aba Use separate prefixes in twolc lexers and parsers.
d3c4de92 Handle '^<' and '^>' operators in lexc regular expressions correctly. Should fix issue #346.
61653471 Space-separated mode
144a3f34 Ignore generated sfst source files.
dc906026 Merge branch 'master' of https://github.com/hfst/hfst
9f7208bf Improve pmatch compilation error handling, still leaks memory at exit
3767b513 Add new function 'compile_sfst_file'.
512245b4 Merge branch 'master' of https://github.com/hfst/hfst
4649b83f Reorder and add norm caching
460cded6 clearer error message for when there is backtracking into substrings that have no analyses
0705c3cf Apply recent changes in filenames and locations.
3560d0b6 Move sfst compilation from tools/src to libhfst/src/parsers.
bc49e55b Fix namespace and class names.
9f356e7a Change file and namespace names.
3f8d4573 Small fixes to sfst compilation.
c28d2c53 Further separate sfst compilation and command line program.
0ccd6fe7 Reorganize sfst compilation into separate compilation units.
b9b85850 Fix xfst and twolc tests. Use rm -f when removing files that are conditionally created.
52d3cbf2 Remove all exe extensions from hfst-twolc and hfst-twolc-loc because they are scripts.
99d37e68 Remove exe extensions from hfst-twolc and hfst-twolc-loc because they are scripts.
b9b678ee Start separating sfst parser and command line program to their own files. Add a prefix 'sfst' to flex/bison functions and variables.
97338125 Rename HfstCompiler, hfst-compiler and hfst-scanner to SfstCompiler, sfst-compiler and sfst-scanner.
ea9fb6ba Add a test for hfst.regex using 'definitions'.
2a7ad84e Add a keyword argument 'definitions' to regex that takes a dictionary mapping transducer names into transducers.
6c761284 Add regexp operator documentation also to web pages.
2322ecc8 Modify regex documentation.
31559550 Tentatively add documentation about regexp operators.
77b64445 use ccache if possible, slightly faster travises
1ab384d4 Update fst_to_fsa and fsa_to_fst. Support both HfstTransducer and HfstBasicTransducer formats.
663224d8 Catch exceptions when compiling pmatch expressions. Add new tests for pmatch functions.
43e06449 Do not exit on pmatch compiling errors, throw 1 and catch it in command line program.
0556161b Merge remote-tracking branch 'refs/remotes/origin/master'
51b811ea Small math fix + revert to float, which wasn't causing problems after all
e1c9af03 Modify fst_to_fsa and fsa_to_fst.
aaf2f8b7 Support special symbols and single-character symbols without separator in fsa_to_fst. Update also documentation.
ebea326d Add simple tests for fst_to_fsa and fsa_to_fst. Also test modifying transitions of an HfstBasicTransducer.
ecb78d05 Rewrite HfstBasicTransducer's transitions() and state_and_transitions(). Tentatively add an implementation for functions fst_to_fsa and fsa_to_fst.
a65ca944 Remove HfstBasicTransitions and HfstStates from HfstBasicTransducer and use hfst::implementations::HfstBasicTransitions and hfst::implementations::HfstBasicStates instead.
4579f7d6 Give the pythonpath as command line argument for tests that use 'from __future__ import' instead of catenating it to the beginning of file. The import must happen in the beginning of file.
3dda726a Rename HfstTransitions to HfstBasicTransitions.
68155b3d Add docstrings and dummy implementations for functions fst_to_fsa and fsa_to_fst.
4a2814dc Document the changes to api.
3d1a1ce3 Use the version of HfstBasicTransducer.transitions() returning a reference. Add HfstBasicTransition.set_weight to python api. Add tests for modifying transition weights.
17e00271 Fix a typo.
04accecf Add help message to copy-for-windows.sh and comments to foma wrappers.
3688695f Typedef and templatize vector math, reformulate as per article description
95c022c6 Add help message.
1cd8cb85 Add help message.
dd2f2302 Fix vector reading for case where lines don't end with a separator
5bcb93db Support hfst.compile_xfst_file only for python3.
49959535 Fix a typo.
20f39cdc Fix copying files on mac as well as pip packaging issues.
7f600058 Update file instead of deleting it...
49b0ef97 Tentatively also support python version 2.
52016b22 Actually copy all files.
01744fa3 Omit sdk directory on windows if compiling for python3.5 or higher. Do not redefine _MSC_VER. Do not call subprocess on mac until it works correctly.
b44f91d5 Update README.
f6779722 Check if we are compiling with visual studio 14.
b17eb3d7 Update PyPI documentation.
a6ca66bf Add missing file.
410d6ef9 Get rid of unnecessary libraries.
f1572e1c Merge branch 'master' of https://github.com/hfst/hfst
75fcdd90 Fix chaining of functions in example.
d6552290 Minor version bump to combat repo inconsistencies
91539f01 oops, reset variable
55664266 add transliterate output mode
416373d4 Update documentation.
b5736dcb Add missing file.
43db07a0 Move ReplaceType under hfst.xerox_rules. Reorganize some internal functions.
bc8983e7 Move python code from swig interface file under module hfst.
3aff1428 Check if readline package is available.
391f73ab Do not keep input to interactive commands in readline history.
68e1b878 Update description.
0ff98211 Support apply up and apply down in hfst.start_xfst().
c687c097 Document return values of compile_xfst_file and compile_lexc_file.
12da4aef Warn user about missing readline support if --with-readline is not requested. Issue an error if --with-unicode-handler is called as such or with a value of 'yes' or 'no'. Make it possible to disable loading entries from shared object (openfst back-end feature).
9731e75a Update version number to 3.12.1. Handle links to future and earlier releases so that they are easier to update. Add a README.
764399e4 Remove doc/libhfst.py as documentation has been moved under directory doc/hfst/.
f3df0b9f Add more documentation for XreCompiler. Fix links to installation instructions.
d2666d6e pre-increment (thanks, cppcheck)
57120541 Add the script for creating python package. Convert README to dos format.
4abf93e7 Update package scripts.
4888fc6a Add a script for creating package which contains 64-bit python bindings for python3 for mac osx.
18df4685 Add a script fpr creating 64-bit python bindings for python 3.3 and 3.4 for windows.
47909865 Add README files for python packages for mac and windows.
3519e4c2 Add a script for creating the lgpl package.
0cb9fb53 Add alternative files for making LGPL release of HFST.
fea67231 (tag: v3.12.1) Add missing files to dist. Do not print termination message when performing tokenize tests.
65962bf8 Ready for release 3.12.1.
77c845dd Fix flag elimination bug (reported in issue #342).
c827bb62 Do not call fsm_destroy until issue with 'double free or corruption' has been fixed.
3b0e58a2 Provide a get_current_dir_name() for platforms without one
b0fb7a67 Merge branch 'master' of https://github.com/hfst/hfst
65a4d799 Search for included files under scriptdir, not working dir. If an included filename is non-absolute, prefix it with the directory the script was in when trying to read it. This only happens in non-Windows environments for now.
f5e94d73 Free allocated memory.
6144e75c tokenise -g: check for eof
cf46e69d Apply more changes from latest release of foma.
a9f8397b Apply some changes from newest release of foma.
cc8c96ba Make a separate directory python/pypi for creating pypi packages.
479789e3 Rename the pypi package to 'hfst' and use a four-number versioning scheme. Also define '__version__'. Add an empty config file as hfst c++ headers include it.
eb304f0f tests: trying sed instead of awk for exit on <STREAMCMD:FLUSH>
798383dc tokenize: handle Apertium-style superblanks in --giella-cg format
7922399a Provide a way to scale the vector similarity calculations via variable
166d418d Merge branch 'master' of https://github.com/hfst/hfst
9bc5315e parse word2vec -style text format and assume first line is info
5d3f6b8f Do not allow unescaped dots in regular expressions.
ce5e2a5f Merge branch 'master' of https://github.com/hfst/hfst
a8f21f0b Observe HAVE_SFST and HAVE_FOMA when including back-end header files.
8a9c0dec nwords catenate-style argument for Like() function
1433e453 Merge branch 'master' of https://github.com/hfst/hfst
1ec8e03e Add experimental two-vector model for word sense
72d4f72c (tag: v3.12.0) Ready for release 3.12.0.
da866477 Read vector models and do simple ranking
42df833e Small fixes to memory leaks and efficiency.
e2abd896 Fix some memory leaks.
da54acca Fix leaks reported by valgrind.
92cf4dc9 Fix more memory leaks.
9bd97498 Fix some lost memory issues reported by valgrind.
6e129a4f Remove unnecessary variable. Delete transducer definitions and latest regexp when destructor is called.
d7003095 flush given outstream, not necessarily cout
8d4f8249 Add a destructor to XfstCompiler.
6b10aabc Update README.
863e94aa Add missing files to dist and remove generated ones.
e4c13049 Rename hfst.rules into hfst.sfst_rules. Fix hfst.types into hfst.ImplementationType in tests.
80f18a82 Use single quotes instead of double quotes in docstrings.
52a9d75c Add test for hfst.xerox_rules.
00b020c0 Rename xerox_replace_left and xerox_restriction back to replace_left and restriction in package hfst.xerox_rules.
d1556dd0 Fix documentation of parameter 'output' of function 'extract_paths'.
3df4a48c Sentence separation for finnpos-mode
d3058c74 Issue a warning if automake is too old, which can make build fail if pre-generated yacc files are modified.
9166cdc5 Move ImplementationTypes and ReplaceTypes to their own classes and get rid of package hfst.types.
6d6c6315 Fix datatypes used in for loops.
c6aa5e16 Fix some typos in documentation.
8073675b Use renamed versions of xerox rule functions.
ae70056c Tentatively add HfstXeroxRules to python API in package hfst.xerox_rules.
dee0176a Fix some more warnings.
37edbbb9 Fix minor warnings generated by clang.
faa76472 Fix typos in ifdef guards.
41c5a32c Add a simple script for fetching statistics about hfst downloads.
d9d80771 Update pypi setup.
822f4bb8 doh, forgot to add test expected output
cccba226 hfst-tokenise giella-cg: fix indentation issue with backtracking
5a262692 Fix still some warnings.
32c82b30 giella-cg backtracking: ensure ignored spaces are still output
03d6a237 Do not warn about pragma push/pop issues or unsafe cstring functions on windows
9c12094a Use hfst::hfst_fopen instead of fopen. Fix some warnings related to datatype conversions.
d64cb49a Add function hfst::hfst_fopen that uses fopen_s when compiling with msvc.
5ed27eae Fix some warnings caused by conversions (possible loss of data).
b93a1442 Do not use extern templates when compiling with msvc.
73f7e18e Remove files generated by the tests.
61ac4f83 Add a batch file version of test.sh for testing the python bindings on windows.
b53a128a Fix warnings.
7cb8740c Update pypi setup script.
cc9cca46 Remove unnecessary back-end include directories as well as commented code from Makefiles.
fd5f0074 Fix some minor warnings.
ec114ea8 --giella-cg support for @PMATCH_BACKTRACK@ symbol (#337)
15fbb1fc Update windows script.
097826c1 Remove extra comma from initialization.
d6b001d7 Add missing header file.
9b3c1227 Fix warnings generated when compiling with -Wall.
a0dee642 Fix most of the warnings generated when compiling with -Wall.
89b20612 rename --gtd to --giella-cg, and more descriptive help text
335df14e Add support for compiling on windows.
85585c4b Fix a typo.
e628f556 Add a manifest file for pypi packaging.
48c34ef0 Tentatively add a setup file for creating pypi packages.
6abde490 update to proc to make it possible to weight surface forms
5e379fc7 Update windows compilation script.
1f55a514 Define int64 for msvc.
21f52d0e Implement snprintf for msvc compilation.
c05469e2 Replace definition _Bool with Boolean, as _Bool conflicts with typedefs in msvc header files.
00bbb1d2 Get rid of unnecessary typenames and typos noticed during compiling HFST on windows.
b160c07d Forward declare back-end implementation datatypes and wrap their destructors.
067c3c92 Add missing <iterator> header.
2f0c8dff Add missing std:: to vector.
34e99ad6 Issue an error message if glib unicode handler is requested but not found.
d28075c3 Declare all functions that are needed in start_xfst().
ec34dea1 Add math.h
4a3be6b7 Make preparations for supporting OpenFst version 1.5.4.
c9940f65 Include HfstBasicTransducer.h instead of HfstTransitionGraph.h.
8560638b Add extern templates.
dc969538 Remove unnecessary OpenFst headers and use forward declarations instead.
699c0985 Wrap StdVectorFst and LogFst destructors as both classes are forward declared.
7f81f41b Forward declare as much as possible when using OpenFst library. Also remove unnecessary include directives and add missing std namespaces.
a0b2fb35 Prepend missing 'std::'.
c8137fac Make HfstBasicTransducer and HfstBasicTransition fully implemented classes instead of generated template specializations.
eb5b877a Improve the way error messages are handled.
b8278ec5 Change 'import libhfst' to 'import hfst'.
2b9e44b6 Move implementations of HfstTropicalTransducerTransitionData away from header file.
ce05c672 Merge branch 'master' of https://github.com/hfst/hfst
3ec0d09d Alter expected result to conform with newly expected behaviour
759605c7 Do not allow newline characters inside quoted literals.
f296b6bd Merge branch 'master' of https://github.com/hfst/hfst
63d7544b Allocate enough memory for XRE error message.
f968ebfa Trying out facility for communicating sentence boundaries to plain tokenization mode
6a78c4c2 Merge branch 'master' of https://github.com/hfst/hfst
bf2ded3e Don't print size info when unverbose
5a33841f Don't separate tokens with empty lines in plain tokenization mode
d0a1b550 Warn about undefined multichar symbols when compiling lexc in verbose mode.
5d2c67d8 Fix a typo in option handling.
e2fee814 Tentatively add an option --do-not-minimize to hfst-regexp2fst which performs determinization instead of minimization. Also make hfst-xfst use determinization if variable 'minimal' is set to 'OFF'. Implement this by adding function hfst::set_minimization(bool) to HFST API and a function HfstTransducer::optimize() which performs minimization, or determinization if hfst::set_minimization(false) has been called. Replace calls to minimize() with optimize() in parts of HFST code relevant to the tools mentioned.
82aa26ec Get rid of CR-LF newlines.
44f2e212 Remove commented code.
94b76073 Print an error message if output type is defined several times.
966a8954 Fix mismatched free/delete reported by valgrind.
6dc1d8f4 Do not use strdup with -std=c99.
407a5178 Use delete_foma instead of delete/free when deallocating memory used by a foma transducer.
07508b50 Limit to C++11 for now
d15b4be3 Use free with malloc, not delete.
b693b3af Add a function 'void LexcCompiler::reset()' and use it in XfstCompiler when reading lexc input again.
bc5ac275 Remove revision info from output of hfst-info as it doesn't work correctly in git.
ace48016 Pass cascade argument by reference in lookup_cascading functions.
e2ea4cb1 Fix a typo in function 'strip_percents' which resulted in incorrectly escaped zeros.
92a7152e delete more lines
9f940a35 remove osx lines from yml
f7bd0858 Unobservable change - nothing happened here
d1bf79b0 close #336: off-by-one segfaults in OlLetterTrie
2574e27a Avoid using projections in casing functions (which cause minimization issues with large weighted transducers).
73a3ff0a Parametrize optionality in casing functions
0adc1f93 Make sure there's no case fallthrough
58eb35db Fix other casing functions too
c366c4cb Another fix to Cap()
704c1c7d Merge branch 'master' of https://github.com/hfst/hfst
5a5df2d7 Change operation of Cap() and make timing per-object
44f0ae7a Fix missing empty lines in output when cascading composition is used.
284f9d0d Add an option --max-number=N for OL transducers which limits the number of results. If not defined, defaults to 5 for infinitely ambiguous input and to infinity for finitely ambiguous input.
5686864b There are some problems in using the bindings with python version 2, so refer only to python3 in README.
5a17fb02 Add a missing 'break' to switch block, making variable infinite_cutoff work correctly.
4c2f26fb If no results are found, print it only once.
d703776c Tentatively add support for --xfst=print-pairs for OL transducers.
b7fcab93 Remember to flush the streams also when no results are found.
7ba32cf8 Merge branch 'master' of https://github.com/hfst/hfst
bc7a80a5 Implement a pair-lookup for optimized-lookup transducers.
ce3133fd Add a fourth argument of type hfst::ImplementationType to the constructor and a function get_type() to ImplementationTypeNotAvailableException.
4ffe105a Print to console on windows if print-pairs is specified.
587f3c04 Do not print extra newlines in cascading composition.
43538943 Fix a bug/typo in string comparison.
72de7821 Improve documentation in HfstTransducer.lookup.
0c888e9c Rename HfstBasicTransducer.lookup_fd to HfstBasicTransducer.lookup and add an argument 'obey_flags'.
d5493aca Rename HfstTransitionGraph's lookup_fd to lookup and change some parameter names to be more descriptive.
56a8d2f6 Add instructions for linking hfst c++ library statically in comments.
67e25ffd Add missing file to windows scripts.
8db14ce0 (tag: v3.11.0) Ready for release 3.11.0.
fe37aa00 Update morphology checks.
8d3f162b Update EXTRA_DIST.
6dac2464 Do not use binary transducers in tests.
ee62c447 Add missing doc file.
a3676454 Mention that python tests are no longer available.
da2678db Skip Python tests as they can now be performed in the folder python/test.
ea47f0b1 One more stray memory leaking cache
95e3480c Memory conservation: only cache named objects and always minimize them.
6b113b24 Merge branch 'master' of https://github.com/hfst/hfst
974d2403 Turn off automatic delimiter-contexts ("set need-separators off") in the presence of user-defined contexts
01fccfd5 Use docstrings.i instead of swig's autodoc. Document the contents of hfst and hfst.exceptions modules.
4322ca72 Add a docstrings file created semiautomatically from doxygen documentation.
816ce6c1 Add at least a simple docstring for all functions.
82c8f6a5 Merge branch 'master' of https://github.com/hfst/hfst
1b11fbff Obey -i option for giving the pmatch archive filename
4620b1b7 std=c++ works on OS X but not Linux, so disable Linux+Clang
d245a745 Test if gnu++ appeases Travis' clang
3d9c66b5 Include needed m4
5e73db33 Copy hfst-ospell's C++11+ detection
9af66d0b short int -> short; long int -> long
daa457e9 Yet more documentation..
e1b01ff0 More docstrings.
711a0898 Add docstrings.
acf56964 tokenise: don't skip output-empties
b5b4434c tokenise: don't let empty analyses affect weight classes
18f5a2b6 travis: fix indentation
62d216a6 getopt_long() returns an int, not char; this is important because char is unsigned on many platforms which means char(-1) is 255
39d28d1f Do not check if lookup is infinitely ambiguous if --time-cutoff is defined.
c42af376 Remove redundant flag diacritic check. It was once removed, but got pushed there again.
b7b58825 Add support for --cascade also for optimized-lookup transducers.
9169e38a Support combination --cascade=composition --xfst=print-pairs.
51a0b61a Merge branch 'master' of https://github.com/hfst/hfst
ef6ed81d Fix the way cycle limit argument is passed.
9ad09503 Merge branch 'master' of https://github.com/hfst/hfst
6d94c906 Add a preliminary omorfi-friendly conll-u mode
45a0bf29 Remove redundant filtering of paths containing flag diacritics. Flags are handled already in function HfstBasicTransducer::lookup_fd.
5c7135ac Implement option --cascade.
18d5e66e Tentatively add option --cascade to hfst-lookup.
1dcad09d Print error message if --xfst=obey-flags is used with optimized-lookup transducers. They always obey flag diacritics.
670ee39d Merge branch 'master' of https://github.com/hfst/hfst
f903d9cf When doing apply up or down in hfst-xfst's, check if 'obey-flags' is ON or OFF.
8e9c6e65 travis: --enable-silent-rules
a253b8d4 Add an argument obey_flags=false to is_lookup_infinitely_ambiguous functions.
14bdd9c7 Make is_lookup_infinitely_ambiguous aware of flags. Use typedef hfst::StringVector in HfstLookupFlagDiacritics to prevent conflicting definitions in hfst-twolc which defines a class StringVector.
349fb735 Add option obey_flags=false to lookup_fd.
fdf912ab Add tentative script for converting from doxygen documentation to docstrings.
7618907d Modify function visibility.
bc01c0bc Control visibility of functions in HfstTransducer.
b535aa6f Follow foma and xfst convention for operator ignore '/', i.e. expand any identities and unknowns before inserting the strings to be ignored.
c1b43412 Add an argument 'force' to HfstTransducer::harmonize which makes the function harmonize also FOMA_TYPE transducers.
724b4e95 Obey variable 'print-sigma' when print_net is called.
6bec5076 Add brief descriptions for hfst packages.
ca892d14 Document the implementation types.
98764da7 Fix bugs in options --xfst=show-flags and --xfst=obey-flags when non-optimized lookup is performed.
93b9fe27 Close the outstream when done. Not doing this was causing a bug where some buffered data never got written to the output file.
7d971066 Obey variable 'print-pairs' when performing apply down in non-optimized format.
ac2fc3ab Add parentheses in conditionals of function 'flag_build' to ensure correct interpretation. Also remove redundant code.
28867c16 Allow punctuation as context in the "need-separators on" mode
0cd05ee8 Add "need-separators no" to test script to reflect changed default in pmatch
25977996 exclude gcc from osx builds
7dfa8ec8 Track used symbol names
81128c23 Allow lowercase define()
52a4ca45 Merge branch 'master' of https://github.com/hfst/hfst
af71662c Add side arguments to casing functions
7c1b3886 Revert back to the original extra_link_args in python bindings.
2d5c670a Rename examples.py to test_examples.py. Perform tests in a more logical order.
4c8a1e30 Update documentation about linking to the HFST C++ library.
9f8ca0f6 If --enable-no-tools is used, skip tests in test/tools/.
41dc9a27 Mention module hfst.types.
a8c3e566 Remember to apply modifications to weight pushing to tests and hfst package.
835f0efc Get rid of TO_FINAL_STATE and TO_INITIAL_STATE and replace function 'push_weights' with 'push_weights_to_start' and 'push_weights_to_end'.
d05b9dbf Wrap HfstTransducer implementation types in package 'hfst.types'.
5e1a2456 Add missing files to dist.
58d46022 Update README. Handle linking to extension library better.
de888b6f Update README. Handle cases where an implementation type is not available. Add missing files to dist.
1a68126f Fix a typo in URL.
7a2296be Update link in README.
861baf6b tokeniser: --weight-classes=N opt, as in hfst-proc
151f1fa4 Add functions lookup_optimize, remove_optimization and copy to HfstTransducer.
4efc9af7 Make HfstTransducer() create the empty transducer.
0f237c17 Comment out hfst_ol::set_extract_tags_mode(bool). Improve error messages in att and prolog readers.
57426cae Remove unnecessary commented file.
b7139cde Fix HfstException documentation.
3b13bf58 Merge branch 'master' of https://github.com/hfst/hfst
7e6d27c9 Fix a typo.
9f4ceb76 Merge branch 'master' of https://github.com/hfst/hfst
829c9e21 Implement remaining variables in the library and support them in hfst-pmatch, with the exception of need-separators which has to be specified during rule compilation, at least for now.
457c5de5 Document AttReader and PrologReader.
8695c47c Add HfstException::what() to python api.
2cffffac Add an alias what() for operator() to HfstException.
5fdf61a1 Add class PrologReader and tests for it.
77eda555 Add a new class AttReader and tests for it.
18edac59 Add package hfst to setup. The package is essentially a wrapper around libhfst module which is still available as before.
e41f131e travis: re-encrypt channel to #hfst
6329bb89 Use all implementation formats in tests.
5a5018c6 Do not use default_fst_type as pmatch transducers are always in ol format.
b38f87bd Add stream tests.
ccaea0e0 Represent empty transducers as empty lines so that read_att_transducer will work correctly.
fe66b184 Remove unnecessary files.
bf5511d5 Forgot to do git add before committing...
3710f91b Update test examples.py and add it to test.sh. Remove extra files.
7d5064de Add directory tree for package 'hfst' and its subpackages 'exceptions' and 'rules'. This package encapsulates module libhfst.
7818c1ee Delete unnecessary file.
91b86a0e Rewrite part of tests. Document the interface file.
657013d5 Add new tests.
3b4c1867 Merge pull request #323 from unhammer/tokenise--gtd-tags-are-multichars
67125ab1 rm unneeded import
d927d795 pmatch: store input/output_symbol_strings, _parts refer to them
c665f8a8 Add missing endverbatim tag.
884d1dcc Add missing file.
bc87fc58 Get rid of HfstFile wrapper.
30cde028 Try to preserve the names of transducers when converting.
90b33287 Expand prolog tests. Fix a small bug in naming transducers.
8e61537f Forgot to update testfile.
63a04567 Add a new function for reading prolog input and improve the way att input is handled.
9ef38f7d Split part of code related to reading att input into a separate function. Change the way transducer name is set when reading prolog input.
6d12e0d4 erge branch 'master' of https://github.com/hfst/hfst
22f1c475 Modify documentation, update tests, add new functions.
d14bc2f2 link to travis; bug tracker link sf→github
e8c7fd61 README is now symlink to README.rst; for more readable github
22583542 travis: no irc notes on success
1d8a1f42 travis: encrypt IRC channel for hfst/hfst
38c95442 Add HfstTokenizer examples as tests.
c635cd4d More documentation to HfstTokenizer.
c1ca3b64 Add more examples for HfstTokenizer.
0dc0cf42 Reorganize doxygen documentation to represent the new module structure of hfst python api.
95d658c0 Remove hfst.py, a directory structure is needed instead for doxygen.
5025aabb Add new documentation for hfst python package.
4749745c Add more tests.
8f8b345f Split python tests to several files.
45aeb38a Skip one more failing pmatch test.
83dee76a Add missing files to dist.
a44b99e8 erge branch 'master' of https://github.com/hfst/hfst
b5ca1ec3 Add more documentation and modify functions.
579ee713 Merge branch 'master' of https://github.com/hfst/hfst
80505754 A PmatchContainer constructor taking a std::vector<HfstTransducer>. This should leave the passed vector untouched.
88aa6829 Add more documentation.
ab820477 Add a simple test for XreCompiler.
a25eb2b4 A new test.
caf1ca8a XreCompiler::setOutputToConsole is needed, uncomment it.
f6acf2a1 Split c++ code to multiple files from libhfst.i. Update URL in setup.py.
780374ae Merge branch 'master' of https://github.com/hfst/hfst
4c9ca194 Settle on a way to handle variable names with dashes (which look like minus signs)
967cc78e actually run the tokenize tests
5df87b6d Add more failing tests.
775a499d Fix some typos.
2e77e2df Add missing files to dist.
11ebca0a Merge branch 'master' of https://github.com/hfst/hfst
067e637e Merging master
4c9d43a5 Variable setting and reading
881b81cf Use -Wno-deprecated also for tests.
85cb2773 Modify messages that get printed during make check.
07617302 Skip tests that do not exist. Make prints more informative.
573324e9 Skip failing pmatch tests.
164e621c Add missing tests for ConvertTransducerFormat. Compile them with -Wno-deprecated.
f945e17d travis: irc notifications on build failure
0229baad Merge branch 'master' of https://github.com/hfst/hfst
93ed0db8 Remove const declarations from openfst back-end that cause compiler warnings.
5df2a2bc Run g++ with -j2 (not -j) on travis; o/w runs out of memory
54b8f45e Comment out a function that is not defined.
48c91b82 Specify -Wno-deprecated also for tests in folders hfst-tagger and hfst-twolc.
810c3306 https://travis-ci.org test rules
708171e2 doh, define multichar-symbols (for 2eb6888)
2eb68888 test for #320 / bf26db9
3aecdbc2 Merge branch 'master' of github.com:hfst/hfst
bf26db9c only change case if sf and lm case differ (#320)
b3782dee Remove a rule that cannot be matched since a rule has already been defined for "DownCase(" earlier.
5ad3487d Add the keyword const to arrays used to store xfst commands.
79c7dfed Make compiler happy by using strdup instead of plain const char *.
f6d99530 Handle return values of fread in foma and sfst back-ends to prevent warnings during compilation.
214b7b68 Fix a typo.
862e858f Replace foma binary files that are used for testing results from lexc with equivalent files in prolog format. Then the tests will not depend on the foma back-end.
680389b8 tokenize tests
2b7cf7e6 tokenize: new --gtd Giellatekno/Divvun mode
adb37d47 tokenise: fix #318, header parsing for TOP
76e9069f Fix a bug in flag elimination involving multiple flags. Fixes bug #315.
f6537805 Evaluate transducers (as opposed to argumentless functions) where declared
e6c6e82d (tag: v3.10.0) Ready for release 3.10.0.
97c12853 Swap implementations of commands 'apply up' and 'apply down' in hfst-xfst, making them work as they do in foma and xfst.
85ef34cd Forgot to add before latest commit...
13280ea6 Edit warning about converting native foma transducers.
b31a6c12 Fix typo: swap left and right.
808d9840 Add hfst-flookup to windows scripts.
66db2854 Update bug tracker url and copyright year.
b127ab31 Git ignore hfst-flookup.
1b40de68 Add tests for hfst-flookup.
12e4ccab Trandsducer operations no longer return a reference to the transducer, fix this in tests and interface operations.
a128c7f4 Add a tool hfst-flookup which does lookup from right to left, i.e. in the same way as foma's flookup and xfst's lookup. hfst-lookup and hfst-optimized-lookup stay as they are, but mention in their help message that they do lookup in the opposite direction.
cea21b49 Add a comment about the way gzipped input is handled.
da7ebcc3 Make HfstInputStream constructor throw a FileIsInGZFormatException if input is in gzipped format. In hfst-fst2fst, issue an error message that asks user to unzip the transducer.
5c2755b0 Merge commit frankier:add-gitignore manually and also add more files to be ignored by git.
99b470d9 Merge pull request #313 from frankier/escape-dot-labels
f6905a1e Improve handling of argumentless functions and funcalls
d804d3d2 Remove some memory leaks in compilation
ef740e09 Fix nested funcall issues
61e1e0a1 Add optional versions of cap functions and AnyCase().
68e078b0 Corrections and additions to casing functions, and add Implode() and Explode()
2484633b Add a facility for detecting when hfst-tokenize is given a "bare" dictionary and generating a somewhat intelligent tokenizer when that happens. Due to awkward architectural reasons this involves changes and additions in many parts of the library, but they are not major.
089f3ca6 Add .gitignore
d1d81db3 Merge branch 'master' of https://github.com/hfst/hfst
485f5755 Fix bug that caused exclusionary lists to be missed much of the time.
03ff992b Escape " in dot labels.
99840963 Factor out replace_all into string-utils.cc.
ca10d770 Revert swapping of hfst-xfst's 'apply up' and 'apply down' commands.
7352cf70 Swap implementations of 'apply up' and 'apply down' in hfst-xfst, so that these commands work in the same way as in foma and xfst.
db2d4dc0 Add an example for HfstTransducer constructor.
f37b8d16 Tentatively implement lookup also for transducers that are not in optimized-lookup format using composition and path extraction.
1cb29924 Add an HfstTransducer constructor that takes a StringVector as argument.
f32c0498 Support python 2 unicode strings in lookup.
8d2c6b85 Fix a typo
f442b91d Make HfstInputStream iterable by implementing functions __iter__, __next__ (for Python 3) and next (for Python 2).
2c660ff4 Add functions in namespace hfst::rules to python API.
947bb62a Add some information about Python interface in README and advise the users to see file python/README.
067b0c24 Remove slashes from headers used when processing the file.
c4b5ce82 Update documentation.
0e4e8a72 Redefine functions which return a reference to themselves as void to avoid problems on the python side.
5f97eac9 Tentatively add a C++ function 'std::vector<hfst::HfstTransducer> compile_pmatch_expression(const std::string & pmatch)', mostly copied from hfst-pmatch2fst.cc.
21da5004 Add a const declaration to string& argument of PmatchContainer::process and PmatchContainer::match.
cd7f2f88 Tentatively add some pmatch functions. Add a missing typedef for HfstTransducerVector.
a9fbd44d Add a function to HfstInputStream for finding out whether hfst headers are included in transducers. When reading native foma transducers in hfst-fst2fst, warn that inversion may be needed for lookup to work as expected.
fa4565ec (tag: v3.9.2) Release 3.9.2.
3f7a57ee Update hfst binary file used for tests.
5cd8a71d Add option --pythonpath to python tests. Document the test tools better.
927da9c9 Make fst2txt functionality test silent.
ad8c82c0 Update installation check scripts.
ddf749d4 Remove test.py from EXTRA_DIST, the tests have been rewritten.
dd7d0be0 Merge pull request #309 from unhammer/pmatch-input-marks
97b6e3fb Merge pull request #308 from unhammer/tokenise-minor-cg-fix
37818064 Make small modifications to documentation and typedefs used in function declarations/definitions, so that doxygen can generate documentation right.
00d89ce4 Update Python and C++ API documentation files.
4319b945 Move api documentation file under folder 'doc' too.
67e1e8c3 Move C++ documentation file under folder 'doc'.
8e2e6519 Do not catch exceptions derived from HfstException inside the constructor of HfstInputStream and throw them as HfstException. Instead, throw them as such.
56ffca8f pmatch: special symbol @PMATCH_INPUT_MARK@ for input marks
90aa6951 pmatch: on seeing a special symbol, mark the offset in the input string
bc345c64 CG forms always go in "<>", not before the tab
8dceb24a Merge pull request #307 from unhammer/tokenise-short-options-missing
2c207de6 short option -m missing from getopt_long
a9314ad2 Add m4 for older distros
6c944bac Bump patch version since revision has decreased
859f8da3 Add builting function Interpolate()
c6f9908e Fixed condition for including symbols in exclusionary lists Rather than the printability vector, which may not be completely set yet, check the symbol itself for printability.
780b9af9 Fix bug involving stale caches of function call results
26d3d8f6 Change python3 into python in README as version 2 works now too.
7af6f3f2 Move tools/src/hfst-string-conversions.[cc|h] under libhfst/src/.
a1b5bb20 Update version number to 3.9.1.
d23eb93b Do something useful in the case of a missing root rule
9e4e6638 Make Makefiles in openfst back-end licensed as LGPL.
03e1949e Candidate for release 3.9.1.
27f80ee4 Fix names of files whose names were changed.
8e4f3240 Set server based search to NO.
5b2c6988 Add word boundaries in xerox mode.
6e78e1fb Fix copypasted typo
b6970ab3 Fix ToLower and ToUpper, which had transposed names and flawed implementations
c9394139 Use iterator instead of range-based loop on windows.
e5897fb7 Fix a couple of typos inside 'ifdef _MSC_VER' detected when compiling on windows.
7694ee0b License for the library code is now lgpl 3.
b8a5b989 LGPL license
a0315269 Compatibility with python2.
1baee136 Add exclusionary lists (eg. Exc(Whitespace) matches symbols not in Whitespace)
a92e1d06 -X option fixed in hfst-pair-test. Additionally, the user gets an error message if unescaped :-symbols were found.
8d0a0289 Fix a bug in HfstTransitionGraph::find_replacements. Implement compile-replace without composition in XfstCompiler to avoid alignment issues in result.
887381eb Fix epsilon handling in compile-replace expressions.
2891a27e Allow epsilons inside compile-replace expressions as xfst does.
bcfea4ac Warn about '@_EPSILON_SYMBOL_@' not being an ordinary symbol in hfst only in verbose mode.
5fc456b5 Allow space after operator '^' in regular expressions.
07587308 Fix binding precedence of :
9f4628ed Add throw to appease rpmlint no-return-in-nonvoid-function
78efc5d8 Fix a bug in function 'fst'.
23a52850 Print a blank line even if there were no results or if tokenization failed
73d317be One && too many
5f4a8fc0 Rename hfst-proc2 to hfst-tokenize, with hfst-proc2 and hfst-tokenise as aliases
732e37e3 Add tests for python bindings.
51cbc5e3 Complete rewrite of pmatch compilation This is a cleanup and rewrite of pmatch compilation, hopefully allowing easier extensibility and maintainability in the future. While this has been reasonably well tested, it is likely that there are some regressions; please report any. Various additional features are pending.
31de91e5 Move scripts under subdirectories and remove scripts that are not needed.
61562159 Move windows scripts under scripts/windows. Get rid of libhfst_win.i.
0bf33d73 Remove HfstFastTransition.h also from dist.
02fa3263 Remove HfstException.cc and HfstException.h files, HfstExceptionDefs.cc and HfstExceptionDefs.h are used instead.
93565178 Remove class HfstFastTransitionData and the template HfstFastTransducer that uses it. HfstBasicTransducer is the template of HfstTransitionGraph that is used everywhere and it is fast enough.
dd6873e4 Remove tentative reentrant version of xre compiler. The normal xre compiler can handle expressions that the reentrant one was originally intended for.
05c78d8d Update python API documentation.
f9335250 Remove old commented code. Add a warning about python bindings not being under autotools; advise users to see python/README.
b127d489 Set -std=c++0x unless a higher standard is required with the -std option.
f2738be1 Do not generate hfst-scanner header file. Make hfst-scanner depend on hfst-compiler header file to make sure they are generated in the right order.
a44eb61b Check automake version in configure to determine whether header files generated by yacc/bison should use extension .h (automake < 1.12) or .hh (automake >= 1.12).
7f206c09 call_counter is a member of the container, not the transducer
2aeda9f3 Reweight arcs iff all conditions given with --input-symbol, --output-symbol and --symbol are met.
fddc9ea1 Forgot to increment call counter
a56089b1 Only call clock() every millionth time
1c4c871d Add missing declarations
a710ddda Only check clock() every million calls to get_analyses() to save time
8cffb74c Move start_clock setting to where the line is read to make the time limit per-analysis
6488877c Add short option for time cutoff (-t)
b6fdeb65 In pmatch, make the time cutoff "soft" to ensure we at least return something (this was an issue for rulesets with RTNs only - all the time could be spent in one leaving nothing for anything else, causing needless failure)
70e025ff Ignore previous commit, I had some old library code confusing me
8dfcd668 Add time cutoff options
8a9f0c33 Fix what seems like a typo in hfst-lookup, &infinite_cutoff for infinite_cutoff It seems like the size_t and size_t* can substitute for each other, but the prototype is for size_t.
2134ef4a Fix time cutoff check in ol library code
8ff4aa34 Omit infinitely ambiguous check when we have a time cutoff
969d479c Fix time cutoff handling
dd30c338 Switch from !€ / !$ to !!€ / !!$ as the test case prefixes, to bring it in line with the conventions in the rest of the Giella infrastructure (where this testing setup is used).
fd175f1f Fix option --to-transducer of hfst-substitute.
d213839f Update HfstTransducer::priority_union declaration. It takes only one argument instead of three.
0dec5067 Add new test files to dist.
7915ca01 Priority union bug solved.
daee8f74 Update output of morphology tests.
1148413c Add a check for hfst-ospell.
a179a32b Add new replace markup tests to dist.
c56d1f0e Markup rules completely changed
cb9f08af Update installation check.
5a174af6 Finally ready for release 3.9.0.
0ec3e97c Add missing files to dist.
61d1d148 fix for --xerox bug#328 introduced by weight fix
b6e49fd9 Revert -r4579 "Revert changes to hfst-proc made in revision 4427. This will (temporarily) fix bug #328."
f0e85c81 Remember to update the man pages, too.
e31c5d96 Ready for release 3.9.0.
64c985c0 Revert changes to hfst-proc made in revision 4427. This will (temporarily) fix bug #328.
9ca8ab03 Change all pythoncode delimiters from '{' and '}' to '%{' and '%}'.
128331b8 Add two missing %
7b961c52 Add tests for hfst-substitute.
67f3dce9 Add tests where one or both inputs are archives.
26aa3315 Set name of composition transducer after composition is called.
1e2ed004 Add tests for binary tools that use input streams whose types differ.
4c1e6559 Make HfstTransducer::compose throw and exception if transducers have mismatching types.
e21378d8 Implement conversion of mismatching transducers also in hfst-substitute.
43e107de Make binary command line tools convert mismatching input streams into common format.
573609f8 Make 'harmonize_flag_diacritics' throw an exception if transducer types differ. Make 'is_safe_conversion' public.
d7508470 Fix a bug in hfst-regexp2fst related to comments and remove support for legacy way of defining weights from regexp parser.
ee3b02c6 Xre parser - added check if contexts in replace rules are automata
d5258af2 Implement commands 'view net' and 'write dot' as system calls for non-windows platforms.
7e041e81 Update windows scripts so that they will work with the latest changes in dot and pckimmo printing functions.
8cb95f8c Move dot and pckimmo related functions in their own files.
70199c88 Throw an exception in HfstTransducer::eliminate_flag if flag feature does not occur in the transducer or the flag includes a value or operator.
6d723cad Add missing file
aa6325f6 Move python tests to their own directory.
bf0db238 Rename directory 'swig' to 'old_python' and directory 'new_python_api' to 'python'.
da9b806c Add information about the possible values of option --xerox-composition to the help message.
f0682841 Modify infinite_cutoff and time_cutoff arguments in lookup functions.
8645f60d Lexc, removed option -M + cosmetic changes to the alignment
95e18d3d Handle standard and console streams when libhfst is run from IDLE.
1275140e Throw a FunctionNotImplementedException in HfstTransducer::lookup_fd instead of converting HfstTransducer automatically into HfstBasicTransducer.
7d112762 Update python interface.
dc5a1f47 Added option -A to lexc, it aligns same input and output transitions
1d3e8af7 Fix a minor bug in python interface on windows.
897a7dde Fix LexcCompiler::printConnectedness const declaration.
de59f0ed Replace rules support flag diacritics (fixed lexc with -FM flags)
6817b355 Use separate ostringstreams for standard output and standard error on Windows.
f30e6e9e Improvements to error and output stream handling.
1178005b Support printing to console on Windows.
0f0dd44b Update foma back-end composition algorithm to fix a bug noticed in lexc tests.
6e93227e Replace system exit calls with HfstFatalExceptions.
c73aa33e Check for epsilon cycles with negative weights.
4679a3a1 Optimize function 'has_negative_epsilon_cycles'.
3bd5ce00 Add a test for checking that unknowns are not expanded into flag diacritics.
6d803dff Add option 'check-negative-epsilon-cycles' to hfst-txt2fst and tests for it.
a678e81a Add function HfstTransitionGraph::has_negative_epsilon_cycles.
4405b781 Improve performance of transducer harmonization.
12d3d8ef Time cutoff for the bundled standalone optimized-lookup tool too
05e7b6bc Add parameter to control output in 'compile_lexc_file'.
879d01e6 Add function unsigned int LexcCompiler::getVerbosity().
acd442ee Allow user to control output in xre and xfst compilation.
cf2a1d79 Improve error handling in lexc compiler.
688823f6 Improve error handling mechanism in xre and xfst parsers.
083f8483 Add a missing period to the end of symbol lines when printing in prolog format.
50f7ebf9 Added warning about very slow lookup when using HfstTransducer::lookup_fd.
0cf95533 Added warning about very slow lookup when using HfstTransducer::lookup_fd.
c8458dcb Added possibility to transform individual transition weights in HfstTransitionGraph.
fb502563 Add option to suppress multichar tokenization and default to on in proc2
8e370a6d Replace part of XfstCompiler's FILE pointers with equivalent calls to ostream.
5a773d1d Tentatively move towards o(string)streams instead of FILE pointers when printing output.
b792ca8f Remove debug printing
42585f51 Implement a time-based search cutoff for hfst-lookup in optimized-lookup mode only
2aafec31 Make 'define_function_args' return a bool instead of calling exit on error in xre parser.
18d6b121 Fix error handling in xre and xfst compilers.
d1f2b2ab Rewrite regular expression examples to be more informative.
52dc82e2 Scale weights to be non-negative in function n_best.
21d8577c Update documentation.
5c71e4bf Small fixes to read_att_string.
727e74fd Update documentation about special symbols. Add a function for reading AT&T strings.
7b2e11d0 Add function start_xfst() that starts an interactive XFST compiler.
ca3143ce Update documentation between HFST and backend conversions and add the examples to tests.
9ae43113 Fix issues related to level of verbosity when compiling xfst and regexps. Add missing documentation.
94e7535f Merging of modified or added functionality from the xre parser
116a0765 Remove obsolete comment
9130ba19 Revert HfstXeroxRules files to version 4484.
04d3a644 Rewrite 'compile_lexc_file'.
4988bb64 Improve handling of error and exit conditions in xfst.
4ba4909a Add missing namespace: 'string' -> 'std::string'.
b3c446d7 Greatly reduce memory leaks when using hfst-ol format
f9b44a44 Update new python API.
3239aa92 Update windows scripts after moving XfstCompiler.
ba65f77a Move XfstCompiler from tools/src/parsers to libhfst/src/parsers.
330447dc Add three missing files to dist.
2570d1f9 Add flushing to allow instrumentation of hfst-lookup
5a1307dd Replace rules now should support Flag diacritics.
571878b1 Fixed lenient composition (it used to be unimplemented).
209e9694 Move HfstStrings2FstTokenizer from tools to libhfst directory.
9105ea61 Use the actual transition data object when copying transitions in conversion.
7b06fa59 Fix a warning about comparison of signed and unsigned value.
4d67d870 Throw StateIndexOutOfBounds exception in get_final_weight, if state does not exist.
4c4f56cf Add examples used in documentation to the tests.
8a372988 Test examples and fix them, if needed.
12825fe6 Add missing documentation and functions.
b2aa4b12 Add a function that takes tokenized input and returns a transducer.
b3da97ac Define new functions for creating transducers from string, string list and dictionary.
c6ebab5e Fix some issues in the new python interface.
4be6a9dc Add more documentation and tests to the new python interface.
3b312a31 Add aliases for different substitution functions.
ca7834be Modify substitution functions to be more pythonic.
21c7d016 Add more stuff to the new python interface.
5c949955 Fix output of hfst-xfst's command 'help'.
fb4d2ef5 Fix small bugs in hfst-xfst.
21f9fbb9 Modify also determinization function so that negative weights are handled correctly.
9b42c700 Added more documentation.
833f6c6b Added more documentation.
ceb8de6c fix NUL flush, bug #240
5ed671d0 Add function __iter__() to libhfst.HfstBasicTransducer.
323778ad Modified libhfst files.
6bc7b3fe Create directory for new python API.
d929a126 Update Doxygen HFST version number.
39b7bd72 Update README and Doxygen documentation.
a7efde80 Add some tests with negative weights for contain, merge and parallel rule operators.
8bda8790 Modify minimization of weighted transducers so that negative weights are handled correctly.
e7f3b911 Always include 'generate-cc-files.sh' script in dist package.
be9d1ad2 Do not allow omitting the question mark on either or both sides of colon in regular expressions. Xerox has also removed this feature from their regexp syntax. Interpret expressions such as [foo : bar], [foo: bar] and [foo :bar] as [foo:bar]. Expressions of type [foo:], [:bar] and [:] will yield an error.
98c84237 Test if pmatch-tests.sh.* files exist before removing them to avoid 'rm: cannot remove:' messages.
b8b1039b Remove foma license warning. The current license of foma has no compatibility issues.
13553bdc Make some changes for the python interface. Move HfstFile wrapper class to the swig interface file. Make HfstStates and HfstTransitions typedefs public. Add a destructor for HfstTransition. Update version number to 3.8.3.
c64206bf Handle a frequently encountered exception
c48b828d Reverted changes done for class HfstFile in revision 4431.
a8cf4ea4 Removed HfstFile wrapper class and functions using it. A better place for them is in the swig directory.
65b418c7 Added destructor for class HfstException. It needs to be defined for SWIG.
99652176 tests for --weight-classes 1 and ordering by weights in proc
3e74d14c make tests pass again (only analysis order changed here)
a882fefa use std::vector as ProcResult to get output sorted by weight-class
a0bcb700 fix --weight-classes, bug #308
bd9ac6b0 Now the script generate-cc-files.sh creates also lexer files.
c37b24c6 Add $(EXEEXT) for Windows
2fa67a4b More Foma fixes for cross-building and clang
aa58fe3c Small modifications to foma code: struct and variable had the same name.
f0b6c857 Updated to foma version 0.9.18alpha.
d566985d GNU address removal
638efc5e Fixed some memory leaks in hfst-compare reported in bug #167.
4a7ea0c2 Adding the renamed files.
28182f15 Trying to rename README files...
cdd1ad39 Updated scripts.
6906e45b Now all files in this directory are listed in README
ab78a6b1 Added a README file for the scripts directory.
23746658 Fix help message
445eeb10 Fix containment, implement some additional wanted operations
892fdd8e Ready for release 3.8.3.
4eab0f4d Updated windows scripts.
9d0aba07 Now a warning message gets printed if stack contains transducers whose types differ (can happen if lookup-optimize is called before pushing another transducer onto stack).
f7a8ce86 Added a windows README file for the package that contains eight command line tools.
e4977d4d Added a Windows README file for hfst-xfst.
9e2ef69a Added tests for windows. Made small fixes to hfst-proc and hfst-twolc for windows.
3b3a3970 Replaced alternative tokens for logical operators with the standard ones.
65aa9494 Checking for existence of config.h and getopt.h.
f2f7fc3c Updated scripts.
aa1f4dff Added make scripts for twolc.
e33968b7 Added twolc to windows compilation scripts.
c9c02b9b Updated windows scripts.
91a19454 Fixing again reading and writing to/from console on windows.
3fd62f31 Using hex values for unicode characters when compiling with msvc to get rid of warnings (and possible segfaults).
023e5333 Terminator needs reading, and yes there are partial reads
8575c6d7 std::string all the way
4ba02e98 Added missing HAVE_CONFIG_H ifdef. Added globals-common.h header as some compilers complain about missing extern variables. Now creating variable size array 'next_u8' dynamically with operator 'new'.
0a364708 Renamed help_message.cc into xfst_help_message.cc and added a header file xfst_help_message.h. Now XfstCompiler includes only the header file.
e25e08d0 Fixed a typo and added explicit braces to an else statement.
0061f6c1 Now option --pipe-mode takes an optional argument { input, output, both }, 'both' being the default.
62f26f0f Install hfstdll.h
d09ea9c9 Now hfst-lookup and hfst-optimized-lookup read from console and print to console by default on windows.
09eaafb6 Commented out one debug print...
cb24cf29 More improvements to utf-8 handling on windows.
e29bdce5 Small modifications for better utf-8 support on windows.
75a25d45 Small modifications for some command line tools for windows.
b1415604 Added tool compilation scripts for windows. Argument handling on windows is also supported for some tools needed for hfst-xfst testing.
8b73295b Fixed a bug in argument handling.
e516f167 Edited help messages and warnings of hfst-xfst and hfst-lexc on windows about character encodings.
9d31b76d Moved getopt implementation to a separate file.
57f9a81e Added an implementation for long options for windows.
bb564a47 Fixed some issues noticed during msvc compilation. Renamed function 'min' to 'min_' to avoid collision with a macro with the same name. Changed arguments of comparison function given to qsort to (const void *, const void *). Added support for commandline arguments of type '--option=argument' to hfst-xfst for windows. Also changed hfst-xfst's option --print-to-console (default false) into --no-console (default false) on windows.
34d1c54c Updated windows script files.
b46445dd Deleted getopt_long cc and h files. Processing command line arguments manually in hfst-xfst.cc on windows.
88c44eb8 Added some ifdef _MSC_VER directives.
eed630e6 Applied patch for musl libc compilation.
f3ffa447 Fixed some bugs noticed when compiling on windows.
00291014 Blank line between outputs in --finnpos mode
5c3f801e Updated windows scripts. Added ifdefs for msvc compilation.
1734803c Updated swig scripts for windows.
1169774b Add FinnPos mode and reorganise
d9c60381 Updated and added scripts for compiling python bindings on windows.
5a001660 Fix harmonization issue and speed up the common case of only on top-level matcher
4d8a4a67 Documented the tsv file format, added some linebreaks to make the helpt text easier to read.
28711352 Always skip comment lines. Added verbose output for TSV file reweighting.
3d2e4db6 Fixed bug #293 by initialising the line variable before using it.
bfd227b3 Whitespace only.
d8501809 Added missing file 'hfstdll.h'.
ed20de25 Edited and added scripts for windows compilation.
25974b18 Added HFSTDLL's for some functions for windows compilation.
9fd87e05 Added option --arcs-only to hfst-reweight. Now weights of all arcs and end states are modified unless --arcs-only or --end-states-only is used. Also fixed '--end-state-only' as '--end-states-only'.
e8c1be4a Added some explicit type conversions.
2a773926 Changed the short form of the xerox mode from -x to -X (upper case), to be more consistent with the other options, and to match the test on line 195.
3ba83d91 Now it is checked if foma back-end and zlib are available before hfst-lexc-wrapper is generated. This also affects the way HfstTransducer::read_lexc_ptr is implemented for FOMA_TYPE: native HFST lexc compiler is used if lexc wrapper is not generated.
8557d614 Updated hfst version number in the python setup file. Fixed the argument type of LexcCompiler::setVerbosity from bool to unsigned int.
4a315651 Changed again some 'not' operators into \!.
e1c0779d Added a missing // in the beginning of a comment line noticed during windows compilation.
f4ecd940 Edited windows batch scripts and added a new one for creating python bindings.
590d56cc Changed declaration 'class yy_buffer_state' into 'struct yy_buffer_state'.
1f83336c Fixed declared return type from 'void' into 'int' for functions hlexclex_destroy and pmatcherror.
cfe7852a Xerox twolc test support in hfst-pair-test
48dcdd73 Tentatively fixed missing paths problem in compose-intersect
2f75bbc7 Moved definition of member 'static const HfstState START' of classes ComposeIntersectFst and ComposeIntersectRulePair from header to cc file.
d4bdfb38 Add [].t() syntax for delimiting tagged (and context-checked!) regions
5eb00695 Added new scripts for testing native windows compilation.
982836a2 Added missing return values for some functions. Replaced ciso646 keywords for logical operators with standard ones.
ad4273d5 Added make scripts for windows.
e94301ef Updated script.
7fccd3a8 Removed unnecessary HfstUtf8.h and HfstStrings2FstTokenizer.h inclusions from libhfst/src/ files. Also moved HfstStrings2FstTokenizer into hfst namespace.
d726d57c Now handling all stdbool definitions in foma headers.
7137e4d3 Now HAVE_CONFIG_H is checked before including config.h. Removed inclusion of zlib.h that had no effect.
b1ba306d Changed ifdef WINDOWS into _MSC_VER. Now zlib.h is included only when requested with -DZLIB.
a16942dd Using dynamic arrays instead of static ones to keep cl.exe happy. Also added definition for 'ssize_t' on windows.
72034133 Replaced 'and' with '&&'.
beb57e3f Replaced 'and' with '&&'. Also made a variable defining the size of an array const.
9b26e064 Replaced ciso646 aliases and,or,not with &&,||,\!. Also changed sizeof(variable) into sizeof(variable_type) in ol transducer header because cl.exe complained about that for some reason.
237842ac Fixed a bug where operator '==' was used instead of '=' when setting the value of last element in a char array.
dc51b2e1 Added missing files.
83d7d32d Added scripts and headers for windows compilation.
33409ade Changed operators {not,and,or} into {\!,&&,||} because cl.exe complains about them on windows.
d3779cd6 Added a script for windows compilation
1d327ca0 Added support for compiling openfst and foma back-ends with cl.exe on windows.
27b69942 Include zero in signed and unsigned integers
06517c71 Correction to handling trailing nonmatching material in locate_mode
f03b1626 Forgot to commit these files too...
da83b108 Added xfsm support for some command line tools.
3097ddfe Now HfstTransducer functions throw a FunctionNotImplementedExeption if xfsm implementation is not available.
40080541 Added more xfsm implementations.
818ad511 Added more implementations for XFSM_TYPE for HfstTransducer functions.
664573a7 Add ignoring
7b7d13fa Correct parsing order of PARALLEL_RULES
40066a17 Added support for composition for xfsm transducers.
3c11736d Added a variable to control if (xfsm) transducers that are already minimal are still minimized for profiling purposes.
877c46f2 Fixed some bugs in escaping special symbols in prolog format. Added support for xfsm format in some command line tools.
3cf0029c Fixed an error noticed when -Werror=format-security was enabled during compilation.
d0ed6328 Now xfsm conversion works in hfst-fst2fst and between prolog and binary format in hfst-fst2txt and hfst-txt2fst.
f6885672 Added XFSM initializer class.
c8039643 Added function minimize to xfsm interface
4c4ca0b1 Now hfst-compose-intersect does NOT harmonize transducers by default, as harmonizing produces unexpected results with hyperminimized transducers. Option --harmonize turns on harmonization. This is a partial fix to bug #288.
2e99ef0f Added a cross-build Windows patch given in bug report #289.
4cd63b07 Added a missing #if HAVE_XFSM statement.
e000b0cf Fixed some bugs in xfsm stream handling.
a478fb20 Added more functions for writing and reading xfsm transducers.
fcc4c5d2 Tentatively added input and output stream functions for xfsm transducer.
091ca1d1 Modified and added scripts for creating static binaries.
8d6dd330 Fast compose intersect available using option -f
1915dd1f Reorganized xfsm interface.
ed7f4489 Hopefully speed up compose_intersect by doing l .o. (l.2 .o. r) instead of l .o. r directly.
4b965b76 Updated script.
cc3632a0 Disable hfst-xfst's auto-complet with rl_insert instead of rl_abort, which is not found on Mac.
c41e726e Added a script to generate static binaries.
3a957724 Now xfsm conversion swaps the state numbers to make a more readable state numbering.
b2348d72 Now special symbols should also work in xfsm conversion.
b3af023b Xfsm conversion functions almost work, escaped symbols still need some attention.
b15f184f Conversion functions between xfsm and HfstBasicTransducer formats implemented, alphabets and special symbols still need some work.
5f791069 Autoindent entire file after removing tab characters to appease commit script
68ffd8eb Handle single-char ascii symbols shadowing multichar ones
b1bdd276 Modified hfst to support xfsm format.
05da3ffc Fix cg quote interpolation
38273d0c Clarify help string
b3450b4e Add segmenting and cg mode; improve handling of special cases and weights
5766bbf1 Tentatively modified hfst to handle xfsm implementation type.
db747e7c Preserve nonmatching sequances in locate_mode and parse the appropriately downstream
b7156e23 Tentatively added a skeleton implementation for xfsm library that can be included with configure option --with-xfsm (default is no). No linking to the library is performed, this is just a test version.
74270814 Added a preprocessing directive that defines hfst-twols's Alphabet as TwolCAlphabet if HAVE_XFSM is on. This will prevent variable collisions.
f5794586 Changed hfst-txt2fst's read_prolog into read_prolog_format to avoid conflict with xfsm library function.
424e9f21 Named tokens 'UPPER' and 'LOWER' in pmatch and xre parsers into 'XRE_UPPER', 'XRE_LOWER', 'PMATCH_UPPER' and 'PMATCH_LOWER' because the xfsm library (that might be added to hfst at some point) has an enumerator with the same name in C namespace.
cc5e2fbc Named foma's write_prolog into foma_write_prolog because the xfsm library (that might be added to hfst at some point) has a function with the same name in C namespace.
b7b9a5b5 Took away unused token 'LOWER' from xfst parser.
7701e12e Added an option --do-not-harmonize to hfst-compose-intersect.
ae06ceab Now hfst-compose-intersect harmonizes the rule transducers with the lexicon.
3bfd0e88 Added option --beam to hfst-strings2fst.
b96dd3b9 More accurately name set_locate_mode as set_extract_tags mode (locate vs. match is determined by function call)
47ef7f00 Fix bug in compiling UNKNOWNs on the left side of pair separators
b97a28cb Fix bug affecting locate-mode where ?-matches were being seen as IDENTITIES in input.
ec66bdd1 Added option --beam also to hfst-lookup. It is mostly untested for this tool.
00578471 Tentatively added option --beam to hfst-optimized-lookup.
e236dd2a Fixed a typo in warning message about missing symbols in input tapes of rule transducers ('output' changed to 'input').
43d99480 In locate mode, zero-length hits could cause infinite loops. Commit fixes it.
f5c14676 Now hfst-lookup checks it it is possible for an input to go through a transducer before calling is_infinitely_ambiguous. Should fix bug #278.
b4e46558 Now reserved symbols are detected in composition if xerox-composition is ON. This will prevent symbol collisions by throwing an error message, giving at least a temporary solution to issues with reserved symbols.
d2ef4729 Fixed substitution operator in regexp, now flag diacritics are allowed both as substituting and subtituted symbols. Should fix bug #284.
92d2bc0f Added better support for special symbols in hfst-xfst's substitute command.
7cc650e9 Added two tests for hfst-xfst's substitute command. They are skipped until substitute can also handle them.
1ff2f400 Fixed a bug in composition when xerox-composition is ON. Flag diacritics @SOMEFLAG@ were earlier transformed into $ and back into @SOMEFLAG@, but symbols of form $...$ are already reserved for lexc. Now diacritics are escaped as %SOMEFLAG% during composition. This creates a new set of reserved symbols which maybe needs more consideration though... Also added debug prints for LexcCompiler.
ef9407ed Removed xml checks from configure, since they cause unlinked xml references in hfst-edit-metadata on some platforms.
791aeb46 Fixed flag diacritic recognition, now also flags of form e.g. '@D.FOO@' are accepted and an empty string is returned as their value. Also added full support for flag handling in hfst-xfst.
94b723e6 Added some conversion functions between StringPairVectors and StringVectors. Added checks for flag diacritics in hfst-xfst non-optimized lookup.
6df280cf A small fix to hfst-summarize option handling.
479630d7 Ready for release 3.8.2.
d5c85458 Added option --print-symbol-pair-statistics(=N) to hfst-summarize.
adff4fc4 Small fixes to compile-replace. Also restarting the char counter every time a new xre parsing is started. Removed the xml2 dependecy in configure.
b8fd9814 Omit unnecessary step in stringification
adb2d917 Another slight speed improvement
b8564f46 Some double free insurance related to previous commit
786a502e Some more pmatch runtime speedup (around 5-10% in most cases)
6c383b40 Remove commented-out lines (same ones I was intending in the previous commit)
8c8be0be Remove extraenous index table fitting test
c150b351 Further improvements to conversion to optimized lookup format
ebb5d632 Large speedup in conversion to optimized-lookup format
5eaf6790 Added test cases for compile-replace.
80059f5a Added test cases for merge operation.
e8c709f4 Fixed an error in hfst-fst2strings --print-separator where two consecutive lines of -- were printed between non-empty transducers in some cases.
9a307605 Now minimizing the merging automaton before merge operation so that epsilons do not cut a succesfull merge path. Also allowing epsilon-to-regexp-marker transitions in the merge filter.
5d89abab Improvements to compile-replace function, now it should work for input and output sides of a transducer.
372d184d Revert bungled change to precedence order
06fa073a Now xre compiler of function merge does not increment the char counter, making it possible to have many merge operators inside one regex.
bc322864 Added a constructor XreCompiler(XreConstructorArguments & args) to facilitate passing xre variables to merge function which needs them in its internal xre compiler.
de47594c Now using internal starptr variables in functions hfst::xre::compile and hfst::xre::compile_first instead of global hfst::xre::startptr. This should fix the strange memory errors which occurred when calling merge operation inside a regular expression.
e7c15a33 Now merge operation filters out non-optimal paths.
e1532810 Tentatively added a function 'tokenize_and_align_flag_diacritics' to HfstTokenizer.
fe5cc6d6 Runtime speed improvements Prereserve table vectors, eliminate special_symbols map
a62fe686 Profiling support with Counter() and --profile & a bunch of smaller changes
09703ef4 Allowing 1-to-n composition of automata in archives. Fixes bug (or feature request) #277.
035edcd2 Now passing verbosity to LexcCompiler as an unsigned integer via setVerbosity(uint). Also made small fixes to warning prints in lexc compiler.
5fb6ba72 Added tests for one-sided flag diacritics for hfst-lexc.
602ec805 Now lexc compiler warns about one-sided flag diacritics in verbose mode.
fa95239e Added a tokenizing function that warns about symbol pairs, if needed.
4ad8eb44 Added test case for previous lexc commit (sublexicon defined more than once treated as an error).
45826225 Now multiple definitions of the same lexicon in lexc are treated as an error unless LexcCompiler::setAllowMultipleLexiconDefinitions(true) is called first.
95f58f45 Now lexc parser updates the error status hlexcnerrs when hlexcerror is called. In case of warnings, the error status is nor updated.
946bd68a Speed up list arc processing by replacing some maps and sets with vectors
373721dc Changed list to set in merge operation. Removed commented code.
52fe5647 Added option --encode-weights to hfst-lexc.
038ebd2e Reinstate undefined symbols as valid tokens as per documentation
212f1771 Add defined lists
25e7a028 Add Sigma()
491d7719 Added Lst() and support for list arcs in runtime
fe95b0ab Sync precedence rules
482bad08 Switch order of precedence of concatenation and other binary operations
93d28878 Added Lit()
a5114b5a Modified the function merge, it now takes as an argument a map of list symbols.
7ef05096 Fixed a bug in xre parser, now definitions and unknowns can be used together in expressions such as 'regex [def:?] ;'.
ed742e98 Now the epsilon symbol is not added as a multichar symbol to hfst-lookup tokenizer if it is the empty string. Should fix bug #275.
0fb0fdbb Allow \U00NNNNNN syntax for code points in utf-8 but above U+FFFF
2db80622 Allow \UNNNN as well as \uNNNN
4954a409 Allow \uNNNN in range notation and make some fixes to utf-8 handling
7784091d Added utf-8 character range expressions
f2674b24 An untested implementation of the merge operation added to hfst-xfst.
565d9bb7 Syntax-level completion of functions (arg placement still not completely free)
9bb6843d Fixed a small bug in hfst tool tester.
83312f73 Ready for release 3.8.1.
bcae2c3d Fixed std::cout into &std::cout in stream pointer comparison.
246258f0 Now using definitions USE_TR1_UNORDERED_(MAP|SET) when defining what unordered maps and sets to use.
c7c5ee8b Fixed a typo tr2 -> tr1.
97a3e595 Unordered maps and sets are used from std namespace if -std=gnu++11 is requested.
a5a62cbd Forgot to update version number in swig bindings.
a5783237 Various improvements and additions to function syntax, particularly empty args and string-args
5fb9c8db Added functions for merge operation in HfstTransitionGraph.
1df7d58a Modified weight handling in HfstTransitionGraph::intersect.
1e5bbfc3 When the alphabet is constructed from a symbol table, set identity to NO_SYM this was supposed to always happen anyway but didn't matter until recently
417bc818 Modified intersection algorithms in HfstTransitionGraph.
67f0a92c Added functions to be used in xerox's merge operation.
05651a46 Small fixes to list definitions in hfst-xfst.
2e1b79da Added an implementation for compile-replace in hfst-xfst, it still needs lot of testing.
244d6bfd Yet some more functions added to compile-replace.
e18a645d Added more functions for compile-replace.
39800a19 Tentatively added functions in HfstTransitionGraph to be used in compile-replace.
7b5981f4 Added function is_well_formed_for_compile_replace to be used in compile-replace command.
c0995960 Forgot to comment out debugging prints in tests.
f695bb45 Added brackets around member calls 'Interval.end' and 'Interval.begin' to avoid them getting confused with std::end() and std::begin() templates in C++11.
b0e0fe58 Rolled back earlier revision in interval-set.h
d223f728 Made small modifications for better c++11/c++0x support.
51ff0da2 Fix bug where identity wasn't being set to NO_SYMBOL when absent from alphabet
c2ee5876 Refuse to enter flag loops more than once fixes bug #250 bug - arguably this could be applied to epsilon loops too, pending discussion
15f25c64 Fixed a typo in Makefile.
1faecef4 Now hfst-xfst gives a warning (or exits) if a binary command tries to access a stack with less than 2 transducers.
a7fe5d6f Now hfst-xfst exits if a command tries to access an empty stack if quit-on-fail is ON and hfst-xfst is not in interactive mode.
25255dec Comment explaining usage of hfst-fst2tesseract.xfst.
ed8f0404 Added script for converting morphological analyzers to Tesseract word models.
61b43f5e Improvements to loop finding
4d3bab0d Further corrections to loop detection slowdown back to ~10x but may be improved from here
38bb11f4 Fix some cases of overdetecting infinite ambiguity, there's still some left
f99de3f4 Forgot to keep adding the repeated states in the loop detection phase
f327e834 Only try to catch infinite ambiguity at epsilon arcs This is the big speed win and presumably correct.
aa0928dc This order of comparison is a bit faster since sizes never differ
90b43870 Speed up is_lookup_infinitely_ambiguous() somewhat
3edea32a Try to avoid using negative indexes for arrays
1eedad1d Optimised lookup tests
a1eb3e1c Runtime handling of identity and unknown
dd8a9838 Forgot to remove one thing in the last commit
e1db16fa Take out our own harmonization hacks now that they're unneeded
6b0a8888 Don't use delimiters when they're not necessary, also don't insert everything to RTNs anymore and provide the is_special() function the previous commit required
e6a35c34 Treat special pmatch symbols like flag diacritics for harmonization, also after harmonization add all symbols, including flags, to the alphabets
40feabba Fix bug where delimiters were shadowing the named transducers' names also remove extraenous parsing path
7432ed41 Updated pmatch functionality tests.
6a89ad8b Add string literal syntax for standalone %-escaped chars (they used to be considered symbols which now have to be defined or cause an error)
18b6039d Use minimization guards to keep multiple negative contexts separate in disjunctions
817ec290 Fixed some more spelling errors noticed by lintian.
dcb9e44c Fixed spelling errors found by lintian.
727436d2 Added again hfst-train-tagger man page which is no more a symlink.
ee795957 Added missing man pages.
3e650c83 Now hfst-reweight-tagger --help returns EXIT_SUCCESS before trying to access uninitialized values.
9665cdc6 Warn about shadowing definitions
86089753 Alternate syntaxes regex for Define TOP and .#. for #
1874455b Revamp LABEL parsing and introduce curly literal pairs
7b2c44e4 Updated and added man pages.
cbb4a40f Moved option checking after possible returning from program so that option --help will not generate error messages.
7f699486 Require backslash character to be escaped as \\ in curly literals
832d7b30 Fix bug in unescaping function
6918fd0b Ready for release 3.8.0.
9336e7a8 Added variable 'lexc-rename-flags' to hfst-xfst.
73c2a334 Desperately fiddle with the way the minus operation expands things
ee6d6fa6 Add lambda-like anonymous definitions for controlling subexpression boundaries
9ef36769 Changed the flag handling behavior of hfst-xfst and hfst-lexc. Now both tools by default use Xerox's way when composing, i.e. flag diacritics match unknown and identity symbols. This can be controlled with variable 'xerox-composition' (the default is ON) in hfst-xfst and with option '--xerox-composition={ON,OFF}' (the default is also ON) in hfst-lexc. hfst-regexp2fst also has the option '--xerox-composition' which by default is OFF, as it was earlier.
6a572102 Fixed a typo in hfst-regexp2fst option handling.
a64b247c Now one-sided flag diacritics are allowed in composition when flag-is-epsilon is used.
1dfbb23e Added option --log10 for 10-based logarithmic weights in hfst-strings2fst.
0bcf18f1 Now an error is thrown if flags are not twosided in composition when xerox composition is used.
a959c4b9 Added exception class FlagDiacriticsAreNotIdentitesException.
6d8c1ab5 Add pmatch functionality test suite
d5d441eb Added option -X flag-is-epsilon to hfst-regexp2fst.
33b17d77 Fix another symbol-leaking issue
91d5ce0e Allow nested logical operations on contexts
d4b01d24 We need to avoid symbol pollution for more than just special symbols (this doesn't completely resolve pollution issues, just some urgent ones)
89149b06 Don't forget to pop the rtn stack when there's nothing matched
6a3552fb Minimize after adding delimiters, not before
fdf46b1c Revert bracket-bounding behaviour
fecbf548 Use brackets for extra delimiters to control tag and context boundaries more
51382e21 add_delimiters was happening in the wrong place since recent syntax changes
84678128 Fixed a too strict assertion.
7ac7e496 Added a return value for a case that should never happen to make scan-build happy.
09b12609 Reverted back to the buggy behaviour of function 'getMarkerNumber', since some HfstXeroxRules tests will fail if it works correctly...
c7d12d46 Fixed an error in function 'getMarkerNumber' where istringstream was not properly initialized and returned random values. Also added a print method for class Rule.
7b7f0fb5 Fixed a typo in filename in EXTRA_DIST.
caad8cb2 Now alphabets are copied when encoding and decoding flags in composition. Fixes bug #267.
ad47d327 Oops, forgot about restoring scope state afer entry arcs
f325949e Add a scope facility for local context boundaries
591c5b55 Fixed most scan-build issues other than dead store, memory leak and errors from foma back-end.
d2075681 Added implementation for twosided flag-diacritics in hfst-xfst.
d3c0506f Now harmonize-flags and flag-is-epsilon are by default OFF in hfst-xfst. Also added a new, mostly untested variable xerox-composition that matches flags with unknowns and identities in composition (default is OFF). All flag-is-epsilon functionalitites are moved under HfstTransducer.
ce348d20 Purge commented-out lines
37524618 Major syntactic changes and additions (AND & OR) to bring in line with Karttunen's documentation - amazingly doesn't break backwards compatibility (brobably / mostly)! Function syntax still lags and is incomplete.
79f7b428 Added missing return value to a function in openfst back-end.
636451e0 Reverted back to version 3992.
e0442eea Instead of NULL, return a new HfstTransducer in a condition that should never occur. This is for some compilers.
0d7cdee7 Fixed errors reported in bug #265. Also tentatively added a switch --xfst-harmonization to hfst-xfst that treats flags as ordinary symbols in composition.
08462a75 Resolved another rpmlint issue, hopefully the right way (A HfstTransitionGraph method returning nothing when it seemed to be meant to be returning *this)
68ed253e Resolved some issues and nonissues revealeled by rpmlint
0e2f6766 Now xre parser compiles expressions of type foo:bar^{0,N} correctly.
be1e33dc A better way see if we have a better location than before
ba31a12a It seems to be generally faster to defer minimization at [] boundaries
17b833ba Made 'harmonize-flags == ON' the default for hfst-xfst. Also hfst-xfst's xre parser now harmonizes flags according to variable 'harmonize-flags'.
90cff758 Fixed some issues noticed on c+11 and reported in bug #258: a space between literal and identifier, missing cstdlib header and ostringstream conversions.
9da1a0fc hfst-fst2strings now gives an error message if option --nbest or --random is used with transducers in optimized lookup format.
ddea524b Apparently older gcc's libstdc++ require operator< to be const
f6df4d28 Added locate mode on the library side; support for multimatch, weight access and various internal changes
084d1a64 Now all command line tools should give an error message and exit with >0 if they cannot process input in hfst optimized lookup format.
19595377 Now hfst-invert prints an error message if given a transducer in optimized lookup format. This should be fixed in all command line tools.
5a5a3e46 Function set_expand_definitions added to the python interface.
d89d63bd Updated HFST version number.
26bad712 Updated HFST version number.
78a7e30e Added option --renameFlags to hfst-lexc for testing purposes.
9e7a3ff7 Added option --encode-weights to hfst-determinize.
c9ae91fa Now hfst-lexc has options --withFlags and --minimizeFlags and hfst-xfst variables lexc-with-flags and lexc-minimize-flags to control if hyperminimization is used when parsing lexc files. Flag minimization can be the default behaviour when it has been properly tested.
57cc2df7 The Xerox output should have an empty line between each cohort.
01928da5 Now flag minimization in should work in LexcCompiler. Commented the changes out until all expected results from lexc tests are changed accordingly.
38b37247 Apparently failing to read a stream now throws HfstException so catch that instead
c29a94e5 Tentatively added a piece of code that filters out multiple flags in lexc result. Currently commented out.
8b0216c9 A hopefully temporary tool called proc2 for simple tokenization, to become part of something else or be renamed in the future
ae425925 Some fixes to python wrapping of pmatch and showing original input in pmatch
cee590b7 Make locate() return a simple data structure rather than a string
3ad4e8ef Convenience loader function for pmatch
0cc41abc Fix outdated prototype of printConnectedness() and add pmatch prototypes
125ed8a9 Added option --Werror to hfst-lexc that treats warnings as errors.
2e28445e Added tests for the fixed bug #243 that will pass now.
24e7b465 Now keyword LEXICON is allowed without a preceding newline in lexc parser. Should fix bug #243.
a50dd1a1 Added a variable encode_epsilons to function priority_union whose default value (true) should fix bug #254. Function lenient composition still calls priority_union with a false value of encode_epsilons.
94dc75b4 Added support for cross-products such as {foo}:[bar] and foo:[bar] in regexp parser. Tests still needed.
d42799e2 .NOTPARALLEL for now
85a69f9d Fix serious bug with epsilons in contexts - they were incorrectly moving the input tape.
b04de378 with pipes
941baa99 A super cool progress bar functionality \o/
30cf7bd8 Fix context-related bug
13c19049 Readability refactoring
7f22c97e Comment fixes and some more deadwood elimination
cf24a3bd Remove more unneeded stuff
8d58ce1e Remove some obsolete & commented out bits
dd779dd8 Locatefy option
93d00f87 Don't make noise about almost always harmless result truncation
9abaa725 Large refactor of almost everything (mainly to better serve tokenization apps w/tape synchronisation)
8f8d8460 Undo previous commit, was confused by polluted automake cache
df23ceda It seems that newer toolchains don't like c++ classes being defined over multiple .cc files, or something..
1958ec8f Don't count flag diacritics when locatefying
516bf5ca Remove obsolete bogus -> rule from when @-> had problems
e5d943a4 Handle missing @bin files more nicely
04cf7dd9 Autoconvert @bin arguments if necessary
9ce5490c Remove obsolete dependency readme that hogs the svn trunk page
c427c3d4 Add constructor option to make diacritic strings blank or not; make pmatch consider them blank
2c2e664e Added file 'test' to EXTRA_DIST..
b1f01a2b Fixed hfst-twolc test Makefile.am
211935ee Disabled outdated unit tests.
6b74d9fe Tentatively implemented variable 'flag-is-epsilon' in hfst-xfst.
531acff8 Fixed issue with left arrow conflicts for contexts with impossible word boundaries.
2881c9e8 Also added tests for regexps with different continuation lexicons in hfst-lexc.
489474ad Now regexps with different continuation lexicons are allowed in hfst-lexc. Should fix bug #247
21db219e Fixed list center rules and conflict resolutions.
6942188a Tests for list centers.
cd1193bc Tests for list centers.
6e909fde Exit on parse error Once we've printed a parsing error, exit rather that print intimidating hfst exception messages
efdd01bd Pattern-locating mode
90612986 Some changes to approach to avoiding stack overflow should fix problems with very long input lines
2cbb071d Add harmonization
1c1007c2 Add extract tags -option (this commit also includes a secret commented-out facility for profiling activity on the input tape by drawing a funky ascii histogram)
79c629f9 Fix incorrect check for whether tokenization needs to respect a long symbol
76336868 Allow epsilon-symbol pairs
01eb2112 Don't print huge data dumps when parsing fails
22732372 Revert back to the original TropicalWeightTransducer.cc and forget splitting it..
d7755c32 Moved minimize and push functionalities from TropicalWeightTransducer.cc to separate files. This will make compiling TropicalWeightTransducer faster and prevent 'File too big' errors on Windows.
1c749307 While scanning for possible first symbols, don't accidentally go beyond the end of the input tape
c50a51fd Ready for release 3.7.1.
f17327e3 Now removing CR characters from output of hfst-xfst so that tests will pass on windows too.
f3740222 Now an error message is thrown in HFST when reading a native SFST transducer that uses the empty symbol.
6596539c Added option --encode-weights (-E) to hfst-minimize.
e7d0c0e5 Added documentation to parallel replace and removed unnecessary marker insertions.
1d1251ab Modified conpilation of weighted parallel rules by inserting markers to keep same mappings with different weights separate. Also added a StringPair variant of insert_symbols_to_alphabet to HfstTransducer and HfstTransitionGraph. Added a test case for hfst-xfst. Updated NSIS installer.
60b45cb4 Tentatively added a function for substituting weights with markers.
fd297514 Remove spurious obsolete comment
2925fb70 Bounds check accesses to ascii_symbols when reading input too (ascii_symbols used to have a NO_SYMBOL entry for any char, but now without a check some accesses were resulting in bogus keys)
d23d42fa Add -D_XOPEN_SOURCE=500 in order to prevent -std=c99 from preventing strdup() prototype being visible
9f7311e3 Correct bounds check for ascii_symbols alterations
988e259a Now checking that ascii_symbols vector index is not out of bounds when removing shadowing symbols. Should fix bug #235.
7b89f3f5 Replace rules bug fix
6bbf8bd4 Parallel rules bug solved
4ad528d8 Ready for release 3.7.0.
6fd62726 Added missing files to Makefile.
180ebd2b Added option --encode-weights also to hfst-regexp2fst.
04837d3f Added option --encode-weights (default false) to hfst-compose-intersect.
ec55eaba replace rules - weights are removed from calculating contexts (they were unnecessary there)
34fcf5fa Removed commented code.
d469a2be Modified TropicalWeightTransducer::are_equivalent so that determinization is carried out in a way similar to 'intersect' and 'subtract'.
aee15150 Updated configure.ac
b4fbd3f5 Reverted hfst-ospell changes.
edb752e2 Tentatively added hfst-ospell under tools/src, it is only enabled if --enable-ospell is requested.
911b8a0f Use precision 1 in weighted tests.
e45e2fca Reinstate special ascii tokenization while avoiding shadowing longer symbols
5ad406c1 Disable ascii-lookup when tokenizing
e45fc596 Set encode_weights off in tests to make sure that they work also if it is on.
6f1b3de5 Fixed a bug in parsing regexps such as '?:{foo}'.
95fcfa38 Make pair notations a bit more consistent; enable RIGHT_ARROW
8b3f3128 Now incompatible replace types in parallel rules yield an error.
65884a77 Added variable 'att-epsilon' to hfst-xfst. Also added more weighted parallel replace tests.
4f932312 Now epsilons, identities and unknowns are printed correctly.
4e6f6644 Fixed a bug in handling unknowns in hfst-xfst's 'look up'.
b3ac9731 Added more weighted parallel replace tests.
2ca06d96 Allow runtime contexts anywhere to facilitate some efficiency strategies
b628ccf9 Added rest of the replace tests.
d28fef52 More tests...
9cd39cb2 Added more hfst-xfst replace rule tests.
77d98df9 Fixed weights in markup rules
f62a2ce5 Added more tests.
42c8ecb4 Added more replace tests.
9ad639df Added replace tests, not all of them are yet performed.
095ed972 Add tentatively new replace tests.
8fb7cc9f Added tests for containment operators.
e49b2026 A couple of small fixes to containment operator.
06ee31a0 Sorting arcs before subtraction in TropicalWeightTransducer::subtract.
e2e71f51 Removed unnecessary determinization from TropicalWeightTransducer's intersection function. Added function HfstTransitionGraph::get_transition_pairs. Made some small modifications to xre parsers's containment operator.
e46dcf91 Fix definition of curly literals
0fa18962 More PAIR_SEPARATOR fluff (which should be completely refactored)
8cb51e1f Build input scanner without using a stack guard; also refactor where runtime behaviour is stored
8fb36bbe Added tentatively new functions for implementing containment in xre parser. Removed determinization from intersection.
07215346 Changed containment operators so that they also work with transducers that are not automata, i.e. '$.[a:b]'.
c08ec08d Added documentation to TropicalWeightTransducer::intersect.
8455c8a4 A simple stack guard for first-input -collection (only, for now)
c0f23cfa Now all containment operators work, at least with unweighted containment transducers.
562e77f6 Fixed a bug in '$.' operator in xre parser.
fcb43f5a Use a vector in the prescanner to make it slightly faster
7790325c Add class-identifier to friend declarations; required by some compilers
8623961f Remove extraneous debugging message
be473f1c A very conservative and (for now) computed-at-runtime input prescanner
fb8361cd Added a test for empty context in replace rule.
413741c1 Resorting to global variable 'encode_weights' instead of passing it around as a variable. Allowing empty contexts in replace rules, i.e. 'a -> b || _ '. Fixed a minor printing issue in hfst-xfst.
9c6f3ab7 It's perhaps a little clearer if is_like_epsilon(epsilon) is false
3c4f82e4 Fix bug where conversion from ol to basic transducers was missing @I.foo@ arcs in some situations
834a1296 Fix bug where weights on the insertion arc were being ignored
0e7e0a65 Log insertions when verbose
3cd12813 Added variables 'max-weight' and 'encode-labels' to hfst-xfst.
981f3cba Tentatively added functions to control label and weight encoding in weighted minimization. Also added max_weight parameter to lookup_fd of HfstBasicTransducer.
c2653459 Fixed weighting in containment operators.
ae39ecb8 Edited xfst tests.
9765b5b7 Fixed a bug in parsing multiple parallel replace rules.
fd0614c9 Fixed a bug in the contains one '$.' operator. Improvements to the xfst parser.
700c746a Updated regexp2fst help message.
53585c53 Use "" for special parsing and symbols, {} for tokenizable strings; also allow "define" alongside "Define" Future note: consider adding another syntax for "totally bare" symbols with no special parsing
56d86bc3 Only print prompt in pipe mode if --verbose is used.
5440fa9d Added yet more tests for weighted rules.
f0abf044 Made xre parser ignore weights in rule contexts. A warning is given in non-silent mode.
b481854e Added more tests for weighted replace rules.
49fd9ba8 Added optional variants of replace right tests.
17712e1b Forgot to apply weighted_rules -> weighted_replace_right renaming scheme to test.sh.
a0566e42 Renamed weighted_rules_* to weighted_replace_right_*.
54e62ac3 Addded more weighted rules tests.
92b67e71 Added function HfstTransducer::is_infinitely_ambiguous() and command 'test infinitely-ambiguous' in hfst-xfst.
6633e61e Copy the before-seen states so different paths to the same state don't conflict Reinstates a skipped test
72955a85 Made hfst-summarize print whether transducer is infinitely ambiguous in verbose mode.
946d8412 Fixed a couple of typos causing errors when calling is_infinitely_ambiguous..
ec80b660 Tentatively added function is_infinitely_ambiguous() to HfstTransitionGraph. Added variables 'lookup-cutoff', 'print-words-cutoff' and 'precision' to hfst-xfst. Added test cases for weighted rules. test_transducer_functions now returns a skip value until hfst optimized lookup transducer's function is_infinitely_ambiguous is fixed.
9cf83ecb A slight followup to the previous commit
b26cd8f4 Fix stupid bug in tricky control flow part of loop checking
2b0e3b53 Bugfix: distinguish between transitions and transition indexes when determining state equivalence when looking for epsilon loops for a given input.
674b883a Remove extraneous debugging message
9dba5e02 Fix bug where infinite ambiguity checking was polluting the flag diacritic state
daad246b Reorder things so that we should usually run into max_results before running into the stack depth
6bafb973 Respect result limits and implement a recursion depth guard so instead of blowing out the stack we discard overlong results and give a warning message
14318482 Rip out empty main test function so there's only one in the ol compilation unit
bec743dc Add string version of is_lookup_infinitely_ambiguous() to HfstTransducer api too
4845339a Added variant is_lookup_infinitely_ambiguous(const std::string &) in HfstTransitionGraph which is used in hfst-xfst to check infinitely ambiguous cases in lookup. Also added a variable precision in hfst-xfst to control the number of decimals shown.
391c4c0c Add a facility for determining whether a given input produces an infinitely ambiguous lookup
7a6fa366 Replace rules - weight now works on both sides, with context.
e6b4af87 Insert special symbols into ?-containing transducers' alphabets right away, fixing some incorrect harmonization behaviour
42dd77c4 Updated weight management in replace rules
e903072c Finally ready for release 3.6.1.
9db3dddb Better conflict warnings
27a66277 Now it is possible to save and load a stack of transducers in hfst optimized lookup format.
ae845016 Give a more informative error message if tranducer is not in hfst optimized lookup format in hfst-proc.
eb61bef8 Added hand-written man page for hfst-foma.
49e1c5f8 Updated man pages.
415d8f0e Now removing symbolic links to command line tools when performing 'make uninstall'.
eda041b5 A new candidate for release 3.6.1.
6c6081ad Updated installation check.
e515805d Edited help messages of lexc and lookup tools.
f2398e20 Made creating and checking flag results less verbose.
dbb84a02 Changed 'return' to 'exit' in shell script.
9dc775e2 Updated version numbers in NSIS and swig files. Also made lexc wrapper disabled by default.
d76dfcd2 Fixed a typo.
d2efde2d Corrected syntax error, typos.
2f1296e0 Ready for release 3.6.1.
672a2b44 Edited help message of hfst-lookup.
86ca3827 hfst-lookup now also handles a transducer archive.
0a73c82d Added more options to print (random-)(upper-|lower-)words commands in hfst-xfst.
9ae6bfb7 Fixed a bug in HfstTransitionGraph::remove_transition and added a test for it.
1f124935 Fixed some reserved words in lexc parser. Added weights for command 'look up' in hfst-xfst.
d98f5c26 Scaled down weights in replace rules.
1d953fce Lex the left paren in function calls along with the function name to make a distinction between a symbol followed by an optionalized transducer and a function call
c17b5be0 Added a lexc test that contains reserved words in lowercase. The test does not compile in the same way as in foma.
247d3654 Lexc weights fixed
8e97a2f7 small lexc bugfix, tests enabled
e4eb4c0a Allow underscore in symbols (couldn't discover a malignant side effect)
2d0993d6 A lot more expressiveness involving DefFun()
b251769e Fixed #227 hfst-lexc and multichars with zeros
54bb1e41 Made hfst-xfst use foma's lexc. hfst-lexc-wrapper is enabled by default.
06c35e9c Allow tsv input for reweights
c2006bdc Forgot to update -version-info in Makefile.am.
cc52d4b4 Ready for release 3.6.0.
c4bab385 Edited morphology tests so that they agree with changes in finnish-analyze.sh.
cfbc0285 Now checking in configure if tr1::unordered:map/set or std::unordered_map/set is available.
a04fcb37 Replace rule longest match bug fix
5c6d4ae2 Now hfst-xfst gives better error messages if a file cannot be opened. Removed h-files from SOURCES in tagger Makefile because they confuse some compilers on Mac.
792a4afa A preliminary version of DefFun(args...  and Map(fun, @txt...)
6ed03ab6 Now apply up/down are done using HfstBasicTransducer if the transducer is not in optimized lookup format, which is relatively fast.
9b8c2987 Removed unnecessary comments.
b610b50b Added prompt '>' to hfst-lookup, it can be disabled with option --pipe-mode. Added command 'convert net' to hfst-xfst to convert between optimized lookup and standard formats. Now also supporting 'apply up' for optimized lookup transducers.
0f645692 Removed redundant declaration of SfstTransducer::harmonize that caused an error on Mac OS.
501ef0f2 Added command 'load FILENAME'. Fixes bug #225. Also allowing a ';' after 'define VARNAME'.
d8bb71d6 Now command 'define foo' is recognized. Fixes bug #224.
dbcdbe95 Set permissions of flag check scripts. Fixed some typos in other tests and in flag diacritic handling (== instead of = in if condition).
52eba535 Added hfst-xfst to the environment, help and version checks.
8d654500 Made minor fixes to tests so that they work correctly on windows and mac too.
0b5d7090 Now HfstInputStream constructor throws an exception if file cannot be opened. This exception is also handled in the regexp parser.
980ed3e2 Added experimental options --enable-no-tools and --with-openfst-log to configure.
15c1ea27 Removed unnecessary warnings from hfst-lexc.
f327366f Fixed a typo in configure file, extension should be 1, not 0.
90a6953e Now creating a variable length array dynamically to avoid compilation errors.
cb847d72 Renamed hfst-lexc2fst to hfst-lexc and hfst-lexc to hfst-lexc-wrapper.
43f5303c Now att format is printed directly to the console in windows environment.
11b0dcbd Updated reading and printing functionalities based on tests in MinGW environment.
a57a9b4a A more reasonable parse order for weights and special functions; fixes various bugs concerning weight placement that should be legal
a0ee0c2c Added missing <stack> include to HfstTransitionGrap.h.
f8fd3d12 new lexc2fst, with tests
29d8cbea Fixed a bug in harmonization, now flag diacritics are not harmonized.
9f5f620b Transition to weighted, prioritizing pmatch; recompilation of rulesets required at least for now
b62bfd35 Removed a line from HfstTransitionGraph::substitute that printed debugging info to stderr..
36699e08 Added function HfstTransitionGraph & HfstTransitionGraph::(std::map<HfstSymbol, HfstTransitionGraph> &, bool harmonize)
08d4506b Now standard streams are defined as binary in command line tools when compiling on windows.
03ed6960 Updated batch, NSIS and swig scripts for windows.
5b43fe8a Support \uXXXX with a codepoint -> utf-8 generator - this should probably be shared somewhere
e325a0b2 Know about uppercase ẞ
a207a509 Do not warn user about using foma type in hfst-lexc if they have explicitely requested it with --format foma.
9c19e8f3 Now hfst-lexc obeys argument --format. Fixes bug #213. Also updated NSIS scripts.
84b99fdd Updated NSIS script. Added hfst-twolc and hfst-train-tagger bat files for windows compilation.
882e7f4b Distribution package includes flex/bison-generated cc and hh files. Make check runs tests only for those command line tools that are enabled.
66f323bc Allowing tagger tools in mingw.
caa6f26d Fixed a bug in xfst_getline, also added freeing the result from it.
32c6b57e Fixed a memory error in hfst::xre::parse_quoted noticed during make check on MinGW.
8d395978 Fixed some bugs noticed during MinGW compilation: added typedef 'off_t' and compiler definition '__NO_MINGW_LFS' when mings compilation is enabled, and escaped percent sign in fprintf as double percent. Also added tagger tools and hfst-xfst to NSIS installer script.
158e8ef5 Allow plain input-output -replacement with tokenized quoted literals, like "this":"that"
87a7276c Some refactoring & warn about unused definitions
8ad5fe2b Ready for release 3.5.1.
83d0dc0a Applied lexc patch (bug #208).
4a95514b New lexc, but still incomplete
b8907ea6 Removed directory fsmbook-examples as fsmbook-tests replaces it.
5c0a7570 Now using foma's lexc parser in hfst-xfst by default. Also using verbosity of hfst-xfst and tagger tools in lexc parsing. Also added an option --enable-all-tools in configure.
1213bb05 Use flex's newline count as good-enough parsing error indicator
3ab3a5d2 Actually fixing that bug this time
6c720e2e Fix typo-bug in previous commit
8a3e52cb Much more verbose printing of time spent on harmonization and conversion & a speedup the the process of collecting a unified alphabet
209a12d7 Small fixes to installation check.
b2730bcb Removed redundant command 'rm TMP' from pmatch test. Also removed verbose 'skipping some test cases' print from proc test, returning a skip value 77 is enough.
d9027876 Fixed a typo in test script.
d7b4050e Changed temporary transducer name in test-hfst-xfst so that it does not conflict with any other transducers used in the tests.
ce5564ed Skipping pmatch_blanks.txt when creating binary transducers from att files. Also made check-tools.sh more verbose. Fixed a typo in make-dist.sh.
3ebd13c2 Modified the call to 'nodeindexing' in SFST::Transducer::store as it seems to leave its argument unchanged sometimes. Noticed this effect when testing hfst-xfst and getting a segfault when trying to use a stored transducer.
c41db2e0 Added disjunct_as_tries as a public function of TropicalWeightTransducer.
7c3bd28a Remove ineffectual AM_LDFLAGS line
cb52daae Try setting a toplevel LDFLAGS to standardise to the standards-compliant libstdc++
a1d9ab84 A pmatch functionality test
8fe00aea --flatten and make option passing a bit neater
994689fb Add DefIns and make LTR longest the default RIGHTARROW
0b2a9f64 Added argument 'extra_link_args' to Extension constructor. By default it is empty, but when making the debian package it can be replaced by linker flags that make it use /usr/lib/ instead of /usr/local/lib/.
432da483 Print transducer and function definitions as string pairs in the way they are defined by the user.
7fc05b86 Made print paths observe whether obey-flags is set ON or OFF.
0346f8cc Now transducer names are saved.
efded068 getline() doesn't return a newline-terminated string if one can't be found; in this case we mustn't erase the final character.
000a25d5 Updated ChangeLog, maybe after a couple of bug fixes ready to make the actual release..
efb77b7f Added variable 'print-foma-sigma' to the xfst parser.
f279104d Add a pmatch2fst functionality test with a regression test
6ebc08b8 Fix bug wrt unused networks were getting erased from their container during iteration, causing other networks to be skipped
b911b357 A preliminary "true" negative context facility
39d4db68 Transducer definitions are expanded without calling substitute in xre parser, the new way is faster and less prone to symbol/alphabet errors. Implemented 'print net NET_NAME' in xfst parser. Allowing variable names that can be parsed as numbers in xfst parser.
7078c8f7 Ready for release 3.5.0.
426e74b4 Added missing files in fsmbook-tests to distribution package. Edited READLINE conditional in hfst-commandline and lexc-readline-ui.
90242241 Edited help messages of hfst-fst2txt, hfst-txt2fst and hfst-xfst. Added missing files to distribution package.
64a536ae Renamed hfst-xfst2fst to hfst-xfst and hfst-xfst to hfst-foma.
3edfd469 Added variable harmonize-flags to xfst parser. A warning message is printed when composing transducers with flag diacritics if verbose in ON and harmonize-flags is OFF.
ce71a8ab Added missing variable 'print-weights' to the XfstCompiler constructor with impl argument. Added support for prolog format to hfst-fst2txt and a simple prolog test. Fixed a typo in configure.ac. Removed unnecessary commented text from HfstTransitionGraph.h.
3ba662fd Added support for weights when reading of writing in prolog format.
6094847d Edited xre warning messages.
a8a38ce8 Added a boolean argument 'increment' to function 'set_final_weights' which defines whether existing weights are summed with the new weight or overwritten, the default being overwritten for backwards compatibility. Also added warning messages about hfst-special symbols of form '@_.*_@' to the xre parser that are printed in verbose mode. Also added support for negative weights in xre parser.
fe047407 Fix compilation time output at hundredth-of-a-second precision
998f5536 Make --verbose output somewhat neater
78644014 Time individual compilations
dc0dd3e4 More fine-grained information in --verbose
b782c815 Added option --harmonize-flags to hfst-regex2fst. Also added a function 'set_harmonize_flags' to the xre parser.
763aa83a Using result from foma in DateParser test.
46ace60a Added angle bracket test for hfst-xfst2fst. Omitting it until we decide how symbols enclosed in alngle brackets should be handled.
5efe9b4a Fix some problems wrt ?-expansion and symbol tables in RTN:d networks that have to be aware of each other. Also use dark, forbidden symbol magicks to prevent special markers in RTNs from being clobbered in this process.
9493341a Make harmonize() a public member function (this changes the interface)!
e713b450 Now lexc parser can handle epsilon '0' correctly.
9cc8ad72 Fixed a bug in xfst parser, now semicolon is allowed after 'read lexc <filename>' command and verbosity of the xfst parser is also used when parsing a lexc file. Also added expected results for fsmbook Esperanto tests.
9b7a8d40 Updated fsmbook tests so that they use ready-compiled xfst/foma results given in prolog format. Also fixed a typo in DateParser hfst script. Also checking that fsmbook-tests and xfst2fst are both enabled in configuration.
e22ffb2a Fixed a bug in quoted literals, now ["foo""bar"] is parsed as in xfst: ["foo" "bar"]. Also using foma instead of xfst in fsmbook-tests when compiling Lingala because different interpretations of symbols enclosed in square brackets.
f23f9c89 Now xre parser handles "[.#." and ".#.]", interpreting them as "[ .#." and ".#. ]".
ef252ece Removed unnecessary zlib dependencies.
15ac3162 Updated cross product and apply boundary mark in replace rules.
04ce2d33 Changed include of hfst-scanner.h to hfst-scanner.hh.
fcab62fe Changed include of xfst-parser.h to xfst-parser.hh
bc5b1c45 Changed PmatchUtilityTransducers variable utils into a pointer and added a function get_utils() to avoid variable initialization issues.
c67ec6cf Reverted INTERNAL_EPSILON etc to internal_epsilon..
5501ab91 Pop the local state stack too when exiting rtn calls
966fd9f3 Changed internal_epsilon etc to INTERNAL_EPSILON also in pamtch utilities.
22d93faa Use xfst instead of my local copy of it..
90db8f46 Now using macro INTERNAL_EPSILON etc instead of string internal_epsilon to avoid variable initialization issues (noticed on cygwin).
8f8f9a0a Now fsmbook and hfst-xfst2fst tests should work correctly whether they are enabled or not.
cf28d276 Added condition WANT_XFST2FST to tools/src/Makefile.am. If it's true, directory parsers is included.
43592791 Make parser even more messy by having bodyless tagging contexts at the toplevel - for now
91a171c6 Added missing file.
7ba73be8 Now using directory fsmbook-tests instead of the old fsmbook-examples. Renamed h2hh.sh script to generate-cc-files.sh.
bdb4ca72 Roll back broken empty-bodied contexts for now
9932c0c0 Removed the HH_EXTENSION_USED conditional from configure and makefiles since it didn't work. Now assuming that the flex/bison-generated header files have the extension hh. If this is not the case, the script scripts/h2hh.sh should be run between configure and make.
ba5562b5 Added a script for cygwin compilation.
f373aed0 Edited ifdef conditionals based on warnings when compiling on cygwin. Also added a script for handling errors in flex/bison header file naming.
5c3fe43f HfstTransducer::cross_product should expand ?:? transitions to ?:?|?, added this but inside comments until a bug in xerox rule compilation is fixed.
9bd6dcdf Fixed a couple of typos.
2ab61eef Added more tests to the xfst parser.
9b4ec9b8 Finally full support for quoted literals in xre parser.
8693d535 Tentatively adding quoted literals in xre parsing, this time step by step..
6d2fcd95 Trying again to make quoted literals work better in xre parser..
5f464357 Reverted recent changes to xre parser, noticed bugs in hfst-regexp2fst..
26cc9354 Symbols that contain quoted literals supported in xre parser.
65f17987 Better support for quoted literals in xre parser.
afd146aa Now allowing '0' in symbol names.
6c084004 Give verbose information about networks while parsing
c9200d44 Merge from devbranch
eba09c92 Added cstdio header, it isn't included in cygwin.
15d6ecb5 Dynamically allocate input tapes in case of overlong input - in case of insufficient memory, truncate overlong lines.
ea5ad013 Don't attempt to find regular arcs with symbols newly discovered from input.
4641571a Added option --eliminate-flags to hfst-compare. Commented debug info printing from LexcCompiler that polluted its output. Edited command line tool tests accordingly.
f4dd70ed Fixed a bug in calling FomaOutputStream.
068bd6ae Added variable print-weight (default NO).
11a3786c Added examples to hfst-regexp2fst help message. Now xre parser accepts also semicolon and one or more spaces as weight separator. Moved flag handling from FomaTransducer to HfstTransducer as it does not depend on foma backend.
60896407 Restore tape state after processing rtns; this should fix leftmost greediness bugs wrt rtns. Also: do some checking and print errors in case of broken tags instead of hard crashing.
db770ece Now make check passes also when some of the back-ends is disabled.
ef59b18a Output symbol tables were missing when writing transducers in native OpenFst binary format. This is now fixed.
626d0238 Cosmetic changes to Xerox rules
ab840e4a Removed HH_EXTENSION_USED conditional from configure as it still fails on some platforms. Added function HfstTransducer::get_first_input_symbols and using it in hfst-summarize.
aa76b2c4 Ready for release 3.4.6.
38c638ec Now empty comments starting with one or more exclamation marks are also parsed right.
ea40e35a Fixed errors noticed during hfst-xfst2fst tests.
adfea76e One more special symbol spelling to rectify...
3fb58126 Fixed a bug in fsmbook tests.
1935b06c Forgot to make context markers non-special too!
770efa48 Add a couple of useful combining accents
1578436b Use (faster) string disjunction instead of full disjunction for word lists
9fe28000 Make only TOP, networks with EndTag() and networks with Ins() get the special @PMATCH_ENTRY@ etc. delimiters. This should make various operations on networks that don't use these features work.
c201bf69 Added missing file.
3b4ae5ed Now allowing several prolog networks in one file, separated by empty lines.
e26899d4 Added tests to fsmbook test set.
786baf36 Now Palindroms test works.
7c635b59 Fsmbook test Esperanto is now divided in several script files.
c165109c Now using prolog format to convert between transducer formats in fsmbook tests. Also added the option --prolog to hfst-txt2fst and fixed a small bug in prolog parsing in HfstTransitionGraph::read_in_prolog_format.
864120d8 Remove generated files in hfst scripts.
728b7699 Correct internal endtag format in utils too
84bccf9e Endtags look like @PMATCH_ENDTAG_foo@, not @PMATCH_ENDTAGfoo@
c14098e0 Fixed a couple of bugs.
5976146b Correct typo that was suppressing the functioning of CATENATE_N
88455f17 Rename internal symbols not to match is_special_symbol() to let them pass through eg. composition operations
88ae522d Added tentatively directory fsmbook-tests which will replace the directory fsmbook-examples.
7f5ff0d8 Added the variant 'write att outfile isymfile osymfile' of 'write att' to the xfst parser.
e4f29855 A hopefully working test for HH_EXTENSION_USED - unfortunately it appears that automake lines not in a make target are included regardless
7d7a225a Eliminate else branch
ee4e69d9 A better value for the HH_EXTENSION_USED variable?
dc42b095 Eliminate references to .h targets when HH_EXTENSION_USED is unset, which was perhaps bothering automakes >=1.12
6a89896e A (more) correct understanding of IDENTITY.
510955c8 Switched the *.h - *.hh dependency around, from *.hh: *.h, to *.h: *.hh. This seems to have fixed the *.hh build issues on MacOSX. I have no idea whether this will work or break (on) other systems. Please test!
0a1a72e9 Some systems need to be told that .ll files depend on .hh files so things get generated in the right order
dc1d37e7 Added '@pl' operator to the xre parser. Testing it with hfst-xfst.
9a7a1324 Fixed the h/hh issue once again, now trying to copy instead of linking. Also added assertions when converting from sfst to basic transducer format.
3b37bde4 Removed redundant filenames, added whitespace.
de4fcdec Changing the BUILT_SOURCES to list the cc files instead of the hh files seems to fix the header file issues for twolc.
0d56127f Some support for identity transitions
9bc40e0d Implemented '@txt' and '@stxt' operators in xre parser. Also testing them through the xfst parser.
2f7e9a7f Make the alphabet know about IDENTITY
9fc2d09d Uncomprehendingly fill in the blanks for the LEXC_BUILT target
03cf0744 When composing incompatible types, instead of throwing an exception, convert the latter type to the former for composition (as per hfst-compose's message).
91db5fb4 --force symlink to .h/.hh target
eeaf345d Add combining accents to the end of the capitalizing and lowercase-ifying utility transducers. Also more aggressive minimize utility transducers.
bd1cd1bc Character constants like \xNN are two hexadecimal numbers, not decimal
df7f071c Allow escaped quotes on quoted literals
479cff73 Implemented '@re' operation in xre parser. Added test cases for it in the regexp parser tests.
b3a3c2a3 Fix bug with input tape initialization in certain circumstances.
a0c966a9 Changed rm -f to ln -s in the Makefiles when handling the h/hh rules.
327a83d9 ln instead of mv in .h/.hh rule
1fc8b143 Check for .h/.hh yacc convention and mv accordingly
faf38475 Made some modifications to the xfst print format, removed comments from xfst compiler.
8b608d43 Added missing files.
394603ec Added substitute defined tests to xfst parser.
b38ac6e7 Added -f to mv header files
e4412921 Fixed a bug in converting an empty foma transducer into an HfstBasicTransducer.
3cc8efa5 Added more substitute tests for xfst parser and fixed some issues with alphabet in substitution functions.
fd05f5f4 Fixes bug with Ins()erted transducers named "name"
1b4b69c1 Add toplevel delimiters (in addition to ones associated with EndTags). Fixes a bug when endtags aren't used but insertions are.
e91d5600 ToLower, ToUpper and OptCap made (more) compatible with Karttunen's pmatch
5613090a Fixed bug in LexcCompiler.cc, one that was causing wrong compilation of noflags in cases when there were only identities in lexicon
9178debb Forgot to remove old test file.
c7579503 Divided xfst substitute tests to several files. Fixed a bug in printing unknowns in xfst format.
479c64b1 Now allowing more variable names.
4307079f Now substituting symbol with symbols(s) removes the original symbol from the alphabet.
cf3fd281 Fixed a bug in xre substitution: disabled harmonization that caused unknown symbols to be expanded to temporary variables.
9e1c4f95 Minor changes to lexc2fst and xfst rules
9b459635 Not substituting a symbol pair with an empty set of symbols removes the transitions equivalent to the symbol pair.
c222f0fe Added tests for substitute functionalities.
d9ba4ceb Added 'substitute defined' to the xfst parser.
afa6747f Now configure checks the version of automake and renames the header files generated by flex, if needed.
de7b521c Now 'make clean' removes also files generated by flex/bison.
7b6831da capitalisation modes: last on wins
0fee2aa8 update
a8383d4b Oops, forgot to exclude xfst testing from make check..
5c2ce1b4 Added documentation.
8c9f1578 Now most of the keywords are interpreted as such unless they are at the beginning of the line.
d488c082 Added substitute functionalities to xfst parser.
cec7f36d Added function 'remove_from_alphabet' to HfstTransitionGraph.
e4ae2b8c Added command 'assert' to xfst parser.
f163569e Now all commands are given by tab if readline is used.
961ff5d0 Added tests for writing and reading prolog files in hfst-xfst2fst. Also fixed a small typo in writing in prolog format.
a24c10b3 Removed implicit readline requirements from hfst-xfst.cc and foma.c. Also edited prolog functions in HfstTransitionGraph.h.
eab0624c Now reading and writing in prolog format works. Also addded some exceptions for prolog handling.
1765035e Added prolog printing functions to HfstTransitionGraph.
2cd5695e more header renaming (see commit for r3377)
2c12e0cb change header file names -- this will break compatibility with automake < 1.12 see http://lists.gnu.org/archive/html/automake/2012-09/msg00017.html
055ff21c changes to names of generated header files --- pending testing
77f52805 Now apropos and help messages work in xfst parser.
d5fc1bd5 bump required versions of automake and libtool
9feac555 Ready for release 3.4.5.
c9b1cc51 Solved bug in replace rules with term complement in context, fsmbook tests are now completed.
ea426a76 Edited help messages.
d6b3866a Small fixes to the xfst parser.
0286e7fd Additions to apropos and help commands.
21d49c11 Tentatively added enums and maps for xfst commands.
cdc7f858 Now using variables outstream_ etc instead of stdout etc.
1e431947 Tentatively added commands 'apropos' and 'help' to the xfst parser.
ed0fe8e1 Backport pretty dot files from that better branch
0fe1b1cb Added more tests to xfst parser.
8ccc79de Added new functions write_in_xfst_format(ostream/FILE*) to class HfstTransitionGraph. Using it now in the xfst parser when calling 'print net' instead of write_in_att_format.
df3ab645 Added more documentation to the xfst parser.
90d0462f Added documentation to xfst parser functions. Also fixed a bug in prompt handling when calling apply up/down.
987fe0aa Basic size indications in verbose mode
b66c98d1 Xfst parser: now apply up and down work also in pipemode.
dd5f2e11 Fixed some bugs in the xfst parser. Now readline/getline is used consistently in the command line tool and in the actual parser. Also the option quit-on-fail now works as it should.
cc65d1e4 ToLower() and ToUpper() were missing from the rule parser
5662445e Substituting symbol A with B on both levels now correctly removes A from the alphabet.
4316ee79 Occasionally minimize while collecting large @txt lists (this is a significant efficiency improvement for those cases)
de4c50db Fixed a couple of bugs in including back-end libraries noticed during cygwin compilation.
648dd3ec Tests for version and help checking as well as empty input now check only a set of listed tools instead of hfst-*.
41f0dffc Commented foma headers from hfst-file-to-mem.cc. Also fixed a bug in regex parsing in hfst-xfst2fst.
c77d4ab4 Let's use malloc until we figure out where xxmalloc lives (and why we want to use it in the first place)?
945c765d Error messages from xre parser are now handled through the function get_error_message instead of printing them directly to standard error stream. Also changed 'Transducer' to 'SFST::Transducer' in SfstTransducer tests to avoid ambiguousness.
9898075f Some header files stil used installed back-end libraries instead of those in directory 'back-ends'. That is now fixed.
d12c4dac Now prefix xre is used when generating xre parsers. This should fix bug #185.
19efda54 Now regex comments can contain semicolons. Partially fixes bug #177. There is still some ambiguousness in weighted mode, consider e.g. the input 'cat ; 3'.
66ae9058 Fixed a bug in function hfst_file_to_mem, when using stdin.
1db3b539 lex joiner's format changed to special symbols which do get harmonized
c4e55029 Moved function hfst_file_to_mem from hfst-xfst2fst.cc as a separate file.
43006eb8 Fixed comment parsing in xre parser, hfst-regexp2fst still needs some fixing so that semicolons can be used in comments.
3b8d5387 Added facility for inserting undefined transducers as long as they are defined later
78b697d1 stringify() can and should take a const parameter
4c74eab1 If no boundary markers were used, we need to advance the input tape accordingly to where the real input starts
a0171d92 Print newlines after printing match result
b6798c7c Check for nonzero length before trying to omit final newlines
f978c016 Omit final newlines from input
be48f320 When in newline mode, no need to print final line because it's already been printed
9ac5d42f Reworking of pmatch tool, added default of blank line as separator
1d9d2350 Tentatively added a function compile_first to the xre parser that allows compiling onely the first regex in a string.
459a5188 Now hfst-fst2strings obeys the -S flag. Also removing generated files after make check in xfst parser.
6856a762 Added 'print defined' to xfst parser. Also temporarily commented the bison > 2.6 request in configure.ac until bison on hfst gets updated...
ee6e8d03 Added a test for bison version 3.6+ - now it will stop if older versions are found. This seems to fix at least one of the build issues reported.
18a9ff8d Added tests for defining functions in xfst parser. Also fixed a bug in function definition.
acd3a94b Add diagnostic for rulesets that don't recognise anything
a62e3f5d Avoid compiling transducers that "recognise" the empty string
e196215c New insertion arc handling
83014492 Move insertion arc checking out of flag diacritic handling after all
692840f6 Collect and harmonize alphabets
0e895a7b Detect inserted transducers and only pass them and TOP on from parsing
5bcd4a20 Recognise insertion arcs as flag diacritics
59045141 Skipping regexp tests that do not work. Also added new tests.
62202a7d Changed regex.tab.c and regex.tab.h filenames to regex.c and regex.h because make distclean of some version of autotools removes tab.c files. Should fix bug #184.
d829e212 Catenate functions were giving wrong results, now fixed
0cf3e8d6 No flags part now seems to be working
4155dd96 Don't treat automake warnings as errors.
e568b99a Regexps that contained comments were skipped, now that is fixed. Fixes bug #182.
8590e327 Reverted back to earlier version.
c93a90cd Changed bison-generated h-files to hh-files. Recent versions of bison and automake are needed for compilation.
cbfe23e6 Now binary transducer operations don't modify their second argument. Fixes bug #180.
bcbc96ff Refactor ol conversion to allow giving a harmonization hint
1219953b Fixed error messages.
0bc38c16 Fixed error reporting in xre and xfst parsers.
23eb58dc Now reentrant parsers are used in all xre parsing.
7ab97de1 Fixed replace rules with empty languages
6f2b7080 Added missing file.
2fc94323 Added reentrant versions of all xre functionalitites.
e1ac0fd5 Tentatively added reentrant versions of the xre functionalitites.
7d683c40 Avoid extraneous copying and sorting
707fe424 Now comment lines are correctly parsed by hfst-regexp2fst. Also added error diagnosticprints to proc test.
8cb8ab27 Maybe fixed bison anf lfex header problem. Revert, if it doesn's work.
9b22f582 Added more function handling to xre parser.
31fcb4e5 Fix invert logic and remove debug print
758a647f Added more function handling mechanisms to xfst and xre parsers.
fb73e65f Now xfst parser recognizes function definitions.
9d699fb3 Make the starting position of the input tape be before BOUNDARY_MARKER. This is to enable referring to the boundary in expressions, not just contexts. It should gracefully always fail to match and be ignored in the absence of boundaries in expressions.
368efaba Refactor parsing; rip out tool-specific parts (and, for the time being, some probably unnecessary control options)
32676939 Debug printings and rearrange noflags part to first
59bb75df xre parser now recognizes function calls. xfst parser supports command 'source'.
185a9eaf Add some noflags to parsers
9aa9a7c5 Add support for selecting flagged and composed morphotaxis.
6c6f4f05 Command 'source' implemented in xfst parser.
65fc6de9 Added test case for bug https://sourceforge.net/p/hfst/bugs/177/ .
57132d83 Added symbol substitution to xfst parser.
f7bec384 Remove use of strndup to fix bug #175
d63cd8c3 Added more functions to the xfst parser.
c9048a3e Fix a bug caused by shifts in the input tape caused by entering left-hand contexts that weren't being undone.
49450950 Added more functions to the xfst parser.
16ef95de Removed WeightedString classes as they are replaced by HfstOneLevelPath etc datatypes. Also added new functions to the xfst parser.
9962f1ea Moved lookup_fd functionalities in hfst-lookup.cc to HfstTransitionGraph as member functions. Also moved HfstEpsilonHandler.cc and .h from tools/src/ to libhfst/src/. Removed is_flag_diacritic from hfst-lookup.cc and using FdOperation::is_diacritic instead.
41c6f4eb Removed extra code.
5dfb71b4 Moved EpsilonHandler from hfst-lookup.cc to HfstEpsilonHandler.h and .cc.
24c1781a Moved is_lookup_infinitely_ambiguous(HfstBasicTransducer ...) in hfst-lookup to member function of HfstTransitionGraph: is_lookup_infinitely_ambiguous(...).
3aee716b Added flag tests for hfst-fst2strings.
944e4779 Use ANOTHER_EPSILON for tokenising
b39c7049 Now option combination --obey-flags --random works in hfst-fst2strings.
6597da63 Insert epsilons to all lexicons
5f8d5e26 Maybe align end flags
66184c6e Now hfst-xfst warns about transducer type conversions. The state and arc information should also be correct.
2ab6f016 Added missing test file.
ff916bf5 Many changes to how utilities are handled, prepare to move all parsing away from the hfst-pmatch2fst tool
63793253 Another bug in flag elimination fixed. Now flags that are not eliminated are included in the filter so that they are not blocked.
2c0c5358 'load stack' didn't print state and arc info for all transducers, this is now fixed. Should fix bug #171.
37b640a7 Calling eliminate_flags for a transducer that has no flags to be filtered caused a segfault, this is now fixed.
ecb18bce Updated eliminate flag tests.
95c91cc9 Fixed a problem with default symbols and harmonization in compiling guessers.
72040dc4 Ready for release 3.4.4.
e0e99960 Now proc functionality test proceeds until compounds2.strings would be tested and instead returns a skip value 77.
eef4bc10 Now tests for tools that are disabled by default are only carried out if the tools are enabled...
e25f4099 Fixed a couple of bugs noticed when installing HFST on Hippu.
b7b567da hfst-xfst2fst tests are now skipped if --enable-xfst2fst is not specified. lexc-compiler-functionality-sh and proc-functionality.sh also return a skip value.
dacc7d16 Now foma and sfst backends are enabled by default in configure.
aea2aa0c Now hfst-lexc uses only foma. hfst-lexc2fst tests skipped as experimental. Flag tests in hfst-fst2strings are also skipped until flags are thoroughly tested.
f0a52e92 Fixed missing parameter in fprintf in warning message in hfst-compose-intersect. It was causing a segfault on some systems.
d374ccce At least unification flags are now handled correctly in hfst-fst2strings.
0f5f7152 Moved HfstLookupFlagDiacritics functionality from tools/src/ to libhfst/src/.
60e65332 Tentatively added HfstTransducer::extract_random_paths_fd. Flags still need to be processed right.
d55617cd Now hfst-lookup warns about infinitely ambiguous transducers. Fixes bug #168.
2c03ec76 Added missing files to Makefiles noticed during distcheck.
c608af26 Added warnings to xfst compiler when using longest-string(-size) with flags.
1fc3922a Tentatively added commands longest-string and longest-string-size to xfst parser.
34771a98 Added function HfstTransitionGraph::longest_string_size.
ec8a45d2 Tentatively added function HfstTransitionGraph::top_sort().
3c946a61 Added option --version to hfst-pmatch.
117677da Make sure minimization always gets done
e93a1ca3 Added more test to xfst parser. Now comments are handled correctly in xfst. Edited documentation in HfstTransducer.h.
c1e34470 Merge from pmatch dev branch, with a change related to read_lexc -> read_lexc_ptr.
bd3e893f Now xre parser can parse expression '{foo}:{bar}' correctly. Also added variable tests to xfst parser.
6a48024d Added a test for xfst's shortest-string. Also fixed a bug in including the epsilon string in HfstTransducer::extract_paths.
c0d6df42 Added new xfst tests, also added function HfstTransducer::is_automaton.
21b6858c Invert lexc flags
2b3399e0 Renamed again some testfiles in xfst..
6c55cb5f Edited xfst tests.
fbe891e1 Documented flag elimination function. Fixed some bugs in using transducer definitions.
d1a4df53 Added a new test for flag elimination.
7e73c8e0 Added tests for xfst parser's function eliminate flags.
4f1b6edf Wrote more flag elimination functionalitites.
3ef35f3c Added x: center rules and tests for them.
9bac4fc5 Restored long option name --resolve.
2ddf6e2f Fixed problem where definitions in pair expression were caught too late which resulted in weird syntax warnings.
dde58d74 Maybe use flags instead of joiners in legacy lexc compilation.
855907f1 Started implementing flag elimination function.
0417d5c8 Added new functions to xfst parser.
37f81173 Added new test for xfst parser.
f4117b79 Set property executable ON: test.sh.
4f0096e1 Again doing flag diacritic filtering in flag diacritic harmonization, since it turned out not to be a problem after all.
d7b530c4 insert_freely_missing_flags(...) no longer uses insert_freely(...) becauses successive invocations of insert_freely without intervening minimizations cause exponential state growth.
9cc367ac compose-call in hfst-compose now does flag filtering only if flag harmonization was done
96db17f5 Changes that permit new filtering of illegal flag paths hopefully wothout affecting performance.
1015c1e6 Added more tests for xfst.
26262519 Added more tests for xfst parser.
bb869de7 Now readline is disabled by default in xfst parser unless configure is run with --with-readline.
e9df9ddf Changed 'HFST_THROW' into 'HFST_THROW_MESSAGE' in HfstTransducer::eliminate_flag(s). Should fix bug #163.
64baca68 Renamed some of the functions in xfst parser.
b9fb1df9 Now readline history of 'inspect net' is ignored when returning from 'inspect net'.
4e8a8d65 Commented code.
2921e59d Now 'inspect net' should work for all cases.
b3bda03e Tentatively added 'inspect net' to xfst parser.
28fe0407 Edited XfstCompiler
281668ca Added new tests to xfst parser and implemented interactive lookup mode.
4d9d8014 Added more tests to xfst parser.
7461e46e Tentatively added functions HfstTransducer::eliminate_flag(s). Added function 'compose_net' to xfst parser.
f40c185f Updated to foma version 0.9.17.
c8225030 Added test for HfstTransducer::remove_from_alphabet.
03593c73 Changed bool static is_safe_conversion(...) to static bool is_safe_conversion(...) because syntax error.
b708ce11 Now regex parser substitution does not add substituting symbols to the alphabet of the transducer to be substituted. It is not needed anymore since there is a parameter 'harmonize' in HfstTransducer::substitute(StringPair&, HfstTransducer&).
ee00f6c9 Now HfstTransducer::n_best throws an HfstFatalException if OpenFst fails to allocate enough memory. Kind of fixes bug #159.
2dd7fe52 Now hfst-regexp2fst and hfst-xfst2fst both give a warning message if the user writes a replace rule that contains special symbols.
4f1842ff Added more symbol harmonization tests.
782cbb76 Commented code.
1e7be915 Now the regex parser parses [x:?] so that it includes the transition [x:x]. This also happens in xfst and foma.
2b628227 Now regex parser interprets ?:? in the same way as xfst or foma, i.e. as unknown-to-unknown OR identity-to-identity.
19fddc23 Implemented substitution with an empty set in regex parser.
5d3bcdc8 Fixed hfst-twolc.bug which inserted all diacritics into the first set that was declared.
acdf5e82 Now xre parser never performs harmonization in substitution or ignoring.
2a1e3d49 Fixed a bug in xre substitution.
5a033b82 Added a bool parameter to control harmonization in HfstTransducer::substitute(SymbolPair, HfstTransducer). Fixed xre parser so that it doesn't harmonize when performing substitution. Also changed call to substitute function in twolc but preserved the harmonization.
5282e25f Now binary operations never harmonize special symbols of form @_.*_@. To make xerox rules work, the marker symbols @_SOME_MARKER_@ had to be changed to @SOME_MARKER@ so that they get harmonized as they did earlier.
4312b8c9 Now insert_freely also takes a parameter that tells whether harmonization is carried out.
e979ac2e Now there is an option --do-not-harmonize for all binary command line tools.
72d8c605 Fixed some bugs in harmonization.
b7ad8b63 Fixed missing bool parameter problem.
5890a60f Added a flag -H to hfst-compare and hfst-regexp2fst that turns off unknown harmonization.
004dfbd7 Added a bool argument to binary transducer functions that controls whether harmonization is carried out. Also added harmonization tests.
12c31115 Now hfst-lexc writes all messages to stderr. Fixes bug #68.
cdf4d164 Now FomaTransducer::define_transducer(symbol1, symbol2) works also if symbol1 == symbol2 == identity. Fixes bug #110.
20dfb3fa SfstTransducer.h no longer uses generic word 'Transducer'. Fixes bug #130.
4aa59f63 Now hfst-sfstpl2fst finds included files in the same directory where its input file is located. Fixes bug #145.
aea2877e Now hfst-fst2txt prints an error message instead of segfaulting when given an input stream whose transducers do not have the same type. Fixes bug #115.
1cefd4e1 Implemented functions number_of_states and number_of_arcs.
b0813539 Now all xfst files are read in a buffer and parsed with XfstCompiler.parse_line.
0c088206 Now xfst parser can parse regexps that are on several lines.
d2f3f2ed Edited xfst parser.
9c5a6167 Added tests for hfst-xfst2fst.
721c6030 Added main program from XfstCompiler.cc to hfst-xfst2fst.cc.
21981c98 More command line options added to hfst-xfst2fst.
f036babf Parameter handling added to hfst-xfst2fst.
fea1693f Now hfst-xfst2fst works with readline functionalities if a temporary file is used for buffering commands.
e2fa01a0 Replace rules compilation - boundary marker added to begining/end of all contexts which don't containt it
f674390e Added more functions to the xfst interface.
51d468d9 Added more functions to the xfst interface.
d806ad2e Added variable undefinition to xfst parser.
21d58096 Now definition expansion can be controlled in XreCompiler through a function set_expand_definitions. If they are always expanded, lexc syntax is not parsed correctly.
48adbe90 Added definition expansion to the xre parser.
6ee87cc5 Made a couple of fixes to xfst parser.
bcc413bf Tentatively added tools/src/parsers as a part of the automake build.
87e7cf80 Updated morphology installation tests, now omorfi is included.
0f1d6b3c Updated version number in swig setup file.
dce9471a Now installation tests skip sfst or foma if they are not found.
9dd65776 New apertium proc space handling test from bug [#153].
5092648a Small fixes to swig and installation checks.
f28cddb5 Added documentation.
ad582a87 Made small fixes to swig tests.
daeca5d5 Made permutate-file-params test silent. Edited check-tools.sh so that it handles --prefix parameter right.
a34e9692 Updated hfst tool checks-
cb402e9d Fixed Lingala and MonishGuesserAnalyzer fsmbook tests, minor changes in regexp
bec768db Updated installation checks.
1b414e48 In substitute (symbol_pair, transducer) added harmonization with the symbol_par. Also, updated substitute in regexp
da8b67fd Ready for release 3.4.3.
ce48e9ed Prune alphabet added to regexp, also ? -> x rules fixed
310a1e10 Added a couple of fixes to swig tests.
9ecc837b Now tests pass on MinGW, file handling still has some problems.
4b8f2649 Disabled tagger tools if HFST is compiled on MinGW. They are also skipped in tool tests.
fbad2ce2 Added tests for hfst-xfst and hfst-twolc in installation tests.
a7a109a4 Added 'AUTOMAKE_OPTIONS=std-options' to foma Makefile so that foma wrapper gets installed right on Cygwin.
6fb4082a Renamed option --enable-windows as --enable-mingw.
64302471 Edited configure.ac so that setting executable mode on script files should finally work. Added missing windows conditional to foma Makefile.
3e499444 Prune alphabet changed for easier use in parser
04dbf510 Removed local versions of hfst-xfst as well as hfst-xfst tests.
4dc1af2f Setting executable mode on scripts.
fcd3abfd Fixed a typo in Makefile.am.
4451854c Fixed a bug in foma wrapper. Also added missing test file to EXTRA_DIST.
6ae0cb34 Set executable mode on scripts.
57f2152c Made a local version of hfst-xfst that is used in tests.
9afcfd15 Ignore generated files.
280ff8c9 Added a test for hfst-xfst. Modified permutate-file-params.sh so that it doesn't use curly braces which are not expanded correctly on all environments. Also modified hfst-compose-intersect.cc so that it doesn't use dollar signs when specifying how strings are printed as they don't work on Cygwin.
ae57f524 Now swig tests use libhfst.hfst_get_exception() instead of catching HFST exception classes. This should make the tests compatible with python 3 where catching exceptions that do not inherit from BaseException is not allowed.
6849b3e8 Disabled fsmbook tests from 'make check' by default (enable with --enable-fsmbook-tests) because they currently depend on foma command line tool.
c08fbcc6 Added function HfstTransducer::prune_alphabet which is also used in the tool hfst-prune-alphabet.
c23504bc Added filtering to harmonize_flag_diacritics. It will filter out superfluous flag diacritic paths, which are created during harmonization.
be0e3fd2 Prevented foma from harmonizing id- and unk-symbols with flag diacritics in insert_freely.
8cecacd3 Removed 'words.txt' from EXTRA_DIST as it is not found in svn repository and make check passes without it.
653b6083 Added a function 'std::string hfst_get_exception()' that can be used to get the name of the latest exception thrown by HFST library. This is useful if the version of Python used does not support catching exceptions that do not inherit from BaseException.
d4ceb699 Making sure that flag diacritics are added to the alphabets of both arguments in harmonize and harmonize_
9e56f9f8 Changed id- and unk-symbols harmonization so that flag diacritics are no longer harmonized.
2d801c62 Fixed a bug in flag diacritic harmonization yet again. This time actually using the parameter bool harmonize in freely_insert.
b2139f2d Fixed problem which resulted from changing the insert_freely-function in the library.
65f17f07 Re-fixed flag diacritic harmonization problem, where identity and unknown symbols were expanded by flag diacritics. To prevent expansion in this case, I added a bool parameter harmonize to the insert_freely library function.
220b78c2 Now configure gives warning on missing hfst-xfst correctly.
8499a59f Added a new tool, hfst-prune-alphabet. Also added a new bool parameter to HfstTransitionGraph::prune_alphabet.
0de2ffe8 Changed variable name 'infinity' to 'infinity_' to avoid compilation error on Cygwin.
58531f14 Added -D switch to enable reading from file in htwolcpre2 and htwolcpre3 (for debug purposes).
aa1f4457 Now tagger and twolc tools also pass version and help checks.
edc2f459 Added a missing <cstdio> header that caused an error on cygwin (FILE was not defined).
0e0d644a Now all back-end files are included in the distribution package.
626ae880 Fsm-book test FinnishProsody
40ae452b Fsm-book test Palindromes updated
157ee164 All back-end directories are now included in the distribution package.
bafa96d4 Added PlusOrMinus fsm-book test
5534b007 Updated README. Fixed hfst-info so that it passes version and help checks.
14b7427c Changed the order of paragraphs in README. Added a todo comment about XreCompiler.compiler conflicting with a built-in name 'compile' in python.
b4c8b7bb Updated README.
52eff40b Ready for release 3.4.2.
b38adb54 Now -lncurses is used in foma instead if -ltermcap is not found.
dd13b4c4 Fixed a bug in foma wrapping-
eaa693c4 Added regexp substitution
55d27503 regex parser - added left replace epenthesis rule
145ac045 Added note to remind people to also increment the HFST_LONGVERSION. This is important to ensure consistent behaviour between different types of version checks. The different types of version info returned by hfst-info --version should be consistent, the long version should just be a representational variant of the other values.
d2ff1397 Longversion.
9d02b3e3 Ready for release 3.4.1.
77414856 fsmbook examples change
d73849e6 Epenthesis rules fixed
557c7bfa Updated results of morphology tests.
251f7531 Now hfst_foma is called directly from where it is installed.
533a2d0e Bring the long version string in sync with the other version numbers.
95f87648 hfst-xfst now uses hfst_foma instead of installed foma.
70c60e94 Added test sentences for all morphologies.
eb0a1488 Renamed a duplicate symbol.
7bcb0837 A variable was shadowed by a another variable with the same name, renamed the previous.
2939cf2a Made the names of the tests scripts shorter.
4334be5e Tentatively added tests for installed HFST morphologies.
b7a55def Added lower priority union to regex parser
2eaa8bab Added foma script under back-ends/foma. It is now used by hfst-xfst. Fixes bug #147.
999b9c59 Fixed a couple of typos in swig test file.
5db5e5a7 Updated the installation check script.
e754a74a Tentatively added tests for installed swig bindings.
c6763af4 Now installation tests pass on Hippu.
1bb3ebc6 Updated tests.
faca4990 Now installation tests should pass. Also fixed a cople of minor bugs in test/tools.
7b55aac5 Added new tests.
ecffb8d3 Tentatively added a test for all installed HFST functionalities.
7844d8a5 Another warning-silencer flag that had to be removed to get hfst to build on MacOSX.
8dceaad4 Made hfst build on MacOSX again, by removing the -Wno-unused-result GCC flag. Sorry about the additional warnings, but I really need it to build.
3825fabd Suppressed warnings caused by back-end libraries.
b087ac13 Ignored generated files.
ac73af70 SWIG bindings should now work with both Python version 2 and 3. As FILE* conversions are no longer supported in Python 3, added more wrapper functions that use an HfstFile argument instead.
1a71ed7c Priority union bug fix
52a9da32 Fsmbook tests updated
b1517604 Removed old command line tool hfst-strip-header from tests.
d4e491ce Removed old command line tool hfst-preprocess-for optimized-lookup-format from tests.
f0fbd83c Removed old command line tool hfst-duplicate from tests.
921e328d Revert previous unintended change to unrelated files
bb4a0014 The header from r2888 that accidentally didn't get re-committed with the most recent bugfix in this directory..
5bcd4818 Fix typo-bug wrt tape movement
6c8fc3de Roll back previous changes pending bug fixes
7e449db9 Avoid reallocations in the input tape
55d96ea5 New scheme for input and output tapes to allow them to be arbitrarily large
86942e11 Handle the case where we want to take a default transition when it's the only transition
7a27e64c Changed all includes of type <backend-header.h> to "backend-header.h". Made small modifications to NSIS installer.
34355d96 Added more information to swig/README.
a29cb347 Ready for release 3.4.0.
049de832 Fixed some bugs noticed when making distcheck-
a66c16d1 Fixed a couple of bugs that occurred during installing a dist package..
809b5e7d Fixed a small bug in HfstTransducer::read_lexc, suppressed some warnings from openfst headers.
6daff3ef Updated python tests.
85104fb5 Added all python examples given in the doxygen documentation to tests.
d41d5b39 Added more tests to the Python interface and added throw statements to functions.
1113a6c8 Renamed hfst-duplicate as hfst-multiply.
2ab080eb Now tests remove extra files that they generate.
b7d3c8c5 Moved Python tests to a separate directory.
173566ea Python documentation is now in a separate directory.
48a0405a Now all functions in Python interface are documented.
14a3048b Added documentation on rule functions of the Python interface.
b6485631 Added new internal documentation to the Python interface.
0093e292 More documentation on the Python interface.
dcd281c8 More swig documentation...
2b97711a Added more documentation for class libhfst.HfstTransducer.
33c493e7 Updated swig documentation.
964d693a Added - as a comment - parts of my attempts at building all version info from only one definition. No success so far.
23b3cbdd Added missing directory 'openfst'.
61e0e192 Added a new constructor to HfstTransitionGraph, also added it to swig interface.
75891d88 Updated swig documentation.
3624f889 Updated python documentation.
43a33014 Added more documentation of the python interface.
71455635 Tentatively added doxygen documentation for Python interface.
f888bfc2 Now all back-ends (SFST, OpenFst and foma) are moved under HFST and statically linked to the HFST library.
5835516f Forgot to change the directory name to opefstwin in Makefile.am.
f617fb69 Renamed directory 'windows' as 'back-ends'.
35cccd11 Added a variant of HfstBasicTransducer::add_symbols_to_alphabet that takes a StringSet as an argument.
46143e26 Added all substitute functions to swig.
1a742cf7 Added XreCompiler functionalities to Swig.
009986e2 Now enumerators can be used normally in python, e.g. libhfst.SFST_TYPE.
53db94f2 Added Lexc functions to Swig.
417ecba0 Added more tests.
0d90bbb4 Fixed a typo in configure file.
3b8de303 Bring the HFST_LONGVERSION in sync with the other version strings (again - is there a way to calculate some version strings from others? I.e. change once, update everywhere?).
dc6f3df0 More updates to tests.
4146e0f3 Updated swig tests.
49df758b Fixed MonishAnalysis fsmbook-examples test
2468270c Added before and after to HfstXeroxRules
99ce6786 Added new test for tokenizer.
0b4a0c83 Fixed restriction in HfstXeroxRules
b3755419 Updated restriction in HfstXeroxRules
4b4e1022 Small updates and added documentation to tests and swig bindings.
eace5f2b Added new test cases and functions to the Swig interface. Also moved HfstFile to HfstTransitionGraph.h.
c262db79 Added restriction to HfstXeroxrules
ddf3b777 Added one more test.
79d27580 Updated tests, added some throw statements to the interface file.
7de5e18c Tentatively added a new test for swig bindings.
d7e0b484 Added HfstTokenizer to the python interface.
1676fda2 Added some files to _BUILT that probably should have been there all along (is this the cause of those build problems on some platforms?)
1a14d8f3 Tentatively added python version of libhfst test.
8328757b Added yet one missing lexc file.
77eda4ae Added a missing file in lexc tests.
22ae13ff Ready for release 3.3.15.
8164899f Fixed a re-introduced bug in insert_freely(SymbolPair&), now unknown and identity symbols are expanded during the operation. Also made svn ignore some generated files in swig directory.
f3e79adf Fix unnecessary shell call that breaks portability
e008efd2 Fixed bug in flag diacritic handling. Transducers were harmonized before insertion of missing flags, which lead to undesired behavior when identity/unknown symbols mixed with flag diacritics.
0fe80659 Updated doxygen documentation. Made svn ignore some generated files.
9944411e Removed bashisms from scripts.
012e4d90 Fixed some GNUisms ans bashisms in twolc and tagger scripts to make them work also on non-Linux platforms.
4075404f Fixed some GNUisms ans bashisms in tests to make them work also on non-Linux platforms.
7a9c6c7d Enabled hfst-tagger when compiling on Windows. Excluded the Windows versions of tagger and twolc tools from hfst-check-help.sh. Also updated Doxygen documentation.
5b310dab Added versions of hfst-twolc and hfst-train-tagger that use system calls instead of shell commands.
a5e86c5c Made small modifications to hfst-optimized-lookup so that it works on Windows. Now the transducer file is opened in binary mode and carriage returns are removed from the input strings.
16e7620c Just a test commit, nothing to see here
e4f68715 Parse 0 within @ flags as if it was known multichar symbol
d389a29c Fix escaped 0 at the end
c22c68d9 Warn users about wrong case in keywords
c1e59ecf Removed hfst-strip-header from tools, use hfst-fst2fst instead.
77c82f37 Removed hfst-preprocess-for-optimized-lookup-format tool.
5d6f85ec Updated swig bindings, edited NSIS scripts.
126e4b87 Parse version vector with serial strtouls
ff4d11e4 Bring the HFST_LONGVERSION in sync with the other version strings (?).
1cc8a0ec Changed HfstTransducer(FILE*, ...) so that epsilon symbol is an obligatory argument and added a new obligatory argument that keeps track of the line number in the AT&T file. Did the same to HfstBasicTransducer::read_in_att_format.
92e7a51b Added a directory for NSIS installer scripts.
685194ff Now FomaTransducer::read_net accepts also files with CR+LF newlines.
9844e68b Added wrapper class HfstFile for FILE* that can be used in Python.
41981582 upper- and lowercase versions of convenience acceptors
6d8f659b Added missing files to test/tools, fixed a typo in libhfst.i.
6ed258d8 Ready for release 3.3.14.
e7f77862 Added new testing scripts.
047964b1 Now HfstTransitionGraph::read_in_att_format throws an EndOfFileException if the argument FILE is exhausted. Also fixed a typo in HfstOutputStream.
791ead1f Added an alias for HfstOutputStream::operator<<, HfstOutputStream::redirect. Also added a Python test test_streams.py.
6b162d9d Fixed a namespace issue in libhfst.i, now omor_query.py works.
35360869 Added HFST exceptions to the Python interface. Renamed HfstBasicTransducer::at(HfstState) to transitions(HfstState).
7918308f Added more functionalities to the swig interface.
7300635a Fixed missing symbol table in optionalization bug for tropical weight type transducers.
56211a13 Added wrappers for operators [], = and << so that they can be used in swig bindings.
86104088 Added the function write_in_att_format(char * buffer, bool) to HfstTransducer. This makes it possible to call python's print() to a transducer.
cf198538 Tentatively added HfstBasicTransducer to the swig interface.
3fb1743c Changed Replace rules interface
923eb250 Fallback utf-8 tokenization
13e36ea6 Tokenize string literals for pmatch parsing
f1319a3b Now hfst-lookup handles the '\r' character right when parsing input strings on Windows.
f9b9344d HfstOlTransducer streams are now opened in binary mode.
5380f65c Added dlfcn include path to Makefiles.
65e903e3 Remove mentions of apertium and xml stuff
89ad0250 Multiline handling like regexp2fst
ccfccdee Fixed a typo in Makefile.am.
0902aaf8 Tentative names for charclasses
56e91496 Semicolon separation is default
0166e9e7 Context-checking markers in pmatch2fst parsing
29b863aa Make , reserved. Fixes [#134].
f0a77953 print yacc debug traces if hidden sekrit debug mode is on.
6e9dbd0c Added missing file sfst/interface.h
a359eb1b Changed ../../tools/src to  in fsmbook-examples to make tests more flexible.
55b39ede Changed the pathname ../../tools/src to . This makes testing more flexible.
bfba7cb1 Fixed a couple of typos.
b75728aa Renamed .c files to .cc so that c++-compiler is used.
06fdbfa8 Fixed sfst include path.
e30d3a08 Tentatively added SFST under directory windows.
d300ea2f #include <stack> (seems to be necessary when compiling without openfst at least)
02856854 Some compilers don't believe in universal characters given from the ASCII range, so instead of \u000D, have \r
3ab2f3ab Fixed format specifiers when printing long long on Windows. Also specified windows/foma as include directory in Makefiles.
6c9ef916 Added regex test case from the GT/Divvun sources that presently does not compile with hfst-regexp2fst.
a40bd436 Made fixes to foma compilation.
16309e05 Add charset-defining hardcoded transducers to pmatch utilities for now
85e6ba76 Foma's io functions are now skipped.
75930892 Fixed left replace rule
3270f5d9 Stateful context-checking mechanism; remove recursive functions for same
8c59f9ed OpenFst library check is now skipped and flag -D_MSC_VER is used when compiling on Windows.
9bde6a23 Fixed a typo in library name.
dfeb54cd Tentatively added foma under directory windows.
30a69420 Added generate_threshold parameter to prevent generating suboptimal forms.
9baaaeb5 Generate several forms if several forms can be generated.
67950e51 Added missing --format flag in hfst-strings2fst functionality test.
1a53070d Modified guessify to handle gradation in omofi.
898af0bd ACX tool disabled by default, requires libxml/tree.h and -lxml2
94a22e0d Temporary file - does not belong in svn.
e6666027 Test parallel left arrows [#132] part 2.
9053c679 Ignore expressions that _only_ contain whitespace. Fixes [#132]?
5d486155 Test not contains and multiline expressions with comments and emptylines, Bug [#132]
d92fb748 Bail on parse errors
fe31f8c7 No semicolons, huh
8c7725cf Allow windows or mac legacy \r's to end strings
3e511231 Stop recording input data to headers until header support for longer data gets fixed
5189a69b Only test fsmbook-examples if other tools actually work...
33aedf52 Do not use AC_PACKAGE_URL or resp. parameter to support autoconf's older than 2.63 or so
4ade921f Test that dos strings compile like unix
635fa087 tests should pass 77 if tool is not available
3d323248 Test that strings2fst handles M$-strings
335c4146 Fixed a bug in testing if compilation is done on Windows.
58d38760 Disabled using hfst-lexc with multiple input files on Windows.
2076847c Fixed some problems with automatic LF-to-CRLF conversion on Windows.
3843896c Now hfst-twolc test uses 'uname' to find out if Windows is used.
a0d41f42 Added excutable file extension '.exe' to tests when running them on Windows.
1da9324e Disabled tagger tools on Windows until they work there too.
3d2c49b9 hfst-substitute now recognizes '@0@' as epsilon, fixes bug #133.
4ff55bd7 Noticed missing files and tests that should be skipped during distcheck.
d7576d0b make dist revealed an extra file listed in Makefile, skipped that
348deb65 Ready for release 3.3.13.
970f3550 Merged changes from old repository.
5bf1aa1f Undo conditional compilation of pmatch compiling stuff
004ee605 Limit memory consumption a bit by clearing SequenceModelComponentPair state tables between tagging sentences.
73a589c9 Make pmatch parsing code conditional on WANT_PMATCH2FST
805b7651 Changed sizeof(input_symbol) and so on to sizeof(SymbolNumber) since sizeof can only be applied to static data members according to standard (though not all versions of gcc).
c08e4ec9 Changed WeightedStringVectorCollection test so that it takes into account the model penalty weight line.
e7567b35 Fixed incorrect loop condition in previous commit to this file
0691b075 Added newline between results for different words.
2205e95d Recommit fix to ol -> basic conversion bug (hopefully)
634912e4 missing part of %0 handling
c931c665 Option --enable-windows is passed to Makefiles.
58812ce7 hfst-twolc test is currently skipped on Windows. hfst-twolc is a bash script that will be replaced by a batch file on Windows.
60f72ef4 Fixed multiple defintions of SequenceModelComponent::START_STATE in make check
8e66c759 Comment out xre-specific tests for the time being
8c0cb26a Merge missing header file
d8f8acac Merge changes to tools pmatch2fst and pmatch
87cc83ab Merge pmatch parser changes
b51a7be0 Merge inclusion of hfst.h in HFST_HDRS
b6d5c398 Merge pmatch library parts from devbranch
786ec44a Improved error messages and fixed help message.
8e887c86 Maybe fixed undefined ref. to SequenceModelComponent::START_STATE problem.
b7caa51d Set revision for info
ffbd4f23 Newlines are now handled correctly when testing command line tools on Windows. Version and help checks also skip the tool hfst-info.
849f3dfc Further porting for info
d65900ac Missing square bracket in define
536ebc81 Port basic info app to trunk.
2ab0507f Fix conversion bug where when converting to BasicTransducer, we would stop looking for a particular flag diacritic when seeing symbols that aren't epsilon or this flag. This caused misses of flags when they weren't arranged in a lucky way.
fc0dca39 Uh, actually declare hfst.hpp in headers and not files to get the meta
cdddfe1d Port some versioning and meta from branch
6f2af570 Replaced size_t format specifier %zu with a macro that is expanded to %zu or %Iu (Windows).
a86675bd Removed debugging prints from HfstTransitionGraph.h. Added --help information to hfst-summarize.
d5084f0d Fixed python and utf-8 related problems.
126e2492 Added preprocessing flag -DWINDOWS to Makefiles if HFST is compiled on Windows.
afc1e602 Streams to files in TropicalWeightInputStream and TropicalWeightOutputStream are now opened in binary mode. The global variable _CRT_fmode is also set to binary mode in file globals-common.h when compiling on Windows. This makes sure that streams are always processed in binary mode on Windows, with no extra or missing carriage return or line feed characters.
fe9eab01 Add Mac OS X abort trap to faqs
8218c9b0 Move "stuff that is next to colons" to another level to support quoted literals and some other constructs maybe
375ee882 Provide more info about itnae's in lexc for mac users
cba4f88a A convenience header file including HFST headers necessary for doing practically anything
ea4342fd Added OpenFst and dlfcn functionalities as a part of HFST (at least when it is compiled on Windows).
afc1c34d Added a script for Monish guesser-analyzer test.
81c654ea Installation instructions for dependencies that are not in our control...
223b5fba No alphas in trunk
3c0d30d9 retab!
86115d25 Add port and gentoo installation instructions
1d50ca06 Added reweighting tool.
e5cc7688 Added config-file.
1be6c2b0 Added size() function.
0d8cfdd9 Added dummy-tags to guesser and lexicon entries without real tags, so that the could be parsed in the same way as regular lexicon and guesser entries.
48139458 Fixed output file, so that output is printed to stdout, when no output file is given instead of file "<stdout>".
93d16dec Changed guesser tags.
2826e479 Some default-transitions were missing. Added them.
999fb85f Another #include <hfst/HfstTransducer.h> -> #include "HfstTransducer.h"
68a396b4 Added case sensitivity to suffix counts.
b9123f84 Added 'MonishGuesserAnalyzer' test for parser
87780163 Continued adventures in unsigned short reading: cast to unsigned char, not unsigned short, before shifting and adding the bytes.
aa6d4311 (Regular) containment fixed in 'Regexp2fst'
c975b5fc Fixed kitwiki addresses in the help-messages of hfst-guess and hfst-guessify.
c4e6d9eb Fixed printouts so that they match for guesses and generated forms. Fixed printing of guesses, so that they're not printed reversed.
2bca1a1e Fixed hfst-guess help-message and exit value.
34be6db7 Implement a little-endian unsigned short stream-reader on top of the char-reader. Use it to read the header size rather than fiddle around with signed shorts.
429f8137 Initial commit of hfst-guess utility.
bfb86fad Initial commit of the hfst-guessify utility.
ed7b61d8 Complement added to the 'regexp'
167a297f When reading header size, which on the disk is a two-byte unsigned number, we need to call stream_get(short &) because it's not implemented for unsigned short. Then cast it to unsigned short.
73db847f header_size needs to be able to handle MAX_HEADER_LENGTH=65535 bytes, so make it an int instead of a short
ab3de483 Excluded source files under windows directory from distribution.
f6b281f8 Curly brackets added to regexp
2e53d411 Appended $(EXEEXT) to program names in Makefile.
18a5340f Temporarily disabled 'hfst_mkstemp' in windows compilation, because it uses 'mkstemp' that is not standard C++.
92e33adb Fixed a typo in Makefile.
f3eaf25a Complete purge of trap_transition in try_epsilon_transitions
0165a14c When following epsilon index, set found_index *after* returning. Added rather verbose comment to try_epsilon_transitions to clarify when transition-taking needs to be trapped and when it doesn't.
9090b553 if -> endif in Makefile
6dbeb663 Now all Makefiles should be aware of windows compilation.
838b7d9f Added include flags to Makefiles when using windows compilation.
a1e00225 Made autotools more aware of windows compilation.
25dae67e Edited Makefiles.
17599d99 Added directories to windows/openfst and a script to fetch openfst.
b8702842 Added a script to retrieve dlfcn.
216f22b8 Tentatively added support for windows compilation.
79cb9421 Implement PCKIMMO output mode for fst2txt
b898898e Ready for release 3.3.12.
519d7986 Tentatively added the possibility to compile hfst in Windows to configure.ac.
a2f19707 Maded changes in tests required by previous edit in WeightedStringVectorCollection.
5c7f95a9 Changed a bool to weighted_string_type in add_sequence function.
9dd0f5ef Changed the bool lexical_model parameter in the constructor of WeightedStringVectorCollection to an enum.
9b684a76 Fixed comments
5f896716 Fixed initial comments
01daf8d6 Fixed statistics.
7a6ac4c4 Fixed statistics.
a644eb12 Added documentation.
d49bd992 Use convenient STL set constructor with InputIterator objects for returning alphabet sets
9bd08336 Maybe find unit_test_aux.cc now...
f4da277b Support StringSet get_alphabet() for hfst_ol(w) types
e09ccf7f use am_prog_ar for automake >= 1.12
2ef38381 include unistd for systems where it does not leak in
38af24a9 Handle the "only default transitions in state" case
39979b4d Now perhaps building unit_test_aux.o properly...
1ba1277b Build tagger tools only if the enable-tagger flag has been used during configure.
a3441eb5 Changed NewLexicalModel.h so that it includes the local header before instllation.
1f098c5b Added unit_test_aux.h to the sources of the test for string_handling.cc
692544e8 Added DataTypes.h to the sources of appropriate testa and hfst-tag.
62b23b0f Added version of hfst-tag.cc that compiles.
accb745f Tidied up code in use_model_src.
655db720 Now we pass all use_model_src/ tests.
ca283ad0 #include "HfstTransducer.h" instead of <hfst/HfstTransducer.h>, since often people will be building this without having headers already installed
7c8a1d66 Remove lexc tests as long as parser is not reusable
fc6488f7 Added skeleton for hfst-tag.cc and got all tests in use_model_src compiling. Some tests still fail.
dda2e749 Removed printout from the unit test of FstBuilder.cc
9d1942ba Fixed WeightedStringVectorCollection.cc and string_handling.cc
0dc17bc8 Tidied Makefile.am.
e165644c Tests are now run using make check.
5e922c64 Added invert parameter to compose_intersect in HfstTransducer
b501a645 Added hfst-open-input-file-for-tagger.cc, which was missing.
cc9b9017 Use intree headers for intree build
8f32733b Added some printouts to verbose mode.
278e9b19 Fixed issue with exit value.
017d6534 Updated build files so that tagger tools can be built. They are not built by default.
fc1c5018 Fixed some in input handling bugs.
c9f5a017 Changed comput_data_statistics.py to hfst_tagger_data_statistics.py
fcf3028e Some changes that actually allow compiling and installing hfst-train-tagger
3897533b Added test dependencies.
ebbf67fd Fixed test defines in build_model_src/.
6cd2357d Added unit tests to hfst-tagger.
0fa55e4d Added Makefile.am files to hfst-tagger.
84f74342 Added src and test subdirs to hfst-tagger.
ba485b63 Initial commit of new tagger lib architecture.
e705d555 Copy input to output for default transitions
268a4b98 Allow reserved chars between quotation marks, avoid one more segfault on parser error
bae41c7d Add quotation example that breaks the legacy parser; use all tests on both lexc front-ends
f944ea58 Emit parse error on unopenable file...
df6fb6e0 Pass parse errors upwards from LexcCompiler
b8e33d7c Remove temporary lexc concatenations on successful exit
c948c230 Reword -q to "print only fatal errors and requested output" as it does
30a29471 Notice @_DEFAULT_SYMBOL_@ in other Alphabet contructors than the copying one as well
7a0372e5 Add an internal boolean found_index to keep track of whether an index has been successfully hit in the current iteration, and try to find a default symbol otherwise
8cbdcfa2 Make alphabet know about default
5a95634a @_DEFAULT_SYMBOL_@ and hfst::is_default()
16d45a5b Remove warning messages and XFAIL tests that may confuse users. Fixes bug #3525179
9e9e4215 Added READMEs under BrazilianPortuguese 1 and 2 to EXTRA_DIST.
cbd74310 Updated Doxygen documentation.
a9e2b8ac Ready for release 3.3.11.
6cc30cf5 Now hfst-shuffle is disabled by default.
4f3fcee9 Added the command line tool hfst-shuffle and functionality tests for that.
b455911e Tentatively added function shuffle under HfstTransducer.
eb6581f9 A nicer single production for INSERT
c560a0ff Fixed a bug in SfstTransducer::random_path that happened when indexing states of the argument transducer.
bd37642c Introduction of semifunctional pmatch2fst
7870bf50 No plus sign in front of tags in CG mode, not even the special carry-through tag.
24449d5a Added experimental support for printing the raw analysis tag as part of the Xerox analysis. It is printed as a tag with a special prefix, and with the full analysis string enclosed in quotes. The idea is that vislcg, if enhanced to read the Xerox format, should carry  this "tag" through the disambiguation (and possibly analysis), and print it out on the other end. The tag can then be used as input to morphological generation.
9646a733 Print the raw analysis string as a specially prefixed tag at the end, instead of as a sub-reading in the beginning of the analysis.
92ee0bd0 Now hfst-fst2txt functionality test tests whether the program dot exists before using it.
d4a39a74 Changed call to the installed hfst-regexp2fst to the one in tools/src/.
22d4fcd4 Changed call to the installed hfst-regexp2fst to the one in tools/src/.
ab7257c5 Add containments and term negation, remove nasty windows formatting markers
f321bb34 Added the option --print-separator in hfst-fst2strings. The option will print a separating line "--" after each input transducer is processed.
70fe665c FomaInputStream::write_transducer and SfstInputStream::write_transducer now call fflush after writing the transducer. Now the command line tools should work correctly when pipelined.
f6c6432e The regexp [?] was parsed as [?|?:?] by FomaTransducer::define_transducer(const &string, const &string). This is now circumvented by using FomaTransducer::define_transducer(const &string) instead. This bug will be reported and fixed soon.
e4d291d3 Manpages are not created unless 'make man' is executed in directory man.
e47d8c07 Added support for inverting the composition i.e. computing intersect(rules) .o. lexicon instead of lexicon .o. intersect(rules).
37684b75 Added possibility to invert the composition in compose_intersect i.e. to compute intersect(rules) .o. lexicon instead of lexicon .o. intersect(rules). Also added added support for indetity symbols in the lexicon to compose_intersect.
71a2579e Removed HfstConstantTransducer as it is not used.
a9cbc171 Removed FinnishOTProsody from tests as it does not yet work. Also added HfstXeroxRulesTest.cc to EXTRA_DIST.
7cdf97c1 Temporarily excluded hfst-lookup from man page creation (due to a bug in help2man?). Also updated hfst-tail and hfst-head man pages.
4d54a066 Heads and tails with plusses and minuses
6b15fcf6 Test negative heads and positive tails
58723c30 Added Brazilian Portuguese tests to fsmbook-examples, they now work because missing files were added.
72d1a802 adding missing files
9daa9863 Added missing files
31d424ee Ready for release 3.3.10.
9bc5df7d Brazilian Portuguese tests disabled in fstmbook-examples to get a working release. They probably had missing files.
afce28b8 Updated man pages of binary tools as their help messages were changed.
70d24a54 Added documentation to the binary tools about the restored feature that the first input can have several transducers while the second has only one.
7c2399f6 TropicalWeightTransducer::extract_random_paths now also calculates the weight of each path. Fixes bug ID 3514912.
67a377ee All binary tools now handle the case where the first input contains several transducers and the second input exactly one transducer.
e23d373a BrazilianPortuguese2 fmsbook example added to tests
f941d029 BrazilianPortuguese1 test should pass now
6ad4d9a2 hfst-compare now handles the case where the first input contains several transducers and the second input exactly one transducer.
c55bc43f hfst-compose now handles the case where the first input contains several transducers and the second input exactly one transducer.
abbff8c7 hfst-summarize now prints at the end how many transducers it has read in total.
d765699a Added documentation to different cases that are gone through when checking binary parameters.
e32b8a28 Changed the variable name stdout to stdout_ to avoid namespace conflicts.
1bc12a05 forgotten Makefile.am
8672375f Configure file had to be changed to compile fsm book examples
84850a80 Fsmbook examples update
c1f25103 Fsmbook examples update
7eb574c0 In xfstRules, removed TMP marker from the result alphabet
610da466 Updated hfst-compose-intersect help message.
dad67cc7 Appended namespace std:: in front of vector and pair where avoiding it might cause compilation problems. Also changed calls to 'rindex' to 'strrchar'.
74fe14d9 Doxygen for the speller and some ol internals
c3f629ee Forgot to commit ChangeLog and NEWS, now ready for release 3.3.9.
e55c21a7 Ready for release 3.3.9
34460a62 Fixed help messages of tools.
814a1a98 Changed hfst-lexc so that the implementation format defaults to openfst-tropical.
7b97b483 Changed expected output to reflect current cg printing
97a0786a Integrate ospell, first commit
2624bf62 updated so that tests don't fail
20ab7961 updated one fsmbook example, still not working properly
9b826c10 Double greek PI met opposition in #apertium, hereby changed to double dagger instead (‡). It is still open whether the raw analysis string should end up in this position at all.
03772d6c Fixed a bug in command-line option parsing, where -e conflicted with all case-handling options.
c96e52eb Fixed bug wrt option counting
a7088556 Added a new option to turn on printing of raw analysis string in CG format output. Default is to not print it.
7c8830aa Replaced triple ] (]]]) with double ∏ (∏∏ - capital greek PI) as the CG sub-reading separator. It turned out that inserting +]]] into the transducer was not working very well.
f06aac34 Deleting the + char in front of GT/Divvun tags for the CG output. Also, there is no problem with printing the raw analysis string, all is fine (contrary to what I said in my previous commit). This tool is now pretty close to what we need in the Divvun work :)
990581e0 Experimenting with the CG output mode. Using the -C option, the output will now include the original lemma+tags string, including all tags within compounds etc. It is printed before the CG-formatted tag list, and separated with ]]] (suggested CG sub-reading separator). For some reason, the tags have lost their initial + sign at this point, so the result is pretty ugly and unusable. I don't know why or where - feedback appreciated.
f3c746f9 Make sure the outputs are sorted before filtering and printing
4905d540 Corrected key string for getopt_long
05fe312d Corrected condition in LookupPathW::operator< (or at least made it more understandable)
58c61e59 Correct terminating condition
ec52c631 Stop before end of output iterator
ae8461f6 Clarified comment
074a25e8 Added an option to only print analyses belonging to N equal-weight classes
6844c9de Typo in error message
97926411 Adding identification and processing of GT-style tags beginning with +. When formatting for CG output, this is important, otherwise the output is more or less useless. This change should probably be enhanced with tests for the requested output format, so that it does not contaminate the Apertium analyser. Please have a look.
756d724c indentation
3e37af1c Added a quietFlag variable, which makes the -s/-q flags actually useful. Using either -s or -q the warning "Warning: Transducer contains one or more multi-character symbols made up of ASCII characters which are also available as single-character symbols." will be suppressed. There might be other output that should also be suppressed, but that is not implemented yet.
784bd214 Removed "e.g." to avoid ambiguity in the interpretation of the output.
cdb80a2a Boundary mark in contexts is beeing handled now.
e999674d Xfst rules parser should be working now
8d53567e regexp updated, precedance seems to be fine now, but parallel rules with one comma and context still don't work.
9359776b FomaTransducer::extract_paths bug fixed, now it does not produces duplicate paths.
76e4f7a1 Ready for release 3.3.8.
b55429b1 Add expected arcs per symbol statistic
410a9ec4 Ready for releasing version 3.3.7.
8292ce44 Updated tool man pages.
939b6a90 Updated hfst-regexp2fst help message.
3df0bcb1 Fixed an error in composition introduced by the newest version of OpenFst (1.3.1).
bb6e2fd1 Fixed identities in regexp parser, now [?] is parsed as identity pair.
8f950ad2 Updated regexp2fst parser - rules work fine for themselves, but there remains precedance issue.
2512805b Ready for HFST release 3.3.6.
3d290d35 Changed include directives of implementation headers (foma, sfst, openfst) so that they search for headers in default include path first. Also modified memory leak script.
69f06195 Updated memory leak test.
5ece7d78 Tentatively added a script for testing memory leaks.
cb48f49d Fixed memory leaks caused by hfst_get_name in command line tools.
679d8ca2 Fixed a bug in hfst-lookup man page generation, now option --no-discard-stderr is used in help2man. (However, cannot figure out why this helps..)
942a563b Removed yet some extra code.
b50e3846 Removed commented code.
6e87a259 Fixed some compilation warnings and removed commented code.
bf0508d5 Fixed some more compilation warnings.
1ebf3aa6 Fixed more compilation warnings.
3fd5aee0 Fixed some compilation warnings.
b8026210 Ready for HFST release 3.3.5.
e0966766 Man pages updated.
1aa56860 Modified Makefile.am.
2de59171 Added some comments
d155bcaa Fixed a small bug in hfst-fst2txt functionality test.
e6560e20 Output dot/graphviz
967a5bee Maybe finally the reversion is working..
5373734a Made yet some changes to revert to the older revision.'
b4318deb Reverted to revision 2076.
1101e6a4 Forgot to add the new files for BooleanWeightTransducer.
21b39493 Tentatively added a new HfstTransducer type, BOOLEAN_WEIGHT_TYPE.
6e99a0bf Fixed a small error in the options handling part of the help message for hfst-strings2fst.
8514c035 Added new man pages for hfst-name and hfst-format and links to man pages for tool aliases.
76d8934e Now the man pages also have a short description of the tool, i.e. the one given by hfst-foo --help.
4f996f51 Added a test case for the bug fixed in revision 2016 (out_of_range thrown by substitute function).
5e805977 De-spanishify hint string ("argumentes" -> "arguments")
d29274ee Now HfstTransitionGraph:substitute(const HfstSymbolSubstitutions & subs) adds all symbols in subs to the global alphabet before relabeling, thus avoiding the out_of_range exceptions thrown earlier by this function.
467599ea Changed hfst-compose-intersect so that it gives a warning (1) when the rule does not contain the identity symbol and the output tape of the lexicon contains a symbol not found on the input tape of the first rule, and (2) when the output tape of the lexicon contains multi character symbols not found on the input tape of the first rule.
6bbf8e10 Removed spurious consts from optimized lookup conversion functions. Fixes bug ID 3469227.
7315fe44 Bug on mac os caused by its sort order (bug ID: 3462485) fixed.
e9f37fff Added function write_in_att_format_number. Also fixed hfst-optimized-lookup so that --version and --help print to standard output instead of standard error.
2284d132 hfst-optimized-lookup softlink and some package strings for hfst-optimized-lookup.h
4821cdff Heavy-handed bundling of the hfst-optimized-lookup standalone
a7e3dd17 Rules parser - added epenthesis
4f70b1de Still needed to fix some errors caused by in-place sed.. Now everything should work in commandline tools tests.
33130abb Fixed problems in test/tools that came after updating the Makefile.
bc413c7b Removed commas from filename suffixes (e.g. .hfst.sfst -> .sfst) in test/tools. Also replaced compilation rules using %-symbol in Makefile.am with old-fashioned suffix rules. Now commandline tool tests should work with older versions of make as well.
43082be9 Ready for HFST release 3.3.4.
5b163cbe Added some documentation for HfstSymbolSubstitutions and HfstSymbolPairSubstitutions. Also updated the doxygen documentation.
53c7b7c6 Now HfstTransducer supports making several substitutions at the same time. Also made a test for these functions. Adding documentation soon.
7aa7026a Tentatively added a symbol-to-symbol substitution function to HfstTransitionGraph.
e9db3cea newer version of the rules parser
31446ffa weighted expressions
668878f5 added some xerox rules to xre_parser (hfst-regexp2fst). Also,  slightly modified xerox rules
aed6b501 Remove some debugging printout
45c7b454 Merge xre precedence fixes
81ba993f Now const_cast<char*> is used instead of strdup/free in ConvertFomaTransducer::hfst_basic_transducer_to_foma. Also modified hfst-substitute so that the alphabet is not pruned after substitution.
cff563a7 hfst-substitute now implements the option --in-order.
5020e2d7 Fixed bug, which prevented compilation of grammars with both set constructions and : or ?.
0aa972c2 Unit test for grammars which have both set constructions and : or ?
01839281 FomaTransducer::repeat_n now takes a copy of its argument transducer to avoid double delete caused by new foma interface. Fixes bug 3468373.
253b0fe7 OpenFst linking bug is fixed in 1.2.10; also gqipped some paras
db1e0663 Warn end users about disabled "important" tools
93c0be84 Xerox rules added to the library
a6b19829 more alphabet hacks
2f135747 more alphabet hacks
d41ccd2e Added new substitution functions. Now it is possible to perform many string pair-to-string pair mappings at the same time on a transducer. Also tentatively added a functions that performs many string-to-string mappings at the same time.
b19e466b Now the test EinsteinsPuzzle uses local tools instead of the installed ones.
f574caea Now the test FinnishNumerals uses local tools, no the installed ones.
dfcd75df Conversions between HfstTropicalTransducer and HfstBasicTransducer also made more modular.
e64a2eb0 Made conversion functions more modular by moving code to subroutines. Also removed commented code from files.
ab8c339f Now conversions between transducer formats openfst-tropical, sfst and foma are carried out using numbers always when possible.
4defd162 Tentatively added functions for fast conversion between transducer formats.
90fd45fa Now also conversion from foma to basic transducer uses numbers. Some commented code also removed.
45cfa97c Some conversion functions are not carreid out using numbers directly.
03356460 Renamed HfstTropicalTransitionData variable 'number2symbol_vector' to 'number2symbol_map' to avoid a strange linking error. Also removed transducer conversion functions that are not used.
d964d9cf Removed some extra code.
1f9516e5 Tentatively added alphabet pruning to substitute functions, it still has to be removed from hfst-substitute.
6e4838f4 Conversion between HfstBasicTransducer and TropicalOpenfstTransducer is now more efficient as it is done using numbers directly.
fc65b315 Changed conversion from HfstTropicalTransducer to HfstTransitionGraph so that it uses transition numbers directly. This improves the performance greatly.
6fe20061 Fixed bug which made tests always pass.
7f0d49ab Ready for HFST release 3.3.3. Also disabled DateParser test because it relies on hfst-calculate.
a2b7ede9 Fixed speces so that the space symbol is used in the transition instead of @_SPACE_@.
8bceff5a Skip copying the foma transducer name to the net, leaving an empty string. Rationale: foma has a limiting 40 char ceiling on names, and we might as well use HFST3-style names for everything.
e90d002f Removed hard tab indentation from hfst-txt2fst.cc
5d4e1ab2 Replaced tabs with spaces in hfst-lexc.cc
932a7808 Fixed HfstTransitionGraph:swap_states so that the variable new_target is no longer used uninitialized.
b85d91d4 DateParser test fixed, the path os hfst-calculate was not correct.
7de913d7 Fixed ComposeIntersectLexicon test
97e620de Now the escape '% ' should work everywhere.
abebc486 Added test for rules with space symbols.
f7341b23 Added a conversion test and made the LIBHFST_TST actually run, causing test failure for the time being
69b7b7bf Optimized HfstTropicalTransducerTransitionData::get_symbol(...)
aabea1ce Call sizeof() on type names, not variable names due to OSX compilation thinking the variables are non-static uninitialized fields. Fixes #3445592.
5024ecdc Fixed (serious) typo
f8f2dfd8 (re-)added a char* constructor for tables, made it the default to only read disk once per table
6754159b Eliminate use of non-portable strdupa(const char *). Fixes #3444950.
43cc4e9d Removed some garbage
5ffe9692 Removed some of the unnecessary work done in tokenizing for ol-lookup
0c09d911 Fixed a name shadowing bug
bca34efa Added a lookup_fd(char *), reduced code duplication in interface functions
c9ff4533 Added note about current OpenFst / Ubuntu problems.
0ce433e7 Fixed error message.
c289dcd5 Now terminating compilation, if an empty pair is found in a defintion, rule center or rule context.
652f9273 Ready for release 3.3.2
aefa028d Use NumArcs as a member function instead of using NumArcs from the OpenFst internal namespace.
366a4830 OpenFst has moved NumArcs into the namespace ::internal. This commit reflects that, fixing compilation with the newest version of OpenFst. NOTE: THIS BREAKS BACKWARDS COMPATIBILITY, you need the newest OFST to compile after this commit.
67a5402e Fixed svn properties of executables in fsmbook-tests.
128e3909 Remove PKG_ macros from end user builds since they do not work nicely on all platforms.
0deefb73 Tentatively added to fsmbook-examples tests where the results from foma and xfst are compared.
fff58c26 Now all tests in fsmbook-examples use the tools in tools/src/ instead of installed tools.
955e7602 Tentatively fixed parts of fsmbook-tests where intalled commandline tools are used instead of tools in tools/src/.
b6a2bd3b Ready for release 3.3.1
3a9f7c8e Fixed memory leak in HfstTransducer::get_property
48232c17 Remove escape sequence __HFST_TWOLC_ from symbols in rule names.
37800e66 Changed flag diacritics handling in hfst-compose, hfst-conjunct, hfst-concatenate and hfst-subtract. Now flags are harmonized using HfstTransducer::harmonize_flag_diacritics(...).
9313133a Added a function for querying if a transducer has flag diacritics and added a parameter in harmonize_flag_diacritics for diabling insertion of missgin flags (needed for hfst-concatenate).
e207282c Added function bool HfstTransducer::has_flag_diacritics(void) const
47f9d3e7 When looking for indexes with a particular symbol (which is used for converting to other transducer types), handle flag diacritics specially as per their indexing rules.
8eb758b8 Added HfstTransducer::hamonize_flag_diacritcs(HfstTransducer &another)
f2c0a980 1) Report the correct indexing offset for epsilon as 0 even when there aren't     any epsilons 2) Put the exception about trying to find the offset of absent symbols in     the proper scope so it actually gets thrown
cafa28f3 Tentatively added to fsmbook-examples tests where results from foma and xfst are compared. Also fixed a small bug in remove-complement.sh.
3431ec34 Extra files created during tests are no ignored by svn.
5f9386f7 Added flag diacritics to the epsilon filter.
0780530b Added missing file.
f76ac72b Added a note on configure enable/disable switches to README. Changed fsmbook-examples so that now all examples that can be compiled with the current HFST tools are included in the tests.
9c006b50 Forgot to commit configure.ac last time, now automake should work.
e4370100 Made some modifications to fsmbook tests.
a651082c Same leak issue as in r1846
b2b4ba85 free() char*'s that get strdup'd in a loop (which leaks memory if not free()'d).
edcc8e64 Added missing file.
f24568bc Made small changes to the tests under fsmbook-examples, now make distcheck should work also when scripts in an upper-level directory are used.
e1675735 Tentatively added one subdirectory of fsmbook-examples to the tests performed in make check.
31af546e Tentatively added some testing scripts to fstbook-examples
23978b11 Remove spurious space character from error message
26fac37c Give more guidance when ejecting the user for having a non-UTF-8 charset
a4981ce7 Comment out fsmbook-subdir; it's broken
47a618df fsmbook-examples are now included in tests.
08e8eddd Ready for HFST release 3.3.0.
e3819ad8 missing  added to some tests so that they pass distcheck
3b0667c2 lexc-functionality updated
511b33b5 make distcheck should now find the missing files
b4f2e58c remove hfst2 broken flags from main tree
dccfa4f5 Remove experimental, hard to compile and dependent programs from main compilation
1bad1e88 Added missing files to Makefiles. Also replaced HfstExceptions.h with HfstExceptionsDefs.h in include declarations.
f29786aa Option -s in hfst-lexc and hfst-substitute now suppresses warnings as it should.
a0e51bd2 Made yet some tests less verbose.
fdaba542 The option -s in hfst-lookup now suppresses warnings as they should.
6a4461e3 Made tests less verbose.
0c79b596 Testing if a library exists is now done with the command 'hfst-format --test-format'.
a29cc4fb Now all tests in test/tools should pass.
667fb489 Fixed a bug in hfst-format. Also fixed some tests so that they don't assume that all backend libraries are available.
95227375 Indicate whether apertium2fst is enabled
32d75280 A spurious space was apparently preventing libhfst.la from getting built (!!)
9fe6a3d7 Fixed some small bugs.
373ab19c Added a missing file.
87d5c057 Now all Esperanto tests work.
311297f1 Now EsperantoNounsAndAdjectivesWithTags passes.
5703bf4f Mention both forms of <SFST-1.4.6 error messages
1ccdbf2a Fixed some bugs.
1a85f7d7 EsperantoNounsAndAdjectives works now.
dbe38589 Remove hfst-lexc and feature check that requires newer autotools from default builds
c8bc7d0a Updated the tests.
3dab55c8 New tests.
1158256b HFST scripts for Monish tests ready.
fd3adfe7 Fixed a bug in the xfst file.
1f574399 Adding a new test.
3cb051d8 Added files to MonishAnalysis
5275de0c Adding a new test case
f3064634 Added an alternative solution to Brazilian Portuguese.
65521fb8 Now unicode characters are transformed correctly from xfst att format to hfst att format.
6f3e3f24 Added new test
bb769eb1 Added a new test
7b5175a7 Fixed some tests so that they work when only openfst is installed.
e44b7046 use const char * in bison's error handler
2a8afa1d Fixed tests in test/libhfst so that they also pass when only OpenFst is installed. Also changed HfstTransducer::convert so that exceptions are handled correctly.
4e85a00d Ultimate fix to implementation type availability checking in twolc unit-testmake check
9df0c5fb Tentatively fixed unit-test in OtherSymbolTransducer.cc
ba60bbe3 Added checks for existence of implementation types to unit tests.
3a313b0f Added support for new backend format strings in commandline arguments.
7007722f Added check for existence of backend formats to tests.
34654294 HfstTransducer::is_implementation_type_available changed from protected to public. HfstTransducer and HfstRules tests now work also when all backend libraries are not available.
298d1f71 Only respond to nick + , or nick + :. Fixed flood-handling.
d04b9369 Update instructions for easier svn source
e38782a0 Disable problematic backend libraries from default build
3157352b Remove all version requirements and dependencies and tools that require versions, features, or dependencies to support older tool chains and systems
bbb40b16 allow modifying only end state
65e20296 Fixed character coding in Finnish examples.
7ed69057 YaleShooting test complete, rules must be implemented in HFST commandline tools before it can be tested.
48bc808e Changed 'Palindroms' into 'Palindromes'.
ca49d0c3 Add tool for modifying weights
e737f9da Now the test passes also for foma implementation type when revision 47 of foma is used.
47696bd0 Added test files.
59521566 A bug in HfstTransitionGraph::swap_states fixed, now final states are swapped too.
f41b96e0 Added documentation
42e26a2f alias old proc
098f6a3b add bugs
32eb73d8 Missing -r option
1b3e7247 document input format requirements in the beginning
fc22386a few less apertiums in directories
d01f0725 s/proc/apertium-proc/g
62c5a1f0 Added a test case, Lingala. The test is not included until an issue with writing transducers in AT&T format with the limited licence of xfst.
cc616c6c Added a new test, FinnishOTProsody
3fd80a9e Added a new test, PlusOrMinus
a5b9ea01 Fixed a bug that occurred in binary operators when the calling transducer and the argument transducer were the same.
ef31d2f3 Added more fsmbook examples. The tests have to wait until we get rules in hfst-regexp2fst.
00da234d Terminate xre parsed quoted string, fixes 3400657
444dffde Declare BUILT_SOURCES
5ccdf409 We are one step closer to get the tests in test/tools/fsmbook-examples under autotools..
33db26f1 Added Makefiles to fsmbook-examples. The tests in fsmbook-examples are not executed until they use autotools.
97d473c6 Now the tests do not depend on the tool hfst-xfst. Instead, they use hfst-txt2fst and a transducer in att format.
0d6b28e9 A bug in FomaTransducer::write_net fixed. Now the the function fsm_count is called before writing the transducer so that the number of lines is always correct.
8d28394e A testfile added.
cd193f6c Use basic transducer and trie disjunction
91754dea Works minus regexps
1b73d6e3 dl before openfst though
86921ca0 do we still need pthreads and m?
0c4b53fe Yet another missing header and using
b5671cd1 Removed file training_data, which was accidentally added.
60e47779 Initial add of hfst-tagger. A lot of stuff is still missing and the tool needs to be integrated in the build system.
c2842598 tests modified
9799eb24 some of the tests rewritten
01f60b61 Now the tests are performed for sfst, openfst-tropical and foma types.
f60e5805 typedef HfstTransducerPairSet removed from HfstDataTypes.h because it is not used anywhere and would cause problems because HfstTransducer::operator< is not defined
9c47fc10 Now HfstTransducer::operator= works for the case of optimized lookup transducers and copies the name of the transducer for all cases.
efa33f53 copy props in copy constructor (excepting type); fixes bug #3405831
02e7be94 make compilation work without inclusions leaking from backend headers
16b4d3cb add include for std::map
461af249 add include for std::set
2f44bbce apertium2fst tool added
ca39ff3a Avoid writing name, version or type twice
8119d69c Added necessary std::qualifiers. (With --without-sfst some using-directives apparently disappear.)
1917eee1 Fix UTF-8 bugs in strings: * use error_at_line instead of error when linens are known * remove bogus double try blocks * throw UTF-8 errors with messages
9e80b3ec Fix formulae for filenames and data starting with non-ASCII
62ce06db Revert tab-containing commit r1576
b3f24d81 fix missing paths in tests (would've used $PATH instead of newly builts)
7e11d85f Use new metadata handling in tools * fix signedness in -grep * fix erroneous error printout in -compose-intersect * add few features to summarize
b2b30103 * common practices headers and utils for metadata in cli tools * a simple cli tool to edit metadata
9829fcea metadata header name registry
ab02d032 Make use of HFST3 transducer format metadata: This is basically upwards and downwards compatible change with HFST3 automata and header parsing, but be cautious that everything may break at random
0857f80c Added Beesley and Karttunen's fsmbook examples to the tests. The fsmbook tests are not yet run when doing make check.
5fe18444 Fixed bug that caused hfst-twolc to segfault when variable rules with keyword matched had unequal value lists. Now it will give an error message and terminate compilation.
6fd297d9 Cross product added to the library
cc5671e7 Universal pair update and added identity pair
459bcaad A bug in transducer conversion fixed.
711df751 Don't do foma-SFST conversion if we don't HAVE_SFST
7317a586 Warning about broken foma handling
09419fae More accurate comment
3993996a Do foma-sfst -conversion for substitute also (not just pruning); HfstTransducer doesn't know about foma's substitute.
c3c3dbc1 Typo
687639ed Swing foma substitutions through sfst to prevent horrible leaks
24a982dc Don't leak 1 (!!) byte of empty string when constructing foma transducer
727bcd3f Fixed mismatched free/delete when freeing foma transducers.
f199f2b2 Don't leak "fallback" transducer for each label
a53ea7f7 Strip server junk from username reply
f66229d2 Fixed wrong exception name
3894ab1d Make composition with OL throw a Mismatch exception, which compose is looking for (it still needs to know about the second tranducer being OL)
20290174 add 'using std::set' to fix compilation problem in hfst-affix-guessify.cc
78a820aa Standardized the transducer format options of commandline tools.
544cc06e MAde ComposeIntersectRulePair::get_transitions virtual.
5c6e7e45 made size_t size(void) a const method.
de981e74 Added method ComposeIntersectFst::get_symbol_number, because HfstTropicalTransducerTransitionData::get_number is not accessible from outside the library.
86d4582f Edited Makefile.am so that it installs compose intersect headers properly
5330f224 Fixed bug in tokenizing string with lonely backslashes. Backslash can now escape any symbol and it is ignored when it occurs as the last symbol in the input.
719f6db6 Updated Rules - left replace down karttunen + tests
16dc6a63 Whitespace, punctuation, cleaned double conjunction.
2c63a890 Updates on building on the Mac.
f5d1d510 iterators can no longer be used...
921c6ab8 Update SFST requirements
00217c45 Resurrect the affix guesser maker
6c5a5fb7 Flood protection for channel messages
3cf20f7d Improved message finding and private messaging
25184514 autoconf doesn't understand dnl without space?
72f9a369 apparently mac os x "readline" does not support rl_completion_matches...
c1c9e07e HfstTransitionGraph::iterator is now protected, the public HfstTransitionGraph::const_iterator should be used instead.
894ef7be Replaced all mentions of HfstBasicTransducer::iterator with HfstBasicTransducer::const_iterator in OtherSymbolTransducer.cc.
75f41583 Changed syntax of negative context rules.
e7bbb193 Modified OtherSymbolTransducer::harmonize_diacritics, so HfstBasicTransducer::add_transition is used instead of adding transitions directly to the transition vector.
5bf71cbf Enabled sfst tests.
9712ab83 untabify recent changes
5a7c0e88 Updated HFST so that it works with the newest version of SFST, 1.4.6.
feee3425 detab hfst3 tree - feel free to revert (and report) if this causes any problems
fb6ae1c5 add few trivial features to list
1084eabe Be more verbose about type mismatch
4e99e3c8 check for empty strings
53ba386e Modified test script so that it runs tests for all backend formats. Disabled tests for sfst and log-openfst formats, since sfst segaults and log-openfst gives weird results. These are errors from the library.
495b3070 Added test for negative rule contexts in rules with variables.
251fd1d3 Added negative context rules.
f6a78c28 Added tests for negative contexts in twolc rules.
22e60a9f Now HfstTransducer::operator= throws a FunctionNotImplementedException instead of a TransducerHasWrongTypeException.
6fe21130 Fallback to anything or nothing from settings; resolves bug #1061990
33233250 kind reminder about importance of the README
c7e5b9fe Some logging changes
25d4d189 end readme in nl
25d328a4 Save misses to logfile
bd2ec42b Send notice if word is unknown
29b3b9c3 Don't print ugly weights
f67e60c1 Just dump privmsg to console and don't do anything for now..
3bec1509 hfst-bot demo
ba26d48e Lookup demo
e36e7002 Full api and distutils-based build & install
453163e0 clarify SFST libraries relation to HFST; thanks to spectie for the suggestion
7803cb8e * Save temporary files for foma in /tmp/ * import old readline UI from 2nd branch * deprecation warning if input = stdin * error if output = stdout
a03e3a27 Another readline editline libedit fix maybe
bd5472b8 CHECK_DECLs to support mac os X's fake cripple readline
2ce7d2d2 Tests of replace functions now work for transducers of foma type as well.
0b8ab2ce A new function rules::replace_down_karttunen added. This function returns the actual downward replacement rule transducer instead of rules::replace_down that returns a downward rule transducer as defined in SFST-PL.
b113008b New target for root makefile: valgrind
aaf07342 >=autoconf-2.62 not 2.22
b942a23f Remove libtool 1 support tricks, more trouble than it's worth
86718c20 Mark version requirements in autogen
c5094f14 Move version requirement to beginning of the line to be sure...
95bd5252 debugging tool
063de32f A bug in function random_path fixed
77c7bedb help messages to message_out instead of stderr
a9faa385 Do not cut unknown word forms from first non-alphabetic character in analysis, test case added
700c106f Fixed a piece of documentation on HfstTransitionGraph and added a test case too.
e9f9a79b Now extract_random_paths works for an empty tropical transducer. A test case also added.
89e4743c print more than fstinfo only if verbose
0b38d0b0 Fix arc-based leak in conversion
68c0d397 Make old lexc default lexc again; fixes: 100000000 Mac OS X bugs
637ea552 allow options to be passed via env.var. HFST_OPTIONS
3cf23c83 works but leaks
77a1c0da and default to glib to test if end users are ready for that in next rel
20c88c93 Release changes
6ccfd1c1 delete created test files
ec441723 include new proc tests in dist
25d65308 extra dist m4 macros
ce67ad2c fix how compound words are generated
6c780612 try generating first without compound interpretations; possible fix for bug #3290585
b71cb2e6 do not pipe error messages from tests to /dev/null
a8b07206 fix some -a tests
267a212e add valgrind to stress test targets
4af5dca4 Add a blurb about Unicode support requirements
b2f03ac6 calculate spaces in length of print forms; fixes #3296854
0bd58aa3 Allow use of glib for proc unicode casing (most likely breaks all Macs because of pkg-config and all that)
06620843 Implement type mismatch error messages for all binary tools
dec54f25 add quotation and neat printing in twolc tests
6048824f incompatible format messages in composition
f43dfaf8 document new tests in troubleshooting with examples.
dc5c1c50 Move latin-1 string tests to XFAIL
c30f3d37 Neat error output when incompatible formats; bug #3287458
eb6e0ed9 Troubleshooting for Mac OS X's getopt
0325c9f1 Check for GNU compatible getopt, try to fallback to broken getopts more aggressively on Mac OS X
23223aff Implement print-sigma and acceptor
4a279dde include new command line option in man pages
8e4177d5 Disable reserved characters and escapes in raw mode.
3b5d30f2 remove roundtrip test: mac os x sed does not support new lines
6ce1996e hfst-lexc around lexc_read
df3b87ee more alphabet hacks, i should really fix this properly as opposed to just patching the bugs i find
83fe1009 New analysis mode for finntreebank people: * no lowercase * no uppercases * no compounds * probably some ohter things
c46b8e80 Detail all external backend libraries in troubleshooting section also...
332fb8dc here here
0a94bbc7 possibly hippu-compatible alias installation
917ae736 Now HfstTransducer::substitute(StringPair, HfstTransducer) does not modify the argument transducer. A test for this case also added.
795926d1 Provide preliminary autoconf scripts for HFST
52f08847 Replaced HfstFastTransducer with HfstBasicTransducer in conversions. This might make HFST slower, but it is needed until the bugs in alphabet handling are fixed.
2fd9cfbe When performing transducer conversion through HfstFastTransducer, symbols leaked from one alphabet to another. Now HfstBasicTransducer is used instead and a test case is added that will fail if HfstFastTransducer is used instead. The original bug is to be fixed at some point.
9779c9b5 Added a test case for replace_up with FOMA_TYPE that fails. Currently it is commented so that make check will pass, but it will be fixed soon.
12dfacfa Now binary operations HfstTransducer::disjunct etc. do not modify the argument transducer.
f933bbb9 HfstTransducer::insert_to_alphabet and HfstTransducer::remove_from_alphabet now pass the tests.
25fd77f5 Added function HfstTransducer::remove_from_alphabet. Currently throws FunctionNotImplementedException because it does not pass all tests.
5447d492 verbose test failures
1085c4f8 Documentation updated.
c6d0e148 Removed check for null-termination of string in the test of string_mainpultion.cc, because it fails on Mac.
49f1dff1 Now distcheck passes the tests but still complains about att files..
e8842434 Reverted r1418 - these files are supposed to be here, and 'make clean' is over-cleaning.
e7725b41 Deleted supposedly generated files - they were removed by the make clean target in test/.
62254319 assert fopen != NULL
cf16a6f6 Modified compose-intersect, so that it will treat only flg diacritics not found in the alphabet of the rules as epsilons.
80c2c7ab A small bug fixed in the newest version of foma in the foma svn. Now the FomaTransducer test works correctly.
ea2f6198 Code reorganized. There seems to be some errors in openfst's minimization, temporarily fixed the situation by pushing weights before minimization.
5aca45b4 Retain root joiners while applying morphotax
afa9e112 kill some warnings by settings
16eabd97 New NEWS
ae3d1932 New releasenum (3.1.1) and libvector (6:0:0)
aa7bf174 Changed the test script so that it removes temp.twolc.hfst0, when it's done
97268c20 Fixed typo where $SRCDIR wasn't getting assigned to the file we want to test
077da455 Changed [[ ... ]] in regular expression center rules to <[ ... ]> to avoid introducing syntax errors in previously working twolc-grammars.
7d9a60d1 Added new tests in Makefile.am.
97963cf5 Added support for rules which have centers that are regular expressions of pairs.
9ebc8b9f Added tests for rules with regular expression centers.
b583a0cc Improved test script so it won't exit normally if hfst-twolc crashes.
e358dbec The public and protected interface of HfstTransitionGraph revised. Unnecessary comments and debugging code removed from HfstTransducer.
e1ef196f class HfstTransition moved to file 'HfstTransition.h'
147b99d8 Substitute functions in HfstTransitionGraph should now be easier to read.
add8e685 Code of HfstTransitionGraph made clearer, a bug in including HfstTransitionData.h fixed.
dc9da7bb Code simplified.
4ad9718b Transducer conversion done through HfstTransitionGraph.
f23ba679 Included <algorithm>...
3856fdc3 Changed member const HfstBasicTransducer &t to HfstBasicTransducer t in ComposeIntersectFst
96e5e02a Fixed bug in ComposeIntersectFst which forced all states in a transducer to have the same final weight as the initial state.
6b74eb4b Sorting the arcs of the argument transducer in the constructor of ComposeIntersectFst, because they need not be sorted nowadays.
0a5bcef5 Added arc_sort member function to HfstTransitionGraph.
2f9d3a99 Some additional experimental container-hacking layer stuff
e238723a left arrow rules
f49a6e12 some more hacks to alphabet.cc
6a3b6422 Added support for rule centers which have the form 'X:a' where 'X' is a set and 'a' is a symbol.
972561a7 Added tests for rules with centers 'X:a' where 'X' is a set and 'a' is a symbol.
475b6f90 Undid some changes that stopped working after the interface of HfstBasicTransducer apparently changed. Also changed HfstTransducer OtherSymbolTransducer::get_transducer(void) to const and added the function void OtherSymbolTransducer::get_initial_transition_pairs(SymbolPairVector &pair_container) const.
48487f20 Made max() and is_subset() static functions.
83cfb8a8 hfst-twolc unit tests added.
3bdda4e9 New NEWS
d04a96b8 New release number, 3.1.0 release and 5:0:0 libvector
420e273d Removed an extra debugging print
c23e9199 HfstTransducer::insert_freely(HfstTransducer&) now uses HfstFastTransducer
1bc69629 check locale in some tools
f0633660 HfstTransducer::insert_freely now uses HfstFastTransducer for foma type. More tests also added for insert_freely.
54900958 Conversion functions between HfstFastTransducer and backend implementations added. Now HfstTransducer::convert also converts through HfstFastTransducer. HfstFastTransducer will gradually replace HfstBasicTransducer for HfstTransducer functions that require conversion.
79722bf1 HfstTransitionGraph now has a vector of states instead of a set. All programs and functions updated accordingly.
3b908de2 Reword slow lookup warning
ccad335b An apparently functional way to expose sets by copying them in order to vectors
8708944a Fixed typo
92ef2989 Fixed includes in XreCompiler.h and LexcCompiler.h
8fa24816 added some functions to ConversionFunctions that will be used in fast conversion between transducer types instead of HfstConstantTransducer
0aea7c99 Fixed test script so that it should work in distcheck
6ee3d240 Made lookdown functions consistent with lookup ones, updated docs
1c9307d5 Updated doxygen to reflect changes in lookup functions
5f283556 simple archive tool for duplicating repeats
a43ca610 Added a faster version of lexc compiler for a transducer of foma type.
228feee6 decrease stress by order of 8
b15d21d4 Keep information about the parent FdTable of an FdState in the class to avoid invalid reads if it gets deleted
153862dc Changed lookup functions to return pointer to HfstOneLevelPaths, and take no argument for it. Calling functions are (still) responsible for freeing memory.
577e62cf Add more common problems
cd64166d Added HfstTransducer::get_symbol_number, it will be used in faster conversion between transducer formats.
5588a3b4 A small bug in testing the EmptyStringException fixed.
bda6a257 EmptyStringException is now thrown if the empty string is used as a transition symbol.
6923dc2c Universal pair function added
92ece0da Need more inclusions and usings when they don't leak in from backend libraries
ed3be664 Pathnames of testfiles in test/libhfst are now given as getenv("pwd") + "/tesfilename"
a5e5bceb Tests in test/libhfst now create the test files in EXTRA_DIST, so that distcheck works. This is a temporary solution.
7c1efd37 Remove obvious memory leaks
e8bb1685 Added missing test files.
2241e2d2 Prepare 3.0.3
552dadff New NEWS
0bf8b35d LexC and regexp functionalities moved from tools/src/parsers under the HFST interface at libhfst/src/parsers.
f13146ea Link with libhfst/src/libhfst.la (is this portable?)
5c8d2130 allow debianised names for libraries and headers
348aaed2 Global wrapper - could it be this easy?
e8d88709 Fetch headers from srcdir instead and assume libhfst is installed in the sense of ldconf
34f8ddf4 Fetch headers and libraries from a (presently) hardcoded /usr/local
81cd53b3 Testing a simple swig module
9fb915e2 SWIG generates files with .cxx extensions
52949a7a Removed debugging message
a3e829a5 Initial framework
f08623eb stress test for proc
99c1d62a Optional stress tests now
09baeffc Move stress tests out
8dada878 Priority union function and unit tests
e095ef18 Starting to implement a faster conversion, added file HfstFastTransitionData.h for that purpose. Also separated HfstTropicalTransducerTransitionData into its own file.
50deff4c HfstTropicalTransducerTransitionData moved to a separate file from HfstTransitionGraph.h.
0d2bda51 The pathname of HfstDataTypes.h fixed in HfstTransitionGraph.h
13e58055 A non-existing file removed from Makefile and an include command.
b91dcd87 Including HfstTransducer.h and HfstTransitionGraph.h into header files avoided in most cases by using HfstDataTypes.h instead.
fbcb6904 Made some changes to header files in order to avoid cyclic dependencies.
3b54b9d4 A bug in TropicalWeightTransducer::determinize fixed. The weights must not be encoded, only the label pairs.
f723fba3 Typo in license conflict warning
d23279bb Now the subtraction of two empty transducers works also for SfstTransducer. A test case also added for all transducer implementations.
e6549a36 Now read_in_att_format and hfst-txt2fst also accept a single newline as an empty transducer, if that is the only transducer in a stream.
373523de The library now uses is_epsilon, internal_epsilon etc. instead of string literals.
ceb19424 Added pkgconfig definitions
06fbbe02 Fixed variable LIBHFST_VERSION
40fde0e2 HfstTransducer::get_alphabet declared public. Now the foma implementation returns also special symbols that are not included in the sigma of foma, but foma is still aware of them. Some unit tests also added.
9069af7f Added some unit test for SfstTransducer that test if the alphabet is pruned in operations.
6d7f356d Fixed possible bug in ComposeIntersectUtilities and the find member function of SpaeSavingSet.
d3170ff4 Put stl sort function in stl namespace (which for unknown reasons is apparently unnecessary but it's probably clearer this way).
2d5c8d57 Added more test cases for hfst-subtract that test if the identities and symbols are handled right. Now subtraction seems to work for SfstTransducer, too.
430ab83d Print --version and --help to stdout in standard style
faa9758d clean generated scripts
c5381c74 Remove all binaries and texts after tests
693e2d65 remove generated files on expected failures
ded09b11 remove generated files after testing
dd1143eb Skip broken aliases
da774c3f exempt scripts and tools with broken option parsing from distcheck
014ad55e Use $srcdir in tests and include all sources
26aeef94 Missing text file reconstructed by guessing
6aedb926 Now input and output projections keep all symbols in the alphabet and have unknowns replaced with identities. Subtraction of tropical transducers handles weights correctly, all paths in the second transducer are subtracted from the first one.
41d2454c Comment out failing test prereq for now (it broke make dist).
9e698264 Unit tests reinstated for .cc files under implementations, compose_intersect forced under same regimen (whether MAIN_TEST is #defined).
6d8743b5 Dummy unit tests for every .cc file in libhfst/src except HfstExceptionDefs.cc
693540fa Tests added for hfst-lookup with transducers that contain identitites.
0dc5bd22 Document openfst version requirements in troubleshooting also
c7a37f1d An error in harmonization fixed: now flags are excluded when expanding unknowns. A test also added that checks this case.
b5c65bdb Missing lexc test cases
b9c7075b Reinstate final weights (they've apparently been absent since the missing state -relabeling code was added...)
7677a39f Added overloaded lookup functions passing untokenized strings
f354be54 print verbosely tokenization and blocked flags unless ol
faeb874d use GNU getopt if available
32488297 declare more built sources
008db774 More weight tests
e772be3b More sanity checks for escaped colons
c68e38bf Give weighted example
38440f9c partial fix for 3269636
0cc35699 remove debug printout
70af6e02 filenames and line numbers in error reporting again
339413e7 Build obsolete lexc compiler by default: * remove handbuilt lexicon combinatorics and revert to sfcm2009 algo * use pretty name for lexicons * more verbose printing instead of xerox style again
a75d492a Now hfst-twolc rule names are unquoted.
72ca5757 Add missing files with likely sources
5f924abb Now composition should work correctly
2b052f53 Changed population of symbols sets in HarmonizeUnknownAndIdentitySymbols.cc so that it relies on the get_alphabet method in HfstTransitionGraph. get_alphabet sometimes returns an empty set, so that needs to be fixed in order for unknown and identity symbols harmonization to be reliable.
492ddf3d Added HarmonizeUnknownAndIdentitySymbols.{cc,h} to Makefile.am
08d99b5c Unknown and identity symbols should now work correctly in composition. Still some errors in other tools in test/tools, trying to get them pass all.
ec358ec9 HarmonizeUnknownAndIdentitySymbols.{h,cc}: A model for how identity and unknown symbol harmonization can be done...
8ab38b48 New NEWS for 3.0.2 rerelease
0bcd861a Added missing ComposeIntersectUtilities.cc
d5d1a14f Added missing ComposeIntersectUtilities.h to Makefile.am
e8c4fd68 Prepare 3.0.2 release
2e151c79 s/warning: Warning/warning: /
d108e8d3 Moved the #ifdef HAVE_SFST a bit higher..
a9645b34 Fix ; separated regexps: * add compatibility function for mac OS X missing getdelim * use delim for all cases in regexp2fst
bdc8a29a Converting to and from SFST format is now faster because function 'check_visited' is used instead of a set of visited states.
32a31617 Conversions between tropical, sfst and foma transducers are now carried out through HfstConstantTransducer, which is faster than using HfstBasicTransducer.
893c4eb4 Make fallback dependent on fnie
25b8f736 Removed some unused code..
8711cbc3 Reduced hfst-compose-intersect memory consumption a bit.
0d9c1668 Added hfst-pair-test for testing hfst-twolc rule files.
7594c1af HfstConversionTransducer renamed to HfstConstantTransducer, most functions in ConvertTransducerFormat.cc splitted to separate files
bcc537fb Some optimizations done on conversion functions. Class HfstConversionTransducer added, it will be soon used in a more efficient conversion.
4e74b655 Note that we have licence exception for foma and warn if foma is missing
3fd96ede Epsilon filter compose intersect
77f97482 Only ascii-tokenize a-zA-Z to avoid breaking eg. generating transducers
3155440d 3.0.1 release news
24b0883b Prepare bugfix release 3.0.1
4d7a04d5 Use library function to check for epsilon in to_ol
78f7e603 Pass strings to internal representation checking functions by reference
6a5608cf More control over floor lift behaviour
aeee89fa Clean up conflict resolution mess
47f38236 Better floor jumping + relabel non-contiguous states (could slowdown in extreme cases, possibly revisit)
eaa606b6 Alias hfst-optimized-lookup -> hfst-lookup for compatibility
04d7b57f use default names to trace hfst operations neatly
f6fecd54 Allow name truncation for neater names
af009065 describe input/output formats in man pages
2b9fa098 Extend example to contain all sections and stuff --Denn och nedanstående rader kommer inte med i loggmeddelandet--
da36afc3 New tool for preprocessing automata before optimizing
8bcd08b3 A state in an HfstTransitionGraph now has a vector of transitions intead of a set of transitions.
eeb7acc9 ...I meant HfstException
0a73c66f Catch HfstFatalException in convertion
e6a8e3d2 HFST_CATCH macro
456a02ed hfst-lookup now handles flag diacritics correctly
35ea903b Added functions to check for equality to internal reserved strings
9f4d70bb a bug in printing epsilons in hfst-fst2strings for sfst transducers fixed
ab54f89e Add custom error message and README blurb for ::malloc errors
618d064d When the floor sticks for (even) a single round, jump up to most recent successful allocation. This makes things a lot faster, at the cost of some waste of space. --quick should not be necessary for most purposes now.
ff86e493 hfst-fst2strings now implements option --random for sfst and openfst transducers
bcc68cd8 Misc. speed improvements, in particular removed the need to separately track which indices begin states
17007d89 Actually write the index table finality markers
f436f9a6 see previous commit message
9d863c29 Fixed oopsie wrt removed profiling variables
51049ce3 Run twolc tests in numerical order.
b9bdc86b Fixed problem with word boundries breaking the alphabet.
5234e0e9 Added tests for ensuring that word boundary conversions wonät break the alphabet.
b532c930 Changed word boundaries so that they perhaps emulate what xerox does. A "#" in a rule file is always compiled to both an absolute word boundary (which is an internal symbol in hfst-twolc) and a relative word boundary "#". The absolute word boundary will always correspond to zero in output strings and the relative one will correspond to whatever it corresponded to in the rule file.
f9788b07 Word bounadry tests added.
d797484d Major improvements to --quick and some speed cases
0062cd44 option --random added to hfst-fst2strings, not yet implemented
724121b8 Removed trailing whitespace that broke make
c45ba6b5 Fix lexc tests: * make script in build dir +x * put script dir in beginning of PATH in tests
6621d1c2 git-svn-id: svn://svn.code.sf.net/p/hfst/code/trunk@1181 941e2c2b-deac-454f-805a-451daa25f33c
b4d68dfc Missing files added to Makefiles, make distcheck still fails.
fface6c6 Turned option passing back on for convert function
a31e269e test-script fixed. Previously it required installed hfst-compare, hfst-txt2fst and hfst-fst2txt
94b2b830 fix partial commit of portability c xfunctions
71b3bcde remove unnecessary read
30912710 use pretty names by default
c122b04a Fix name tool installation
45f3fdaf Use automata names from metadata instead of filenames
a3e6bcd7 Fixed a bug with %-escaped variable values, which weren't properly unescaped previously.
39651a46 Added tests for %-escaped variable values.
f59a7e9c hfst-fst2strings help message and special character printing fixed
584c57c6 hfst-fst2strings fixed
3c0acb5e filter flags should've defaulted true
7e4b6cd0 Typo in usage message
8f464a94 Added some files for a transducer testing environment.
c53f6055 fix spacing in examples and describe
28680a9f update version
38108b8c "environment" variables fed to configure get saved
c35977bd Added note about local libraries
8077e3ee Fixed a bug in compiling <=>-rules with list centers.
97dd6fc2 Added test for compiling <=>-rules with list centers.
c748ca34 Changed target to first release version
35fb2218 hfst-lookup should now work correctly
5d4421d0 Added deprecated (and undocumented) -w switch to hfst-calculate for backwards compatibility
d8842d41 Update substitute documentation
d4c7e58d hfst-lookup should now work, still some problems in printing strings
d7773da2 A more organized version of lookup function added to hfst-lookup. It will soon replace the original function.
f850e58a Don't expect hfst-lexc-compiler to have a proper help and version string because it's just a humble wrapper.
09adc46e Added a way for ol-conversion to be done really lazily (for when you want it done quickly - should still be faster, honestly). Also to pass a string parameter to conversion functions if they ever want a hint about something.
d4f4c94e Don't run obsolete tests that break when run after make clean
252e225f more distcheck fixes
53d54a45 small changes made to hfst-lookup
200b383a Added single quotes around and space after the filename for file-not-found errors. Makes it easier to read the filename (earlier the cause of the error was printed flush after the filename, with no space between).
63b19be6 be explicit explicit about foma requirement for lexc and xfst
9b1f9c2f Only claim to convert files that test -f
df545ae4 Small changes made to helps and arguments in tools that convert between text, string and binary formats.
65bebfdc Maybe INFINITY will be defined more portably this way
a9183a25 INFINTY in hfst-lookup was not defined properly. fixed it.
b97ed62b string::npos changed to std::string::npos
105b4b37 Generate binaries needed to test hfst-proc correctly
65b46021 More utf-8 checks added. Command line tool helps also updated.
2342e28f Fixed Makefile.am so that make dist works.
59e6347b Initialise weight
3f70a6dc Missing std qualifier
49eaf2b7 Added HfstTokenizer utf-8 validation tests.
4991081a Enlightened hfst-proc as to the true nature of HFST3 headers
de50e344 made HfstTokenizer::check_utf8_correctness a public static
1850f9f2 Added utf-8 validation to the tokenizer and added the exception IncorrectUtf8CodingException.
6361b293 Fix brokentests
3a4ce96d remove coding style requirements and unit tests since they aren't followed anyways
77529606 Do not print all multicharacter symbols unless in verbose mode
206bef06 Update man pages
92a99245 mark 3.0_rc1
3d3e70be Fixed a bug in compiling rules with word boundaries and added test45.
7ff82000 Don't use sed in scripts since it's broken on MacOS
cb7b1ee2 Fixed bug in compose_intersect which discarded the second rule systematically.
948b3721 @_SPACE_@, @_TAB_@ and @_COLON_@ added to HfstStrings2FstTokenizer.cc
7ad47d77 hfst-lookup now uses HfstFst2StringsTokenizer to tokenize the input strings.
9ab65922 Don't convert same stack more than once
b9dbb3ab fisx some warnings
fa5dd096 workaround mac''s missing strndup
6870c098 Allow pair substitutions...
01d746d8 Fix test for Mac OS
677dddef Replace tabs with random number of spaces
0990ea4a More tests limited to available formats
8ee702b9 Only test available automata formats
96af3b2d Test all string formats in strings2fst
834cf29e Fixed hfst-string2fst parsing of input strings and reading of multichar symbols.
e1880818 hfst-strings2fst should now be able to handle all kind of string formats.
ecc96f2b # is no longer a special symbol, because it is not a special symbol in xerox twolc.
2ca9026a Fixed print in left arrow rule conflict warnings.
3d41fc23 Fixed word boundaries in compose-intersect, again...
b4270dd1 Special symbol handling changed in the commandline tools.
0e1bea82 Fixed word boundaries in compose_intersect. When HfstBasicTransducer substitute starts working correctly, this can be done more efficiently.
161d9032 Fixed word boundaries in compose_intersect.
f507cee7 Added printing the conflicting context in left a row rule conflict warnings.
2851f19b Unknown and identity symbols are now correctly handled in composition. AT&T format now allows spaces in symbols if they are ascaped as '@_SPACE_@'
caaf1098 Tests for ? symbols
a57fcaad Use unknowns and identities in regexp2fst
a6c98345 Documentation added. Exceptions now throw a message.
dfceeeed Modified test so that the script does not have to be changed when new tests are added.
71f1022e Removed a unnecessary newline printed when not reporting left arrow conflicts.
6b311ec8 Added a rule for compiling a local version of hfst-twolc used in tests.
d08533d7 Added hfst-twolc tests.
e032ace0 Added hfst-twolc tsts.
98a54028 Added hfst-twolc tests.
aacbe751 Cahnged order of arguments so that 'cat fst1 | hfst-binary-tool fst2' will treat fst1 as the first and fst2 as the second argument.
262d19eb Fixed compose-intersect
1bb5dec7 Handle the empty transducer in ->ol
34d35770 Now hfst-compare does not print anything in silent mode. More tests added to test/libhfst.
ea109120 Fixed definitions
232b892a Fixed d:0 where d is a diacritic.
58836d2a Fixed symbol offset calculation in certain flag diacritic-concerning cases
d2a766d4 Fixed problem with diacritic pair syntax. Now d d: and d:0 are equivalent for the diacritic d and d:x will give a warning.
a5505dc6 Remove debugging strings
0590acdc Properly copy flag state vector on stack
25a5b9ed Fixed a bug with definitions
32e01ece Add tests for UTF-8 and latin-1 encoded strings
ef669e3d follow print-epsilons in all formats
9331c511 format options like other tools have them
5b8b2b15 More documentation added.
6197f2fe More careful quotation and informative error messages for foma scripts
5d6a2ffa Use "HfstTransducer.hh" instead of <hfst/HfstTransducer.hh> for in tree builds
49db47a5 hfst-strings2fst fixed
6bf1608e Now hfst-strings2fst should handle escaping correctly.
2825f8da A bug in hfst-strings2fst fixed: now the backslash is reserved for escaping characters. Now the library also throws more informative exceptions.
54aaa9ec Fixed operation retrieval
1a0485aa Fixed problem wrt subtracting from an end() iterator of an empty state_placeholder index placeholder map
85d0602c Removed some more debug prints.
fe95ee70 Removed some debugging prints
28844b5b Fixed bug in defining lista using other lists and another bug which prevented variable rules unless there was a Definitions section.
613d39f6 Fixed diacritics handling.
86c83356 Fixed output file handling.
103d9b23 Gave up on making }, ] and ) literal if they are not preceeded by opening {, [ and [, since this is too difficult in flex when brackets get nested. Fixed commandline argument handling properly and separated { and } from [ and ] i.e. they get their own bison symbols.
52ebbb7c Fixed handling of commandline paramters so that the input filename can be given as a free argument without -i or --input.
d630caf9 Fixed syntax so that a lonely }, ] or ) is not interpreted as abracket, parenthesis and so on. Fixed numbers so that they need not be quoted unless preceeded by ^
ade6ed80 nore quotation and script guessing
60c80232 Added hfst-compose-intersect.cc
2b7a964f Changes needed for libhfst/src/implementations/compose_intersect
5718b7b6 Added fiels needed in compose_intersect
803e57e4 configure.ac changed because hfst-twolc was added.
51b9cd2d Removed file hfst-twolc/.txt_fst which was added by mistake.
cfa3c5ce Removed geerated dir hfst-twolc/src/.deps which was added by mistake
38ec61e1 Added hfst-twolc.
e195be67 Miscellaneous packing tricks & fixed a typo
78152220 Correction to flag indexing
51cdc7ac Corrected alphabet ordering (eps-input-flag-other)
a07a6e43 Small changes done in documentation.
f8658bb0 More refactoring
20e8d361 Miscellaneous improvements and refactoring
4e9475fe Fixed bug wrt finding the largest used index
859a1b27 missing test file added
b4ecec5d Parametrized packing aggression. Packing is still a little slow for decent levels of aggression. Consider adding a mechanism to ask conversion to be maximally aggressive (eg. for binary distribution).
fcaea641 Redo flag handling in ol-conversion
dfb26903 A bug in hfst-lookup fixed: reading input strings in spaced format now works.
7856fc99 Fix stupid mistake in previous fix to optimized-lookup internals
2f321110 More coding style conformism
bc68f4ce Coding style conformism
7d828ce8 Fixes to optimized-lookup internals and conversion
d53894f2 Exception classes refined
844d8816 All exceptions changed to HFST_THROW(HfstException).
66ebc7df Renamed TWROW HFST_THROW in HfstExceptionDefs.h
661b615d Added some macro definitions, which make it easy to declare, define and throw informative exceptions.
829953ab Free memory earlier and do some less work in ol-conversion
3c709019 The order of weight and string/string pair changed in HfstOneLevelPath and HfstTwoLevelPath. Now the elements of a set are sorted in ascending weight order.
44f919c2 HfstLookupPath(s) is now replaced with HfstOneLevelPath(s)
ae4f393d Flag diacritics should now be handled right in functions lookup and extract_strings
95eca273 HfstTransducer::lookup now takes a StringVector as argument and HfstTransducer::extract_strings stores its results in an 'HfstTwoLevelPaths'
38c5e2b2 Now coding-style.sh should pass.
7470a480 Overlong lines separated on several lines.
525715b8 More more more OFST->OPENFST renaming
bd8523a7 More more OFST->OPENFST renaming
1a562676 more OFST -> OPENFST name renaming
645f7af9 HfstTransducer::read_in_att_format is now protected
4b37099c A bug in tests fixed.
b1067964 test.cc renamed to test_rules.cc and only tests for functions in namespace hfst::rules preserved
708daa6a TROPICAL_OFST_TYPE and LOG_OFST_TYPE changed to TROPICAL_OPENFST_TYPE and LOG_OPENFST_TYPE.
4a1fd2ff Fixed a bug in reading binary SFST and FOMA transducers.
80e53b48 update requirements and reorganise
44a7d7ab Fixed bug in reading and writing transducer binary headers.
d9ac6ecf AT&T format now accepts all whitespace characters as field separators.
6fd24d8d Added a const-version of HfstTransitionGraph::operator[], so that it is possible to access the transitions of a const HfstTransitionGraph.
3cae2e6a Added appropriate type checks in a HfstTransducer constructor - probably many more needed...
af09e6fb Fixed typo and clarified help message
ce45fb53 Represent epsilon as blank string when returning representations from ol-library
13f7061f More tests added. Substitute bugs fixed in TropicalWeightTransducer.
5632573b New tests added. Substitution functions cause a segfault in TropicalWeightTransducer, debugging under way.
2c7c03e8 More tests added.
91ab1092 Fixed some problems (due to epsilon symbol) in SfstTransducer functions that take a String or StringPair as argument. FomaTransducer::is_cyclic now works correctly, function 'fsm_topsort' must be called before 'is_loop_free'.
2ea66592 More tests added.
ebfd3f97 File test_transducer_functions fixed.
47066b22 Extra debugging messages removed.
e48223df Test for tokenizer added.
8ff3b25a FomaTransducer::define_transducer(char*) now calls fsm_count(struct *fsm) on its return value. In this way the number of fsm lines is correct when the transducer is written to file.
8485a6dd More tests added.
10901e1d git-svn-id: svn://svn.code.sf.net/p/hfst/code/trunk@925 941e2c2b-deac-454f-805a-451daa25f33c
a68b3e3c git-svn-id: svn://svn.code.sf.net/p/hfst/code/trunk@924 941e2c2b-deac-454f-805a-451daa25f33c
0df13e98 File test_constructors.cc added
e1e8ffef Documentation added.
7e81b7b8 Documentation updated. Tool hfst-name added.
ab09facb Now conversion from OpenFst and SFST to HFST handles special symbols correctly. How special symbols should be handled in the reverse case is still an open question.
aaa59d31 remove deprecated conversions and unused parameters
8cbe4a23 ol-basic -conversion
b3a201a2 Function HfstTransducer::insert_freely(const HfstTransducer &tr) added.
63eb0309 Now the output symbol table of a tropical or log HFST transducer is assigned a copy of the input symbol table when the transducer is written. This should make conversions between OpenFst and HFST transducers easier.
473157e5 Fixed two typedefs in trunk/hfst3/tools/src/hfst-summarize.cc
c8c1af8e Fixed an #include in trunk/hfst3/libhfst/src/implementations/ConvertTransducerFormat.h and a typedef in trunk/hfst3/libhfst/src/implementations/convert.h
25fc8e1a Fix verbose messages about HFST3 headers
e856b97a epsilon tests
0e91640d hfst-lookup now prints the results in the FILE requested instead of stderr
758a9565 hfst-lookup has two options more. It is possible to print results in pairstring format (not implemented for optimized lookup format) and defined how the epsilon is printed.
76acf98f extract_strings and WeightedPath changed so that it is possible to include a StringPairVector representation of the extracted paths
d17c308a implement flags and epsilons and xfst options print-space print-flag obey-flags etc.
290b8612 Avoid testing missing backends if possible
843e8d61 Documentation added. Thrown exception types changed for some functions.
ed4c9afc Documentation improved and added. Keyword 'const' added to functions and their arguments when appropriate.
532d9e90 Documentation added.
026a0956 Documentation changed.
a76fccfa HfstFileException changed to HfstStreamException
740219f4 Missing flag diacritics now yield a warning in hfst-compose, hfst-conjunct and hfst-subtract, unless freely inserted by option -F.
fbd97a15 One more portability fix... could be the last one!
200fae13 Yet another without-openfst portability fix
da6dfc85 Another without-openfst portability fix
beae8209 Another without-openfst portability fix
b8f670a9 Added using-directives to hfst-lookup.cc - apparently they don't get included from somewhere else when not HAVE_OPENFST, so this is necessary for portability
e9a2fa57 More optimized-lookup conversion fixes... produces correct results at least for trivial cases now.
37717b2a Various bits of format correctness for optimized-lookup conversion (lookup still doesn't give correct results)
1e51ff91 Documentation added.
c929f0b8 HfstAlphabet, HfstBasic and HfstUtf8 cc and h files moved to tools/src. Some functions in HfstTransducer decalred as protected.
3e5198b9 #if HAVE_HFST_OL perhaps not working correctly - in any case HFST_OL-writing is now enabled, although the results still don't appear to be quite correct
70441e4d Oops, forgot to remove some debugging messages
c930115b Fixed hanging hfst-ol conversion, still doesn't get written
4b3123b6 Ol-conversion (untested as yet)
7f0965a4 Added parsers/xre_utils.h to noist_HEADERS in tools/src/Makefile.am
52ba7b8d Added parsers/XreCompiler.h to noinst_HEADERS in tools/src/Makefile.am
0382dde3 Documentation added.
32100129 Documentation added.
94820cdd Intersecting composition functionalities removed, they will be implemented using HfstBasicTransducer format in the future.
90253d69 Updated configure.ac to reflect dropping of tools/src/parsers/Makefile.am
081c986f Deleted unneeded Makefile.am in tools/src/parsers
aa3fcaad Substituting unknowns and identities now works correctly in composition.
3e96e8ff A bug in reading transducers in AT&T format fixed. Also an alphabet bug in converting HfstBasicTransducers into SFST transducers fixed.
7c9ed801 Corrected documentation link.
7bd61dfb Documentation added.
b03365dd Redundant code removed from class HfstTransitionGraph.
ee7689ac HfstNet renamed to HfstTransitionGraph, HfstFsm to HfstBasicTransducer and HfstArc to HfstBasicTransition
9c3b6a60 extra_dist all scripts
b930429f fix VPATH _build_
324cd91d Substitute functions of HfstNet updated.
366b7839 HfstCompiler.h in noinst_HEADERS
d545f778 Added items to noinst_HEADERS in tools/src/parsers/Makefile.am, evidently needed for building
150715f9 Remove intermediate files and be selectively verbose to avoid confusion
e2e9e79a YAUH: HfstUtf8.h
4c75eabe YAUH: HfstBasic.h
f0e1f0c4 Another unshipped header (HfstGrammar.h)
c9d913a2 oops
b258f1f5 HAVE_OPENFST guards
ab00fb47 A bug in transducer conversion functions fixed.
ab4c8404 Missing \ in another header shipping directive in libhfst/src/implementations/Makefile.am
8bb83a4a Another missing header, TropicalWeightComposeIntersect.h included for shipping (this is probably temporary and the header shouldn't be needed anymore, but for now building fails without it on clean machines)
558f4f35 The alpha testing instructions
cc748ccf skip all tests that might fail
1ac0e16a * some new autoconfiscate features (silent make!) * misc. stylistic changes
e8b5ac08 Made HfstNet::get_final_weight(...) a const member function.
eafc2e8c Added missing headers to Makefile
24277857 changed void HfstNet::add_state(HfstState s) so that it returns s and added HfstState HfstNet::add_state(void), which adds a new state.
51d9a4af HfstNet.cc added
6468e7fa Conversions between transducer formats updated so that they use HfstFsm
236c20d5 Fix the missing includes and linkage
5a74f1f1 Remove hfst's lexc, xfst and xre parsers and use foma's instead of reinventing wheel and doubling update burden
f569bf39 New transducer conversion functions added.
b1b2c7c2 git-svn-id: svn://svn.code.sf.net/p/hfst/code/trunk@841 941e2c2b-deac-454f-805a-451daa25f33c
569ec24e A bug in FomaTransducer::define_transducer fixed.
bb4aa6b2 A bug fixed in HfstNet::substitute(StringPair, HfstNet)
7b4516ee More functions added to HfstNet
c80a3412 More functions added to HfstNet
59c60c3c Use local compare for all tests
4babf628 don't fclose infile too early
f4ecf799 * Separate bug report from further info by newline * remove old and deprecated tools
7db67bb3 * intersect has been called conjunct since forever * move external links in SEE ALSO
c49c6c73 enforce coding style in make check :-)
a5774a96 File new-ConvertTransducerFormat.cc added, it will soon replace ConvertTransducerFormat.cc
3c754389 testing results of parameter vcombinations
1c76cc0e add hfst-lexc script
1fe743b0 Update xfst/foma scripts
9259bbe9 Added a substitute function to class HfstNet.
ea060127 HfstNet.h has now a function write_in_att_format
a0e38e29 HfstNet.h updated and tested.
692777ca Added file HfstNet.h that will soon replace file HfstInternalTransducer.h.
8bd82472 Fix internal tests
a988bf90 HfstInternalTransducer_ updated.
d511822b A new HfstInternalTransducer class HfstInternalTransducer_ added.
5105bb2f class HfstTransition changed to struct HfstTransition
de7d5ce4 A bug in converting empty transducers between internal and OpenFst's formats fixed.
b2b1ee87 use #if's on backend specific code
75bc715f LogFstTrieFunctions .cc and .h removed and their functions moved to LogWeightTransducer .cc and .h and copied also to TropicalWeightTransducer .cc and .h
e5cacc1a Files renamed and deleted.
19e3b71d Including headers should now work correctly. Doxygen documentation also added.
871e1e0f The minimal lexc compiler functionality tests and minor fixes to compilation formula
4d00374a implementations unit tests
4233b9d5 Add unit tests for extensions
e7213c65 don't test default input stream since it requires stdin
135c50f5 EXIT_* needs cstdlib on newere gcc's/glibc's
f0084381 main interface unit tests 3/6
7923d45e It's a verbose test suite
3fdf3eaf hfst-strings2fst now tokenizes multicharacter symbols
4280623c A bug in hfst-strings2fst with (not pairstrings && not spaces) fixed.
dc4b60bc hfst-strings2fst now uses the internal trie transducer format and converts to the format requested before writing the output. This should make hfst-strings2fst faster.
178b960b Documentation added.
39f65532 Added a simple example transducer library, MFSTL,  that can be included by command './configure --with-mfstl=true'
37fa1735 An implementation type FOO_TYPE added for testing, by default it is not included in HFST.
779e27e3 Fix lookup_printf and test it
91cbdcaa Reading hfst version 2 transducers whose implementation type is OpenFst should work now.
72481314 Documentation added. A bug in hfst version 2 transducer handling detected, fixing started.
8a0f6926 Full-ish HfstTransducer unit test
f65f2285 A bug in hfst-strings2fst fixed, now the newline character is discarded before converting the weight string into a float. More Doxygen documentation also added.
76ebbd1a Lookup implemented for optimized-lookup API with new style headers, though hfst-lookup doesn't print anything due to commented-out printing function (it seems to break when I uncomment it..?)
606ddf69 This could be unit test
8887df69 hfst-lookup should now work correctly for transducers having epsilon input cycles. Also added some documentation.
da33a15a fix more of mac os x getline substitute
b88d9ff5 skip hfst-proc dir for empty input tst
b41e5743 Revert man bootstrapping stuff and add stubs for rest of the tools
539ef76f add set and use zu for size_t prints
100123f8 hfst-lookup fixed, still allows too many cycles although limits the results to a finite number
0aeebd28 A bug in hfst-lexc-compiler fixed, still some errors in help2man and tool tests.
1e978756 File added.
b185125e Outputs of help commands of commandline tools fixed.
12ff3ef0 A bug in HfstTransducer(StringPairVector&, type==FOMA_TYPE) fixed.
edbb55d3 A version of infinitely_ambiguous (necessary for at least lookup)
d7f8ba0e Removed conditional compilation guards from non-template lookup-performing function (which way do we want it to be?)
d9efbeee enable lexc, missing ) added
64c686b8 Some additional HFST_OL functionality
75934b3a Update man pages for release :-)
2dc49a31 Tests added.
209fd043 Some fixes to make the right names defined down the line, plus a first attempt at the lookup API
39692d3c Lookup for hfst_ol in hfst3
4887fe93 More documentation added.
07b48388 Files documented, a skeleton file added for adding a new backend implementation
28316461 Files renamed
cc616921 git-svn-id: svn://svn.code.sf.net/p/hfst/code/trunk@769 941e2c2b-deac-454f-805a-451daa25f33c
185380b1 Now libhfst/src/implementations contains files that modify backend transducers directly and libhfst/src other files
a98715af Files moved
636599b1 Files moved
a373e2db Set executable bit + line endings.
fcf17ac0 Fake hfst-xfst using foma-wrapper hacks
21581874 HfstCompiler.cc and HfstCompiler.h added to tools/src/
485406cb Files moved.
b7d01a85 hfst-lookup updated, handling flag diacritics and multiple paths with the same weights not yet implemented
bccb155d hfst-lookup no works on unweighted acyclic transducers
565d9908 set type after exiting error for mac compiler
4cba1aa0 Fix -o handling
a62adc65 hfst-lookup implementation continued, works not yet
edd084b5 beginning to implement hfst-lookup
cb1cd036 Recover from make-breaking stupid mistake
2b402e56 Standardization of HfstOlTransducer's iostreams
b10fc116 just testing that everything works
d20527d0 Clean up few minor details
d1968d9b Documentation added.
0fd33bd7 More documentation added, exceptions divided into subclasses.
f55820f5 Documentation added.
bac207d2 Catch stuff to provide meaningful error messages, provide better error replacement
7bec3d4b Small changes done, documentation added.
ddaf962a Lookup and related functions
94bb62d1 Update string_to_format and some polishing
3067f9fc A bug in reading transducers with old header format fixed.
8bde7d58 harmonization function optimized
29c3abc2 don't use bsd err functions
19df4c88 return paths even if !HAVE_SFST
cba810b1 using std::map, have SFST stuff ifdef'd
218faa62 A name field added to hfst header. Doxygen documentation updated.
7a3b4731 HfstInputStream changed
c7509d0f Missing restriction rule operators added.
fd80271e A bug in HfstRules fixed. FEXPORT must be added in front of foma functions 'sigma_create', 'xxmalloc' and 'sigma_max' in file fomalibconf.h until new version of foma is released.
1a306b54 A bug in HFST header handling fixed. Todo: update HFST code to work with the newest version of SFST.
42e25943 Fix lexc compiler
8739b075 Detail GNU bison version and feature requirements
801ad208 Use obsolete syntax for name-prefix to support slightly older bisons
2d9b05b8 More checks for backend installation sanity§
da681d11 Make compiler tools dependent on relevant features in library
a6f4b681 Also include current foma diffs
322b7f80 Function open() removed from HfstInputStream and HfstOutputStream. HFST transducer header structure changed.
5d197d0d Added MacOS X notes.
6532861d Remove bundled foma and SFST
bf5053ed Mergefail→delete duplicate lines
cba481f2 Add preliminary xfst compiler for offline compilation of static files
ea740e3c Add autogen.sh for problematic oses and systems
2ce19e80 Document library backend requirements
d911ae35 add replacement strndup and error_at_line for mac again
dbcfbf10 commandline tool hfst-compose-intersect and test added
45290c5d more tests added to tools, SFST_TYPE transducer header now has an optional field that tells if the transducer is minimal
0db6a0b7 more tests added to test/tools and small bugs fixed in commandline programs
ce7f006a HfstInputStream and HfstOutputStream now work correctly with tropical and log weight transducers if their file argument is an empty string
b36fd197 Alphabet definition in HfstCompiler is now not done through internal format when using SfstTransducers. This optimization makes hfst-calculate slightly faster.
aaa2eae0 tropical transducer harmonization code optimized
5421e3d9 Moved the #endif to the actual correct place...
3a950b76 Moved #endif to correct place. Suggestion: when there's a lot of distance between #if and #endif, consider placing a comment after #endif to indicate what it's ending (emacs can't figure it out).
8ab00a76 Removed obsolete definitions
15369a72 transducer harmonization code optimized
bf6dfa1f git-svn-id: svn://svn.code.sf.net/p/hfst/code/trunk@692 941e2c2b-deac-454f-805a-451daa25f33c
7eb10ec4 missing files added
3d244cbe git-svn-id: svn://svn.code.sf.net/p/hfst/code/trunk@690 941e2c2b-deac-454f-805a-451daa25f33c
3bc47057 HfstCompiler's code optimized
0a590ffb git-svn-id: svn://svn.code.sf.net/p/hfst/code/trunk@685 941e2c2b-deac-454f-805a-451daa25f33c
dfd51435 replace_in_context's code optimized
0ba83666 git-svn-id: svn://svn.code.sf.net/p/hfst/code/trunk@683 941e2c2b-deac-454f-805a-451daa25f33c
2cad021b git-svn-id: svn://svn.code.sf.net/p/hfst/code/trunk@682 941e2c2b-deac-454f-805a-451daa25f33c
e1f264b4 Class 'HfstTrie' added for quick disjunction of words in hfst-calculate
41328b3e reading words from a file should now be faster in hfst-calculate
c973556d git-svn-id: svn://svn.code.sf.net/p/hfst/code/trunk@679 941e2c2b-deac-454f-805a-451daa25f33c
c075ba73 git-svn-id: svn://svn.code.sf.net/p/hfst/code/trunk@678 941e2c2b-deac-454f-805a-451daa25f33c
256cd2df a bug in hfst-calculate's 'make_mapping' fixed
f46a184f now hfst-calculate does not unnecessarily handle unknown symbols in composition
f2a53597 A bug in alphabet definition in hfst-calculate fixed. It seems that handling unknown and identity symbols in composition is very slow for some reason.
d194e80e A bug in HfstCompiler::complement_range fixed
a4899508 HfstCompiler::new_transducer is now faster
e74f90f7 some bugs in command line programs fixed
d78c45a1 some bugs fixed in tests and hfst library
2a64259d 4cats.txt now contains a transducer accepting 4 consecutive 'cat's, and 4_cats.txt contains 4 consecutive transducers that each accept 'cat'.
acd96ec8 a bug in function 'harmonize' fixed
553674c8 Commit missing inputNamed
d9f95515 hfst now compiles with all combinations of available backend libraries
c5e09736 git-svn-id: svn://svn.code.sf.net/p/hfst/code/trunk@661 941e2c2b-deac-454f-805a-451daa25f33c
68b326b9 Make --without-foma work
71d08c49 optimized-lookup bigfix wrt weights in flagged transducers
41a235e6 Function HfstTransducer::push_weights added. Foma is now dynamically linked to hfst, option -fvisibility must not be used when compiling foma, because then some of foma's functions will not be visible to hfst.
fbe29693 Tentatively implement HFST 2 format
9d4aeed6 move things around
917827d7 Add space separated strings2fst; parse - as filename in hfst_fopen
6c693093 hfst now works without making any changes in sfst files
11f54203 Configuring out broken parts
17cd8d38 Add preliminary lexc compiler
ab50a962 HfstMutableTransducer defined in namespace hfst, small bugs fixed in TropicalWeightTransducer
226ae4cc unknown and identity symbols are now handled correctly in composition
c016a92b HfstMutableTransducer replaced with HfstInternalTransducer
792dfb30 Document the XRE compiler
a2d03673 functions that read and write in AT&T format now use HfstInternalFormat
6f0770db some modifications made to HfstInternalTransducer
0d5c47d4 freely_insert and substitute functions now do not depend on any specific library. They must still be tested.
2bf97cb0 some functions added to HfstInternalTransducer
f26af421 invert XRE autoconf logic
1a8a586e missing files added
b2da1a4a conversions between internal transducers and sfst, tropical and foma transducers implemented
9b896d20 Enable XRE parser
058a4093 conversions between SFST and internal transducers added
66cdf07c summarize more statistics
3260a0a4 Wrap backends around autoconf have_
2c140fa3 Enable basic compose-intersect again; no flags, no #'s
ef39e843 HFST code updated so that it works with OpenFst version 1.2. New internal transducer format added.
47cdd70e fix signs when wno-sign-compare fails
00e079ba Delete generated files
4590274e Now hfst-calculate does not depend on SFST.
1a56a054 hfst-calculate now does not depend on SFST, function implementations not yet written
ea974ba0 a bug in substitute(StringPair, HfstTransducer) fixed, agreement variables added to hfst-calculate
90d62759 Add some functionality tests for hfst-proc
a884b276 Remove obsolete source files of hfst-proc predecessors
4ebe3484 Fix bug causing final weights not to be included in weight calculations
32abdeca Remove obsolete flag diacritics code from hfst-proc
b5a8fe8e Add license information to hfst-proc and optimized-lookup files
eeedf419 Modify the extract_strings algorithm to avoid the non-standard std::vector::data() for older GCCs
2cad922e Perform initial migration of hfst-proc into its new home in the proper HFST tools directory, while refactoring to share as much code as possible with the optimized-lookup library backend
0fc222ea Add 'exclude' option to hfst-fst2strings
af172ca5 git-svn-id: svn://svn.code.sf.net/p/hfst/code/trunk@616 941e2c2b-deac-454f-805a-451daa25f33c
c4577241 Fixed memory leaks in harmonize
fb3fab5b Added libhfst/src/implementations/TropicalWeightComposeIntersect.h and libhfst/src/implementations/TropicalWeightComposeIntersect.cc
ebef427e Composition can soon handle identity and unknown symbols correctly. Substitute can now also modify symbol pairs through a function pointer.
0207962b Refactor extract_strings to invoke a callback on paths, allowing neat features in hfst-fst2strings
25b747a4 Credit self in AUTHORS
fddd8b22 SFST updated to new test version, some modifications still needed to HFST library so that it will work with the new version
17125a48 error_at_line for parsing errors
5df88b53 Allow empty lines and comment lines starting with # except when followed by a tab.
b8da331a Add the basic summarize functionality and the test
5ff805ee rule operators added to hfst-calculate
3618b202 More functionlaity tests
1daa938e Reimplement extract_strings in all backends with cycle-limiting functionality
8ebdabee substitute and test
4c2697a2 Add test for fst2stings
3c0db1e0 more functions added to hfst-calculate
527b7cf5 Ubuntu (dash?) workaround
07c88321 Make string pair implementation work
5a1646a5 Add lexc compiler's old skeleton
768cc079 Reverse how the upper and lower strings are stored during string extraction in the SFST backend
d8228327 Implement string extraction in the optimized-lookup backend
3901e854 Fix a compiler warning about missing a return statement
e55cdabe Remove debug output that should never have been committed
5238401c Fix calculation of several header properties during conversion
7130a070 more functions added to hfst-calculate
35c3e57b Start a document on formats
dbf650c0 test strings2fst and fail
bc5f85db Rollback bogus repeat throw
23f95bcd comment out broken repeats
84d6b262 Fix bug in binary op parameter handling and add test for it
5f3b3172 Invert upper and lower to follow upside-down terminology of HFST
8cb21960 Fix no parameters parsing for unary tools and project
72cb4160 Test basic algebra
8ce84837 functionality test for concatenate
cbecff25 Add text based transducers for functionality tests
124df373 Fix fst2strings to use set_program_name and rest of the basic tools
893186f8 small changes done in functions that are needed for hfst-calculate
8e30ba10 Check handling of empty input file for all tools
11692b29 Default to -Wall -Wextra -Werror for developers
041c510f Convert txt2fst; * use const for global char-* * default stdin, stdout for global FILE*
58dff6eb Add initial support for evaluating and filtering flag diacritics when extracting transducer paths
90f49d95 Add a class for dealing with flag diacritics in string extraction
7f8204c8 Port and enable fst2strings
44154e0d Hook up is_cyclic for the optimized-lookup backend
74dcdfc6 Use warn and err if warnc, errc, error are not available
a425f2b2 Instead of gnulib-tool, handle gnulib manually
aaa4a7f4 Missing file
34e3ecb3 Gnulib weirdness?
075e63cc Convert some simpler command line tools to HFST3 (preliminary)
e535c052 Perhaps gnulib for portability problems? Temp. disabled few tools.
c49af70c Few sample tests for tools in automated tests
02a08edf Include licensing blurb in every single file since it's very important
721ee498 Update docs a bit
56b426af First steps towards unbundling backend libraries
caa77047 Remove debug output inadvertently left enabled
6296b078 Fix sign-problem causing format-guessing for OFST transducers to fail
9444698e Add support for converting to the HFST optimized-lookup format
39bc89ea Add support for the HFST-optimized-lookup implementation in the tool for format conversion
134c6ea2 Add support for the new implementation in the interface files
0fd24ab1 Modify the semantics of the InputStream function is_good in the OFST implementations to match those in the SFST and foma implementations
4883d03a Update makefile for building the new backend
ec67b18c Add conversion HFST-optimized-lookup -> OFST Tropical Weight
41d7301d Initial commit of bridge classes for the HFST-optimized-lookup backend
f8b849b6 Initial commit of an HFST-optimized-lookup backend
3532dca8 Add is_cyclic to the transducer API and implementations
532c6683 Add hfst-strip-header tool for removing any HFST3 headers from a file
c7a64f3e Port hfst-compare to the new library
96b6ca71 Pass the correct output type when creating the output stream
29dc4894 git-svn-id: svn://svn.code.sf.net/p/hfst/code/trunk@471 941e2c2b-deac-454f-805a-451daa25f33c
5834ac0d Port hfst-fst2txt
9d0ade84 Fix the already-ported tools to properly handle IO using the standard streams
5cc51002 Add ability to read the type of the first transducer in an HfstInputStream without first having created a transducer from it
ab1d96cc git-svn-id: svn://svn.code.sf.net/p/hfst/code/trunk@465 941e2c2b-deac-454f-805a-451daa25f33c
714fc13a HfstCompiler class added
7580d1d7 Fix fst2fst to correctly handle inputs with multiple transducers
e34e30c2 Teach HfstInputStream how to deal with inputs containing multiple transducers, of which none have HFST3 headers
50d56868 Port hfst-invert to HFST3
1a86ae31 Port hfst-fst2fst to work with HFST3
de1feba4 Add support in HfstInputStream for loading transducers lacking the HFST3 header
a5e7ec01 Make foma transducer type guesser able to detect non-gzipped transducers as well
7f785cff Add call to open the output stream so the tool doesn't crash
dc85825e Add static methods to the InputStream implementations allowing to check whether a FILE* or istream contains a transducer of a give type
1e6b6cd4 Remove HFST2 include from file with helper functions for tools
0ab49edd hfst-calculate files updated to hfst version 3. At the moment, hfst-calculate only parses single transition expressions and produces a dummy transducer as a result.
fbe8acb2 anonymous transducer constructors added
ef4f1c58 Added HfstTransducer &operator=(const HfstTransducer &)
e352b9ab hfst-compiler.yy added
544c4183 more rule operator functions implemented
6886485c git-svn-id: svn://svn.code.sf.net/p/hfst/code/trunk@409 941e2c2b-deac-454f-805a-451daa25f33c
13ec6820 'HfstRules.cc' added.
d72acf32 rule functions added
cfe73d09 stub for flag diacritic support doc
916c12d7 documentation added
dbd6d6ed more Doxygen documentation added
9759312e Convert binary integers to decimal for old gcc support (OS X)
0067fa6f Tried to make the autoreconf text a bit clearer.
7ded1ccc ImplementationType parameters removed from most functions
91ac6d0c Require fst/fstlib.h as <fst/fstlib.h> and include as such.
55b3eb58 * Remove openfst from source directory because bundling Apache licenced    software with GNU GPL softare is a bad thing * Add licence disclaimers to auxiliary files because FSF suggests so
c695e9af documentation added
49053036 Documentation and type handling added, some problems in type conversion.
7d5dbb0a add ofst include dir for compilation
f76eb453 sgi.h is visible outside libhfst
7bf66dd7 Work around problems with missing hash
c618bd71 Add extra doc stuff to build tree
df2f4126 more doxygen documentation added
8804a385 some typedefs in SymbolDefs.h and ExtractStrings.h renamed
a05ba5d3 copy hfst-proc verbatim to 3
ce06b138 Swap stable old trunk to branches and proposition to trunk