1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
|
=========================
Custom language detection
=========================
`dateparser` allows to customize the language detection behavior by using the ``detect_languages_function`` parameter.
It currently supports two language detection libraries out of the box: `fastText <https://github.com/facebookresearch/fastText>`_
and `langdetect <https://github.com/Mimino666/langdetect>`_, and allows you to implement your own custom language detection.
.. warning::
For short strings the language detection could fail, so it's highly recommended to use ``detect_languages_function``
along with ``DEFAULT_LANGUAGES``.
Built-in implementations
========================
fastText
~~~~~~~~
Language detection with fastText.
Import the fastText wrapper and pass it as ``detect_languages_function``
parameter. Example::
>>> from dateparser.custom_language_detection.fasttext import detect_languages
>>> dateparser.parse('12/12/12', detect_languages_function=detect_languages)
The fastText integration currently supports the large and the small models.
Find more information about `fasttext <https://fasttext.cc/blog/2017/10/02/blog-post.html>`__ models.
You can download your model of choice using ``dateparser-download``.
Downloading small model::
>>> dateparser-download --fasttext small
Downloading large model::
>>> dateparser-download --fasttext large
Deleting all cached models::
>>> dateparser-download --clear_cache
.. note::
If no model has been downloaded, the fastText wrapper downloads and uses
the small model by default.
langdetect
~~~~~~~~~~
Language detection with langdetect.
Import the langdetect wrapper and pass it as ``detect_languages_function``
parameter. Example::
>>> from dateparser.custom_language_detection.langdetect import detect_languages
>>> dateparser.parse('12/12/12', detect_languages_function=detect_languages)
.. note::
From some tests we did, we recommend to use ``fastText`` for faster and more accurate results.
Custom implementation
=====================
``dateparser`` allows the integration of any library to detect languages by
wrapping that library in a function that accepts 2 parameters, ``text`` and
``confidence_threshold``, and returns a list of the detected language codes in
ISO 639 standards.
Wrapper for boilerplate for implementing custom language detections::
def detect_languages(text, confidence_threshold):
"""
Takes 2 parameters, `text` and `confidence_threshold`, and returns
a list of `languages codes`.
* `text` is the input string whose language needs to be detected.
* `confidence_threshold` is a number between 0 and 1 that indicates the
minimum confidence required for language matches.
For language detection libraries that, for each language, indicate how
confident they are that the language matches the input text, you should
filter out languages with a confidence lower than this value (adjusted,
if needed, to the confidence range of the target library).
This value comes from the dateparser setting
`LANGUAGE_DETECTION_CONFIDENCE_THRESHOLD`.
The result must be a list of languages codes (strings).
"""
# here you can apply your own logic
return language_codes
|
