File: README.rst

package info (click to toggle)
python-tld 0.11.11-1
  • links: PTS, VCS
  • area: main
  • in suites: bullseye
  • size: 1,360 kB
  • sloc: python: 1,413; makefile: 20
file content (356 lines) | stat: -rw-r--r-- 7,744 bytes parent folder | download | duplicates (2)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
===
tld
===
Extract the top level domain (TLD) from the URL given. List of TLD names is
taken from `Mozilla
<http://mxr.mozilla.org/mozilla/source/netwerk/dns/src/effective_tld_names.dat?raw=1>`_.

Optionally raises exceptions on non-existing TLDs or silently fails (if
``fail_silently`` argument is set to True).

.. image:: https://img.shields.io/pypi/v/tld.svg
   :target: https://pypi.python.org/pypi/tld
   :alt: PyPI Version

.. image:: https://img.shields.io/pypi/pyversions/tld.svg
    :target: https://pypi.python.org/pypi/tld/
    :alt: Supported Python versions

.. image:: https://img.shields.io/travis/barseghyanartur/tld/master.svg
   :target: http://travis-ci.org/barseghyanartur/tld
   :alt: Build Status

.. image:: https://img.shields.io/badge/license-MPL--1.1%20OR%20GPL--2.0--only%20OR%20LGPL--2.1--or--later-blue.svg
   :target: https://github.com/barseghyanartur/tld/#License
   :alt: MPL-1.1 OR GPL-2.0-only OR LGPL-2.1-or-later

.. image:: https://coveralls.io/repos/github/barseghyanartur/tld/badge.svg?branch=master&service=github
    :target: https://coveralls.io/github/barseghyanartur/tld?branch=master
    :alt: Coverage

Prerequisites
=============
- Python 3.6, 3.7 and 3.8

`Support for Python 2.7 and 3.5`_ is available as well.

Documentation
=============
Documentation is available on `Read the Docs
<http://tld.readthedocs.io/>`_.

Installation
============
Latest stable version on PyPI:

.. code-block:: sh

    pip install tld

Or latest stable version from GitHub:

.. code-block:: sh

    pip install https://github.com/barseghyanartur/tld/archive/stable.tar.gz

Usage examples
==============
In addition to examples below, see the `jupyter notebook <jupyter/>`_
workbook file.

Get the TLD name **as string** from the URL given
-------------------------------------------------
.. code-block:: python

    from tld import get_tld

    get_tld("http://www.google.co.uk")
    # 'co.uk'

    get_tld("http://www.google.idontexist", fail_silently=True)
    # None

Get the TLD as **an object**
----------------------------
.. code-block:: python

    from tld import get_tld

    res = get_tld("http://some.subdomain.google.co.uk", as_object=True)

    res
    # 'co.uk'

    res.subdomain
    # 'some.subdomain'

    res.domain
    # 'google'

    res.tld
    # 'co.uk'

    res.fld
    # 'google.co.uk'

    res.parsed_url
    # SplitResult(
    #     scheme='http',
    #     netloc='some.subdomain.google.co.uk',
    #     path='',
    #     query='',
    #     fragment=''
    # )

Get TLD name, **ignoring the missing protocol**
-----------------------------------------------
.. code-block:: python

    from tld import get_tld, get_fld

    get_tld("www.google.co.uk", fix_protocol=True)
    # 'co.uk'

    get_fld("www.google.co.uk", fix_protocol=True)
    # 'google.co.uk'

Return TLD parts as tuple
-------------------------
.. code-block:: python

    from tld import parse_tld

    parse_tld('http://www.google.com')
    # 'com', 'google', 'www'

Get the first level domain name **as string** from the URL given
----------------------------------------------------------------
.. code-block:: python

    from tld import get_fld

    get_fld("http://www.google.co.uk")
    # 'google.co.uk'

    get_fld("http://www.google.idontexist", fail_silently=True)
    # None

Check if some tld is a valid tld
--------------------------------

.. code-block:: python

    from tld import is_tld

    is_tld('co.uk)
    # True

    is_tld('uk')
    # True

    is_tld('tld.doesnotexist')
    # False

    is_tld('www.google.com')
    # False

Custom TLD parsers
==================
By default list of TLD names is taken from Mozilla. Parsing implemented in
the ``tld.utils.MozillaTLDSourceParser`` class. If you want to use another
parser, subclass the ``tld.base.BaseTLDSourceParser``, provide ``uid``,
``source_url``, ``local_path`` and implement the ``get_tld_names`` method.
Take the ``tld.utils.MozillaTLDSourceParser`` as a good example of such
implementation. You could then use ``get_tld`` (as well as other ``tld``
module functions) as shown below:

.. code-block:: python

    from tld import get_tld
    from some.module import CustomTLDSourceParser

    get_tld(
        "http://www.google.co.uk",
        parser_class=CustomTLDSourceParser
    )

Custom list of TLD names
========================
You could maintain your own custom version of the TLD names list (even multiple
ones) and use them simultaneously with built in TLD names list.

You would then store them locally and provide a path to it as shown below:

.. code-block:: python

    from tld import get_tld
    from tld.utils import BaseMozillaTLDSourceParser

    class CustomBaseMozillaTLDSourceParser(BaseMozillaTLDSourceParser):

        uid: str = 'custom_mozilla'
        local_path: str = 'tests/res/effective_tld_names_custom.dat.txt'

    get_tld(
        "http://www.foreverchild",
        parser_class=CustomBaseMozillaTLDSourceParser
    )
    # 'foreverchild'

Same goes for first level domain names:

.. code-block:: python

    from tld import get_fld

    get_fld(
        "http://www.foreverchild",
        parser_class=CustomBaseMozillaTLDSourceParser
    )
    # 'www.foreverchild'

Note, that in both examples shown above, there the original TLD names file has
been modified in the following way:

.. code-block:: text

    ...
    // ===BEGIN ICANN DOMAINS===

    // This one actually does not exist, added for testing purposes
    foreverchild
    ...

Free up resources
=================
To free up memory occupied by loading of custom TLD names, use
``reset_tld_names`` function with ``tld_names_local_path`` parameter.

.. code-block:: python

    from tld import get_tld, reset_tld_names

    # Get TLD from a custom TLD names parser
    get_tld(
        "http://www.foreverchild",
        parser_class=CustomBaseMozillaTLDSourceParser
    )

    # Free resources occupied by the custom TLD names list
    reset_tld_names("tests/res/effective_tld_names_custom.dat.txt")

Support for Python 2.7 and 3.5
==============================
As you might have noticed, typing (Python 3.6+) is extensively used in the code.
However, Python 3.5 will likely be supported until it's EOL. All modern recent
versions (starting from `tld` 0.11.7) are fully compatible with
Python 2.7 and 3.5 (just works with ``pip install tld``).

**Install from pip**

.. code-block:: sh

    pip install tld

Development tips follow:

Python 2.7
----------
**Install locally in development mode**

.. code-block:: sh

    python setup.py develop --python-tag py27

**Prepare dist**

.. code-block:: sh

    ./scripts/prepare_build_py27.sh

**Run tests**

.. code-block:: sh

    tox -e py27

Python 3.5
----------
**Install locally in development mode**

.. code-block:: sh

    python setup.py develop --python-tag py35

**Prepare dist**

.. code-block:: sh

    ./scripts/prepare_build_py35.sh

**Run tests**

.. code-block:: sh

    tox -e py35

Testing
=======
Simply type:

.. code-block:: sh

    ./runtests.py

Or use tox:

.. code-block:: sh

    tox

Or use tox to check specific env:

.. code-block:: sh

    tox -e py38

Writing documentation
=====================

Keep the following hierarchy.

.. code-block:: text

    =====
    title
    =====

    header
    ======

    sub-header
    ----------

    sub-sub-header
    ~~~~~~~~~~~~~~

    sub-sub-sub-header
    ^^^^^^^^^^^^^^^^^^

    sub-sub-sub-sub-header
    ++++++++++++++++++++++

    sub-sub-sub-sub-sub-header
    **************************

License
=======
MPL-1.1 OR GPL-2.0-only OR LGPL-2.1-or-later

Support
=======
For any issues contact me at the e-mail given in the `Author`_ section.

Author
======
Artur Barseghyan <artur.barseghyan@gmail.com>