1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
|
diff --git a/breadability.egg-info/PKG-INFO b/breadability.egg-info/PKG-INFO
deleted file mode 100644
index d0e4ff6..0000000
--- a/breadability.egg-info/PKG-INFO
+++ /dev/null
@@ -1,280 +0,0 @@
-Metadata-Version: 1.1
-Name: breadability
-Version: 0.1.20
-Summary: Port of Readability HTML parser in Python
-Home-page: https://github.com/bookieio/breadability
-Author: Rick Harding
-Author-email: rharding@mitechie.com
-License: BSD
-Description: breadability - another readability Python (v2.6-v3.3) port
- ===========================================================
- .. image:: https://api.travis-ci.org/bookieio/breadability.png?branch=master
- :target: https://travis-ci.org/bookieio/breadability.py
-
- I've tried to work with the various forks of some ancient codebase that ported
- `readability`_ to Python. The lack of tests, unused regex's, and commented out
- sections of code in other Python ports just drove me nuts.
-
- I put forth an effort to bring in several of the better forks into one
- code base, but they've diverged so much that I just can't work with it.
-
- So what's any sane person to do? Re-port it with my own repo, add some tests,
- infrastructure, and try to make this port better. OSS FTW (and yea, NIH FML,
- but oh well I did try)
-
- This is a pretty straight port of the JS here:
-
- - http://code.google.com/p/arc90labs-readability/source/browse/trunk/js/readability.js#82
- - http://www.minvolai.com/blog/decruft-arc90s-readability-in-python/
-
-
- Alternatives
- ------------
-
- - https://github.com/codelucas/newspaper
- - https://github.com/grangier/python-goose
- - https://github.com/aidanf/BTE
- - http://www.unixuser.org/~euske/python/webstemmer/#extract
- - https://github.com/al3xandru/readability.py
- - https://github.com/rcarmo/soup-strainer
- - https://github.com/bcampbell/decruft
- - https://github.com/gfxmonk/python-readability
- - https://github.com/srid/readability
- - https://github.com/dcramer/decruft
- - https://github.com/reorx/readability
- - https://github.com/mote/python-readability
- - https://github.com/predatell/python-readability-lxml
- - https://github.com/Harshavardhana/boilerpipy
- - https://github.com/raptium/hitomi
- - https://github.com/kingwkb/readability
-
-
- Installation
- ------------
- This does depend on lxml so you'll need some C headers in order to install
- things from pip so that it can compile.
-
- .. code-block:: bash
-
- $ [sudo] apt-get install libxml2-dev libxslt-dev
- $ [sudo] pip install git+git://github.com/bookieio/breadability.git
-
- Tests
- -----
- .. code-block:: bash
-
- $ nosetests-2.6 tests && nosetests-3.2 tests && nosetests-2.7 tests && nosetests-3.3 tests
-
-
- Usage
- -----
- Command line
- ~~~~~~~~~~~~
-
- .. code-block:: bash
-
- $ breadability http://wiki.python.org/moin/BeginnersGuide
-
- Options
- ```````
-
- - **b** will write out the parsed content to a temp file and open it in a
- browser for viewing.
- - **d** will write out debug scoring statements to help track why a node was
- chosen as the document and why some nodes were removed from the final
- product.
- - **f** will override the default behaviour of getting an html fragment (<div>)
- and give you back a full <html> document.
- - **v** will output in verbose debug mode and help let you know why it parsed
- how it did.
-
-
- Python API
- ~~~~~~~~~~
- .. code-block:: python
-
- from __future__ import print_function
-
- from breadability.readable import Article
-
-
- if __name__ == "__main__":
- document = Article(html_as_text, url=source_url)
- print(document.readable)
-
-
- Work to be done
- ---------------
- Yep, I've got some catching up to do. I don't do pagination, I've got a lot of
- custom tweaks I need to get going, there are some articles that fail to parse.
- I also have more tests to write on a lot of the cleaning helpers, but
- hopefully things are setup in a way that those can/will be added.
-
- Fortunately, I need this library for my tools:
-
- - https://bmark.us
- - http://r.bmark.us
-
- so I really need this to be an active and improving project.
-
-
- Off the top of my heads TODO list:
-
- - Support metadata from parsed article [url, confidence scores, all
- candidates we thought about?]
- - More tests, more thorough tests
- - More sample articles we need to test against in the test_articles
- - Tests that run through and check for regressions of the test_articles
- - Tidy'ing the HTML that comes out, might help with regression tests ^^
- - Multiple page articles
- - Performance tuning, we do a lot of looping and re-drop some nodes that
- should be skipped. We should have a set of regression tests for this so
- that if we implement a change that blows up performance we know it right
- away.
- - More docs for things, but sphinx docs and in code comments to help
- understand wtf we're doing and why. That's the biggest hurdle to some of
- this stuff.
-
-
- Inspiration
- ~~~~~~~~~~~
-
- - `python-readability`_
- - `decruft`_
- - `readability`_
-
-
-
- .. _readability: http://code.google.com/p/arc90labs-readability/
- .. _TravisCI: http://travis-ci.org/
- .. _decruft: https://github.com/dcramer/decruft
- .. _python-readability: https://github.com/buriy/python-readability
-
-
- .. :changelog:
-
- Changelog for breadability
- ==========================
-
- 0.1.20 (April 13th 2014)
- -------------------------
- - Don't include tests in sdist builds.
-
- 0.1.19 (April 13th 2014)
- --------------------------
- - Replace charade with chardet for easier packaging.
-
- 0.1.18 (April 6th 2014)
- ------------------------
- - Improved decoding of the page into Unicode.
-
- 0.1.17 (Jan 22nd 2014)
- ----------------------
- - More log quieting down to INFO vs WARN
-
- 0.1.16 (Jan 22nd 2014)
- ----------------------
- - Clean up logging output at warning when it's not a true warning
-
- 0.1.15 (Nov 29th 2013)
- ----------------------
- - Merge changes from 0.1.14 of breadability with the fork https://github.com/miso-belica/readability.py and tweaking to return to the name breadability.
- - Fork: Added property ``Article.main_text`` for getting text annotated with
- semantic HTML tags (<em>, <strong>, ...).
- - Fork: Join node with 1 child of the same type. From
- ``<div><div>...</div></div>`` we get ``<div>...</div>``.
- - Fork: Don't change <div> to <p> if it contains <p> elements.
- - Fork: Renamed test generation helper 'readability_newtest' -> 'readability_test'.
- - Fork: Renamed package to readability. (Renamed back)
- - Fork: Added support for Python >= 3.2.
- - Fork: Py3k compatible package 'charade' is used instead of 'chardet'.
-
- 0.1.14 (Nov 7th 2013)
- ---------------------
- - Update sibling append to only happen when sibling doesn't already exist.
-
- 0.1.13 (Aug 31st 2013)
- ----------------------
- - Give images in content boy a better chance of survival
- - Add tests
-
- 0.1.12 (July 28th 2013)
- -----------------------
- - Add a user agent to requests.
-
- 0.1.11 (Dec 12th 2012)
- ----------------------
- - Add argparse to the install requires for python < 2.7
-
- 0.1.10 (Sept 13th 2012)
- -----------------------
- - Updated scoring bonus and penalty with , and " characters.
-
- 0.1.9 (Aug 27nd 2012)
- ---------------------
- - In case of an issue dealing with candidates we need to act like we didn't
- find any candidates for the article content. #10
-
- 0.1.8 (Aug 27nd 2012)
- ---------------------
- - Add code/tests for an empty document.
- - Fixes #9 to handle xml parsing issues.
-
- 0.1.7 (July 21nd 2012)
- ----------------------
- - Change the encode 'replace' kwarg into a normal arg for older python
- version.
-
- 0.1.6 (June 17th 2012)
- ----------------------
- - Fix the link removal, add tests and a place to process other bad links.
-
- 0.1.5 (June 16th 2012)
- ----------------------
- - Start to look at removing bad links from content in the conditional cleaning
- state. This was really used for the scripting.com site's garbage.
-
- 0.1.4 (June 16th 2012)
- ----------------------
- - Add a test generation helper readability_newtest script.
- - Add tests and fixes for the scripting news parse failure.
-
- 0.1.3 (June 15th 2012)
- ----------------------
- - Add actual testing of full articles for regression tests.
- - Update parser to properly clean after winner doc node is chosen.
-
- 0.1.2 (May 28th 2012)
- ---------------------
- - Bugfix: #4 issue with logic of the 100char bonus points in scoring
- - Garden with PyLint/PEP8
- - Add a bunch of tests to readable/scoring code.
-
- 0.1.1 (May 11th 2012)
- ---------------------
- - Fix bugs in scoring to help in getting right content
- - Add concept of -d which shows scoring/decisions on nodes
- - Update command line client to be able to pipe output to other tools
-
- 0.1.0 (May 6th 2012)
- --------------------
- - Initial release and upload to PyPi
-
-Keywords: bookie,breadability,content,HTML,parsing,readability,readable
-Platform: UNKNOWN
-Classifier: Development Status :: 5 - Production/Stable
-Classifier: Intended Audience :: Developers
-Classifier: License :: OSI Approved :: BSD License
-Classifier: Operating System :: OS Independent
-Classifier: Programming Language :: Python
-Classifier: Programming Language :: Python :: 2
-Classifier: Programming Language :: Python :: 2.6
-Classifier: Programming Language :: Python :: 2.7
-Classifier: Programming Language :: Python :: 3
-Classifier: Programming Language :: Python :: 3.2
-Classifier: Programming Language :: Python :: 3.3
-Classifier: Programming Language :: Python :: Implementation :: CPython
-Classifier: Topic :: Internet :: WWW/HTTP
-Classifier: Topic :: Software Development :: Pre-processors
-Classifier: Topic :: Text Processing :: Filters
-Classifier: Topic :: Text Processing :: Markup :: HTML
diff --git a/breadability.egg-info/SOURCES.txt b/breadability.egg-info/SOURCES.txt
deleted file mode 100644
index d74b2a3..0000000
--- a/breadability.egg-info/SOURCES.txt
+++ /dev/null
@@ -1,44 +0,0 @@
-AUTHORS.txt
-CHANGELOG.rst
-LICENSE.rst
-MANIFEST.in
-README.rst
-setup.cfg
-setup.py
-breadability/__init__.py
-breadability/_compat.py
-breadability/annotated_text.py
-breadability/document.py
-breadability/readable.py
-breadability/scoring.py
-breadability/utils.py
-breadability.egg-info/PKG-INFO
-breadability.egg-info/SOURCES.txt
-breadability.egg-info/dependency_links.txt
-breadability.egg-info/entry_points.txt
-breadability.egg-info/not-zip-safe
-breadability.egg-info/requires.txt
-breadability.egg-info/top_level.txt
-breadability/scripts/__init__.py
-breadability/scripts/client.py
-breadability/scripts/test_helper.py
-tests/__init__.py
-tests/compat.py
-tests/test_annotated_text.py
-tests/test_orig_document.py
-tests/test_readable.py
-tests/test_scoring.py
-tests/utils.py
-tests/test_articles/__init__.py
-tests/test_articles/test_antipope_org/__init__.py
-tests/test_articles/test_antipope_org/test_article.py
-tests/test_articles/test_businessinsider-com/__init__.py
-tests/test_articles/test_businessinsider-com/test_article.py
-tests/test_articles/test_businessinsider_com/__init__.py
-tests/test_articles/test_businessinsider_com/test_article.py
-tests/test_articles/test_cz_zdrojak_tests/__init__.py
-tests/test_articles/test_cz_zdrojak_tests/test_article.py
-tests/test_articles/test_scripting_com/__init__.py
-tests/test_articles/test_scripting_com/test_article.py
-tests/test_articles/test_sweetshark/__init__.py
-tests/test_articles/test_sweetshark/test_article.py
\ No newline at end of file
diff --git a/breadability.egg-info/dependency_links.txt b/breadability.egg-info/dependency_links.txt
deleted file mode 100644
index 8b13789..0000000
--- a/breadability.egg-info/dependency_links.txt
+++ /dev/null
@@ -1 +0,0 @@
-
diff --git a/breadability.egg-info/entry_points.txt b/breadability.egg-info/entry_points.txt
deleted file mode 100644
index 6b5a31c..0000000
--- a/breadability.egg-info/entry_points.txt
+++ /dev/null
@@ -1,6 +0,0 @@
-[console_scripts]
-breadability = breadability.scripts.client:main
-breadability-2.7 = breadability.scripts.client:main
-breadability_test = breadability.scripts.test_helper:main
-breadability_test-2.7 = breadability.scripts.test_helper:main
-
diff --git a/breadability.egg-info/not-zip-safe b/breadability.egg-info/not-zip-safe
deleted file mode 100644
index 8b13789..0000000
--- a/breadability.egg-info/not-zip-safe
+++ /dev/null
@@ -1 +0,0 @@
-
diff --git a/breadability.egg-info/requires.txt b/breadability.egg-info/requires.txt
deleted file mode 100644
index 9345e99..0000000
--- a/breadability.egg-info/requires.txt
+++ /dev/null
@@ -1,3 +0,0 @@
-docopt>=0.6.1,<0.7
-chardet
-lxml>=2.0
\ No newline at end of file
diff --git a/breadability.egg-info/top_level.txt b/breadability.egg-info/top_level.txt
deleted file mode 100644
index bcc6dba..0000000
--- a/breadability.egg-info/top_level.txt
+++ /dev/null
@@ -1 +0,0 @@
-breadability
diff --git a/setup.py b/setup.py
index f41b868..b024941 100644
--- a/setup.py
+++ b/setup.py
@@ -82,7 +82,7 @@ setup(
"Topic :: Text Processing :: Filters",
"Topic :: Text Processing :: Markup :: HTML",
],
- packages=['breadability'],
+ packages=['breadability', 'breadability.scripts'],
include_package_data=True,
zip_safe=False,
install_requires=install_requires,
|
