bookieio / Breadability
Licence: bsd-2-clause
Reworked https://www.readability.com/ parsing library (now https://mercury.postlight.com/ is living alternative)
Stars: ✭ 186
Programming Languages
python
139335 projects - #7 most used programming language
Projects that are alternatives of or similar to Breadability
trafilatura
Python & command-line tool to gather text on the Web: web crawling/scraping, extraction of text, metadata, comments
Stars: ✭ 711 (+282.26%)
Mutual labels: text-mining, text-extraction
Lazynlp
Library to scrape and clean web pages to create massive datasets.
Stars: ✭ 1,985 (+967.2%)
Mutual labels: text-mining
Textfeatures
👷♂️ A simple package for extracting useful features from character objects 👷♀️
Stars: ✭ 148 (-20.43%)
Mutual labels: text-mining
Khcoder
KH Coder: for Quantitative Content Analysis or Text Mining
Stars: ✭ 126 (-32.26%)
Mutual labels: text-mining
Awesome Nlp
📖 A curated list of resources dedicated to Natural Language Processing (NLP)
Stars: ✭ 12,626 (+6688.17%)
Mutual labels: text-mining
Cogcomp Nlpy
CogComp's light-weight Python NLP annotators
Stars: ✭ 115 (-38.17%)
Mutual labels: text-mining
Nlp profiler
A simple NLP library allows profiling datasets with one or more text columns. When given a dataset and a column name containing text data, NLP Profiler will return either high-level insights or low-level/granular statistical information about the text in that column.
Stars: ✭ 181 (-2.69%)
Mutual labels: text-mining
Udpipe
R package for Tokenization, Parts of Speech Tagging, Lemmatization and Dependency Parsing Based on the UDPipe Natural Language Processing Toolkit
Stars: ✭ 160 (-13.98%)
Mutual labels: text-mining
Xioc
Extract indicators of compromise from text, including "escaped" ones.
Stars: ✭ 148 (-20.43%)
Mutual labels: text-mining
Qdap
Quantitative Discourse Analysis Package: Bridging the gap between qualitative data and quantitative analysis
Stars: ✭ 146 (-21.51%)
Mutual labels: text-mining
Datasciencer
a curated list of R tutorials for Data Science, NLP and Machine Learning
Stars: ✭ 1,727 (+828.49%)
Mutual labels: text-mining
Awesome Text Classification
Awesome-Text-Classification Projects,Papers,Tutorial .
Stars: ✭ 158 (-15.05%)
Mutual labels: text-mining
Awesome Hungarian Nlp
A curated list of NLP resources for Hungarian
Stars: ✭ 121 (-34.95%)
Mutual labels: text-mining
Tokenizers
Fast, Consistent Tokenization of Natural Language Text
Stars: ✭ 161 (-13.44%)
Mutual labels: text-mining
Scattertext
Beautiful visualizations of how language differs among document types.
Stars: ✭ 1,722 (+825.81%)
Mutual labels: text-mining
Hands On Natural Language Processing With Python
This repository is for my students of Udemy. You can find all lecture codes along with mentioned files for reading in here. So, feel free to clone it and if you have any problem just raise a question.
Stars: ✭ 146 (-21.51%)
Mutual labels: text-mining
Chemdataextractor
Automatically extract chemical information from scientific documents
Stars: ✭ 152 (-18.28%)
Mutual labels: text-mining
Texthero
Text preprocessing, representation and visualization from zero to hero.
Stars: ✭ 2,407 (+1194.09%)
Mutual labels: text-mining
Multi rake
Multilingual Rapid Automatic Keyword Extraction (RAKE) for Python
Stars: ✭ 162 (-12.9%)
Mutual labels: text-mining
breadability - another readability Python (v2.6-v3.3) port
.. image:: https://api.travis-ci.org/bookieio/breadability.png?branch=master :target: https://travis-ci.org/bookieio/breadability
I've tried to work with the various forks of some ancient codebase that ported
readability_ to Python. The lack of tests, unused regex's, and commented out
sections of code in other Python ports just drove me nuts.
I put forth an effort to bring in several of the better forks into one code base, but they've diverged so much that I just can't work with it.
So what's any sane person to do? Re-port it with my own repo, add some tests, infrastructure, and try to make this port better. OSS FTW (and yea, NIH FML, but oh well I did try)
This is a pretty straight port of the JS here:
- http://code.google.com/p/arc90labs-readability/source/browse/trunk/js/readability.js#82
- http://www.minvolai.com/blog/decruft-arc90s-readability-in-python/
Alternatives
- https://github.com/codelucas/newspaper
- https://github.com/grangier/python-goose
- https://github.com/aidanf/BTE
- http://www.unixuser.org/~euske/python/webstemmer/#extract
- https://github.com/al3xandru/readability.py
- https://github.com/rcarmo/soup-strainer
- https://github.com/bcampbell/decruft
- https://github.com/gfxmonk/python-readability
- https://github.com/srid/readability
- https://github.com/dcramer/decruft
- https://github.com/reorx/readability
- https://github.com/mote/python-readability
- https://github.com/predatell/python-readability-lxml
- https://github.com/Harshavardhana/boilerpipy
- https://github.com/raptium/hitomi
- https://github.com/kingwkb/readability
Installation
This does depend on lxml so you'll need some C headers in order to install things from pip so that it can compile.
.. code-block:: bash
$ [sudo] apt-get install libxml2-dev libxslt-dev
$ [sudo] pip install git+git://github.com/bookieio/breadability.git
Tests
.. code-block:: bash
$ pytest tests
Usage
Command line
.. code-block:: bash
$ breadability http://wiki.python.org/moin/BeginnersGuide
Options
```````
- **b** will write out the parsed content to a temp file and open it in a
browser for viewing.
- **d** will write out debug scoring statements to help track why a node was
chosen as the document and why some nodes were removed from the final
product.
- **f** will override the default behaviour of getting an html fragment (<div>)
and give you back a full <html> document.
- **v** will output in verbose debug mode and help let you know why it parsed
how it did.
Python API
~~~~~~~~~~
.. code-block:: python
from __future__ import print_function
from breadability.readable import Article
if __name__ == "__main__":
document = Article(html_as_text, url=source_url)
print(document.readable)
Work to be done
---------------
Yep, I've got some catching up to do. I don't do pagination, I've got a lot of
custom tweaks I need to get going, there are some articles that fail to parse.
I also have more tests to write on a lot of the cleaning helpers, but
hopefully things are setup in a way that those can/will be added.
Fortunately, I need this library for my tools:
- https://bmark.us
- http://r.bmark.us
so I really need this to be an active and improving project.
Off the top of my heads TODO list:
- Support metadata from parsed article [url, confidence scores, all
candidates we thought about?]
- More tests, more thorough tests
- More sample articles we need to test against in the test_articles
- Tests that run through and check for regressions of the test_articles
- Tidy'ing the HTML that comes out, might help with regression tests ^^
- Multiple page articles
- Performance tuning, we do a lot of looping and re-drop some nodes that
should be skipped. We should have a set of regression tests for this so
that if we implement a change that blows up performance we know it right
away.
- More docs for things, but sphinx docs and in code comments to help
understand wtf we're doing and why. That's the biggest hurdle to some of
this stuff.
Inspiration
~~~~~~~~~~~
- `python-readability`_
- `decruft`_
- `readability`_
.. _readability: http://code.google.com/p/arc90labs-readability/
.. _TravisCI: http://travis-ci.org/
.. _decruft: https://github.com/dcramer/decruft
.. _python-readability: https://github.com/buriy/python-readability
Note that the project description data, including the texts, logos, images, and/or trademarks,
for each open source project belongs to its rightful owner.
If you wish to add or remove any projects, please contact us at [email protected].