All Projects → shivam5992 → Textstat

shivam5992 / Textstat

Licence: mit
📝 python package to calculate readability statistics of a text object - paragraphs, sentences, articles.

Programming Languages

python
139335 projects - #7 most used programming language

Projects that are alternatives of or similar to Textstat

lorca
Natural Language Processing for Spanish in Node.js. Stemmer, sentiment analysis, readability, tf-idf with batteries, concordance and more!
Stars: ✭ 95 (-83.9%)
Mutual labels:  readability
Readability.php
PHP port of Mozilla's Readability.js
Stars: ✭ 280 (-52.54%)
Mutual labels:  readability
Percollate
A command-line tool to turn web pages into beautiful, readable PDF, EPUB, or HTML docs.
Stars: ✭ 3,535 (+499.15%)
Mutual labels:  readability
eslint-plugin-lodash-template
ESLint plugin for John Resig-style micro template, Lodash's template, Underscore's template and EJS.
Stars: ✭ 15 (-97.46%)
Mutual labels:  readability
SmartReader
SmartReader is a library to extract the main content of a web page, based on a port of the Readability library by Mozilla
Stars: ✭ 88 (-85.08%)
Mutual labels:  readability
Overview
中文编程的历史、现状和展望。issue 中进行相关问题的讨论.
Stars: ✭ 282 (-52.2%)
Mutual labels:  readability
gunning-fog
Formula to detect the ease of reading a text according to the Gunning fog index (1952)
Stars: ✭ 16 (-97.29%)
Mutual labels:  readability
Clean Mark
Convert an article into a clean text
Stars: ✭ 414 (-29.83%)
Mutual labels:  readability
Typographic Email
Responsive email template that is optimised for readability.
Stars: ✭ 268 (-54.58%)
Mutual labels:  readability
Elements Of Python Style
Goes beyond PEP8 to discuss what makes Python code feel great. A Strunk & White for Python.
Stars: ✭ 3,308 (+460.68%)
Mutual labels:  readability
readability-extractor
Javascript/Node wrapper around Mozilla's Readability library so that ArchiveBox can call it as a oneshot CLI command to extract each page's article text.
Stars: ✭ 18 (-96.95%)
Mutual labels:  readability
KaryScript
KaryScript is an experimental language to test the possibilities of a more readable textual language. It compiles to ES6 and can be considered a much better ECMAScript
Stars: ✭ 19 (-96.78%)
Mutual labels:  readability
Csharpformarkup
Use declarative style C# instead of XAML for Xamarin Forms UI
Stars: ✭ 302 (-48.81%)
Mutual labels:  readability
readability
readability for golang. 网页文章标题和正文抽取工具
Stars: ✭ 30 (-94.92%)
Mutual labels:  readability
Php Goose
Readability / Html Content / Article Extractor & Web Scrapping library written in PHP
Stars: ✭ 392 (-33.56%)
Mutual labels:  readability
hast-util-reading-time
utility to estimate the reading time
Stars: ✭ 55 (-90.68%)
Mutual labels:  readability
Graby
Graby helps you extract article content from web pages
Stars: ✭ 281 (-52.37%)
Mutual labels:  readability
Simpread
简悦 ( SimpRead ) - 让你瞬间进入沉浸式阅读的扩展
Stars: ✭ 5,352 (+807.12%)
Mutual labels:  readability
Midnight Lizard
Сustom color schemes for all websites
Stars: ✭ 406 (-31.19%)
Mutual labels:  readability
Elixir Scrape
Scrape any website, article or RSS/Atom Feed with ease!
Stars: ✭ 306 (-48.14%)
Mutual labels:  readability

Textstat

PyPI Build Status Downloads

Textstat is an easy to use library to calculate statistics from text. It helps determine readability, complexity, and grade level.

Photo by Patrick Tomasso on Unsplash

Usage

>>> import textstat

>>> test_data = (
    "Playing games has always been thought to be important to "
    "the development of well-balanced and creative children; "
    "however, what part, if any, they should play in the lives "
    "of adults has never been researched that deeply. I believe "
    "that playing games is every bit as important for adults "
    "as for children. Not only is taking time out to play games "
    "with our children and other adults valuable to building "
    "interpersonal relationships but is also a wonderful way "
    "to release built up tension."
)

>>> textstat.flesch_reading_ease(test_data)
>>> textstat.smog_index(test_data)
>>> textstat.flesch_kincaid_grade(test_data)
>>> textstat.coleman_liau_index(test_data)
>>> textstat.automated_readability_index(test_data)
>>> textstat.dale_chall_readability_score(test_data)
>>> textstat.difficult_words(test_data)
>>> textstat.linsear_write_formula(test_data)
>>> textstat.gunning_fog(test_data)
>>> textstat.text_standard(test_data)
>>> textstat.fernandez_huerta(test_data)
>>> textstat.szigriszt_pazos(test_data)
>>> textstat.gutierrez_polini(test_data)
>>> textstat.crawford(test_data)

The argument (text) for all the defined functions remains the same - i.e the text for which statistics need to be calculated.

Install

You can install textstat either via the Python Package Index (PyPI) or from source.

Install using pip

pip install textstat

Install using easy_install

easy_install textstat

Install latest version from GitHub

git clone https://github.com/shivam5992/textstat.git
cd textstat
pip install .

Install from PyPI

Download the latest version of textstat from http://pypi.python.org/pypi/textstat/

You can install it by doing the following:

tar xfz textstat-*.tar.gz
cd textstat-*/
python setup.py build
python setup.py install # as root

Language support

By default functions implement algorithms for english language. To change language, use:

textstat.set_lang(lang)

The language will be used for syllable calculation and to choose variant of the formula.

Language variants

All functions implement en_US language. Some of them has also variants for other languages listed below.

Function en de es fr it nl pl ru
flesch_reading_ease
gunning_fog

Spanish-specific tests

The following functions are specifically designed for spanish language. They can be used on non-spanish texts, even though that use case is not recommended.

>>> textstat.fernandez_huerta(test_data)
>>> textstat.szigriszt_pazos(test_data)
>>> textstat.gutierrez_polini(test_data)
>>> textstat.crawford(test_data)

Additional information on the formula they implement can be found in their respective docstrings.

List of Functions

Syllable Count

textstat.syllable_count(text)

Returns the number of syllables present in the given text.

Uses the Python module Pyphen for syllable calculation.

Lexicon Count

textstat.lexicon_count(text, removepunct=True)

Calculates the number of words present in the text. Optional removepunct specifies whether we need to take punctuation symbols into account while counting lexicons. Default value is True, which removes the punctuation before counting lexicon items.

Sentence Count

textstat.sentence_count(text)

Returns the number of sentences present in the given text.

The Flesch Reading Ease formula

textstat.flesch_reading_ease(text)

Returns the Flesch Reading Ease Score.

The following table can be helpful to assess the ease of readability in a document.

The table is an example of values. While the maximum score is 121.22, there is no limit on how low the score can be. A negative score is valid.

Score Difficulty
90-100 Very Easy
80-89 Easy
70-79 Fairly Easy
60-69 Standard
50-59 Fairly Difficult
30-49 Difficult
0-29 Very Confusing

Further reading on Wikipedia

The Flesch-Kincaid Grade Level

textstat.flesch_kincaid_grade(text)

Returns the Flesch-Kincaid Grade of the given text. This is a grade formula in that a score of 9.3 means that a ninth grader would be able to read the document.

Further reading on Wikipedia

The Fog Scale (Gunning FOG Formula)

textstat.gunning_fog(text)

Returns the FOG index of the given text. This is a grade formula in that a score of 9.3 means that a ninth grader would be able to read the document.

Further reading on Wikipedia

The SMOG Index

textstat.smog_index(text)

Returns the SMOG index of the given text. This is a grade formula in that a score of 9.3 means that a ninth grader would be able to read the document.

Texts of fewer than 30 sentences are statistically invalid, because the SMOG formula was normed on 30-sentence samples. textstat requires at least 3 sentences for a result.

Further reading on Wikipedia

Automated Readability Index

textstat.automated_readability_index(text)

Returns the ARI (Automated Readability Index) which outputs a number that approximates the grade level needed to comprehend the text.

For example if the ARI is 6.5, then the grade level to comprehend the text is 6th to 7th grade.

Further reading on Wikipedia

The Coleman-Liau Index

textstat.coleman_liau_index(text)

Returns the grade level of the text using the Coleman-Liau Formula. This is a grade formula in that a score of 9.3 means that a ninth grader would be able to read the document.

Further reading on Wikipedia

Linsear Write Formula

textstat.linsear_write_formula(text)

Returns the grade level using the Linsear Write Formula. This is a grade formula in that a score of 9.3 means that a ninth grader would be able to read the document.

Further reading on Wikipedia

Dale-Chall Readability Score

textstat.dale_chall_readability_score(text)

Different from other tests, since it uses a lookup table of the most commonly used 3000 English words. Thus it returns the grade level using the New Dale-Chall Formula.

Score Understood by
4.9 or lower average 4th-grade student or lower
5.0–5.9 average 5th or 6th-grade student
6.0–6.9 average 7th or 8th-grade student
7.0–7.9 average 9th or 10th-grade student
8.0–8.9 average 11th or 12th-grade student
9.0–9.9 average 13th to 15th-grade (college) student

Further reading on Wikipedia

Readability Consensus based upon all the above tests

textstat.text_standard(text, float_output=False)

Based upon all the above tests, returns the estimated school grade level required to understand the text.

Optional float_output allows the score to be returned as a float. Defaults to False.

Spanish-specific tests

Índice de lecturabilidad Fernandez-Huerta

textstat.fernandez_huerta(text)

Reformulation of the Flesch Reading Ease Formula specifically for spanish. The results can be interpreted similarly

Further reading on This blog post

Índice de perspicuidad de Szigriszt-Pazos

textstat.szigriszt_pazos(text)

Adaptation of Flesch Reading Ease formula for spanish-based texts. Attempts to quantify how understandable a text is.

Further reading on This blog post

Fórmula de comprensibilidad de Gutiérrez de Polini

textstat.gutierrez_polini(text)

Returns the Guttiérrez de Polini understandability index.

Specifically designed for the texts in spanish, not an adaptation. Conceived for grade-school level texts. Scores for more complex text are not reliable.

Further reading on This blog post

Fórmula de Crawford

textstat.crawford(text)

Returns the Crawford score for the text

Returns an estimate of the years of schooling required to understand the text. The text is only valid for elementary school level texts.

Further reading on This blog post

Contributing

If you find any problems, you should open an issue.

If you can fix an issue you've found, or another issue, you should open a pull request.

  1. Fork this repository on GitHub to start making your changes to the master branch (or branch off of it).
  2. Write a test which shows that the bug was fixed or that the feature works as expected.
  3. Send a pull request!

Development setup

It is recommended you use a virtual environment, or Pipenv to keep your development work isolated from your systems Python installation.

$ git clone https://github.com/<yourname>/textstat.git  # Clone the repo from your fork
$ cd textstat
$ pip install -r requirements.txt  # Install all dependencies

$ # Make changes

$ python -m pytest test.py  # Run tests
Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].