medialab / Minet

minet is a webmining CLI tool & library for python that can be used to collect and extract data from a large variety of web sources such as raw webpages, Facebook, CrowdTangle, YouTube, Twitter, Media Cloud etc.

It adopts a very simple approach to various webmining problems by letting you perform a variety of actions from the comfort of your command line. No database needed: raw data files such as CSV should be sufficient to do the work.

In addition, minet also exposes its high-level programmatic interface as a python library so you can tweak its behavior at will.

Shortcuts: Cookbook, CLI Docs, Lib Docs.

Use cases

• Downloading large amount of urls very fast. (guide)
• Writing scrapers to extract structured data from HTML pages.
• Writing crawlers to automatically browse the web.
• Extract raw text content from HTML pages. (example)
• Normalize batches of urls contained in a CSV file to perform relevant aggregations (dropping irrelevant query items, extracting domain name etc.) (example)
• Join two CSV files based on columns containing urls that need to be matched (guide).
• Collecting data from CrowdTangle API (to collect and search posts mainly from Facebook and Instagram).
• Collecting data from Facebook (comments, likes etc.)
• Parsing Facebook urls in a CSV file.
• Collecting data from Twitter (users, followers, followees etc.)
• Scraping data (tweets etc.) from Twitter's website public facing search API.
• Collecting data from YouTube (captions, comments, video metadata etc.)
• Parsing YouTube urls in a CSV file.
• Dumping a Hyphe corpus.
• Collecting data from Media Cloud (search stories, dump topics etc.).

Features (from a technical standpoint)

• Multithreaded, memory-efficient fetching from the web.
• Multithreaded, scalable crawling using a comfy DSL.
• Multiprocessed raw text content extraction from HTML pages.
• Multiprocessed scraping from HTML pages using a comfy DSL.
• URL-related heuristics utilities such as extraction, normalization and matching.
• Data collection from various APIs such as CrowdTangle.

Installation

minet can be installed as a standalone CLI tool (currently only on mac, ubuntu & similar) by running the following command in your terminal:

curl -sSL https://raw.githubusercontent.com/medialab/minet/master/scripts/install.sh | bash

Don't trust us enough to pipe the result of a HTTP request into bash? We wouldn't either, so feel free to read the installation script here and run it on your end if you prefer.

On ubuntu & similar you might need to install curl and unzip before running the installation script if you don't already have it:

sudo apt-get install curl unzip

Else, minet can be installed directly as a python CLI tool and library using pip:

pip install minet

If you need more help to install and use minet from scratch, you can check those installation documents.

Finally if you want to install the standalone binaries by yourself (even for windows) you can find them in each release here.

Upgrading

To upgrade the standalone version, simply run the install script once again:

curl -sSL https://raw.githubusercontent.com/medialab/minet/master/scripts/install.sh | bash

To upgrade the python version you can use pip thusly:

pip install -U minet

Uninstallation

To uninstall the standalone version:

curl -sSL https://raw.githubusercontent.com/medialab/minet/master/scripts/uninstall.sh | bash

To uninstall the python version:

pip uninstall minet

Cookbook

To learn how to use minet and understand how it may fit your use cases, you should definitely check out our Cookbook.

Documentation

• minet as a command line tool
• minet as a python library

Contributing

To contribute to minet you can check out this documentation.

How to cite

minet is published on Zenodo as

You can cite it thusly:

Guillaume Plique, Pauline Breteau, Jules Farjas, Héloïse Théro, & Jean Descamps. (2019, October 14). Minet, a webmining CLI tool & library for python. Zenodo. http://doi.org/10.5281/zenodo.4564399