All Projects → luckylittle → blinkist-m4a-downloader

luckylittle / blinkist-m4a-downloader

Licence: MIT license
Grabs all of the audio files from all of the Blinkist books

Programming Languages

go
31211 projects - #10 most used programming language

Projects that are alternatives of or similar to blinkist-m4a-downloader

LeetCode
At present contains scraped data from around 1500 problems present on the site. More to follow....
Stars: ✭ 45 (-55%)
Mutual labels:  data-mining, scraper
Twitter Get Old Tweets Scraper
A data scraper for retrieving old tweets in Twitter using Python3.
Stars: ✭ 27 (-73%)
Mutual labels:  data-mining, scraper
Instagram-Comments-Scraper
Instagram comment scraper using python and selenium. Save the comments into excel.
Stars: ✭ 73 (-27%)
Mutual labels:  data-mining, scraper
Colly
Elegant Scraper and Crawler Framework for Golang
Stars: ✭ 15,535 (+15435%)
Mutual labels:  scraper, spider
perke
A keyphrase extractor for Persian
Stars: ✭ 60 (-40%)
Mutual labels:  data-mining, data-processing
website-to-json
Converts website to json using jQuery selectors
Stars: ✭ 37 (-63%)
Mutual labels:  data-mining, scraper
Ferret
Declarative web scraping
Stars: ✭ 4,837 (+4737%)
Mutual labels:  data-mining, scraper
Not Your Average Web Crawler
A web crawler (for bug hunting) that gathers more than you can imagine.
Stars: ✭ 107 (+7%)
Mutual labels:  scraper, spider
unpaprd
An audiobook 🎧 📔 app made using Flutter
Stars: ✭ 73 (-27%)
Mutual labels:  books, audiobooks
Dataflowjavasdk
Google Cloud Dataflow provides a simple, powerful model for building both batch and streaming parallel data processing pipelines.
Stars: ✭ 854 (+754%)
Mutual labels:  data-mining, data-processing
Querylist
🕷️ The progressive PHP crawler framework! 优雅的渐进式PHP采集框架。
Stars: ✭ 2,392 (+2292%)
Mutual labels:  scraper, spider
ant
A web crawler for Go
Stars: ✭ 264 (+164%)
Mutual labels:  scraper, spider
Goribot
[Crawler/Scraper for Golang]🕷A lightweight distributed friendly Golang crawler framework.一个轻量的分布式友好的 Golang 爬虫框架。
Stars: ✭ 190 (+90%)
Mutual labels:  scraper, spider
ECG analysis
No description or website provided.
Stars: ✭ 32 (-68%)
Mutual labels:  data-mining, data-processing
Linkedin Profile Scraper
🕵️‍♂️ LinkedIn profile scraper returning structured profile data in JSON. Works in 2020.
Stars: ✭ 171 (+71%)
Mutual labels:  scraper, spider
evine
Interactive CLI Web Crawler
Stars: ✭ 140 (+40%)
Mutual labels:  data-mining, scraper
Django Dynamic Scraper
Creating Scrapy scrapers via the Django admin interface
Stars: ✭ 1,024 (+924%)
Mutual labels:  scraper, spider
Geziyor
Geziyor, a fast web crawling & scraping framework for Go. Supports JS rendering.
Stars: ✭ 1,246 (+1146%)
Mutual labels:  scraper, spider
Awesome Ai Books
Some awesome AI related books and pdfs for learning and downloading, also apply some playground models for learning
Stars: ✭ 855 (+755%)
Mutual labels:  data-mining, books
crawler-chrome-extensions
爬虫工程师常用的 Chrome 插件 | Chrome extensions used by crawler developer
Stars: ✭ 53 (-47%)
Mutual labels:  scraper, spider

Blinkist M4A Downloader

Build Status GitHub license Version Go Report Card

What is Blinkist.com

  • Listen to key ideas from the world's best non-fiction books in just 15 minutes.

Requirements

  • Golang 1.10.2 or higher.
  • Blinkist (https://www.blinkist.com) Premium account.
  • Roughly 25 GB of free disk space.
  • wget installed and setup in PATH

Configuration

Enter your username and password in:

  1. blinkist/main.go, lines #16, #17.
  2. download/download.go, lines #17, #18.

Application

  • Run go run main.go inside blinkist/ folder to produce books_urls.txt, the list of unduplicated URLs of all of the books.
  • Run go run download.go inside download/ folder to start downloading audio files from the above URLs. books_urls.txt must be present in the download/ folder!

Technical details of the solution

  1. Look for HTML tag data-book-id e.g."5c28f2fc6cee070008e7a3d7" in each book URL.

  2. Look for all HTML tags data-chapterNo e.g."1" and corresponding data-chapterId e.g."5c28f3296cee070007b46369" (both on the same line) from each book URL.

  3. Construct this API link to get the short-lived download link: https://www.blinkist.com/api/books/<data-book-id>/chapters/<data-chapterId>/audio. (e.g.https://www.blinkist.com/api/books/5c28f2fc6cee070008e7a3d7/chapters/5c28f3296cee070007b46369/audio).

  4. Read the output for each book chapter, e.g.:

{"url":"https://abcdefgh12345.cloudfront.net/5c28f2fc6cee070008e7a3d7/5c28f3296cee070007b46369.m4a?Expires=1234567890\u0026Signature=abcdefghijklmnopqrstuvwxyz1234-567890abcde-fghi~jklmnopqrstuvwxyz1234567890abcdefgh~jklmnopqrstuvwxyz1234567890abcdefgh-abcd~abcdefghijklmnopqrstuvwxyz1234-567890abcde~jklmnopqrstuvwxyz1234567890abcdefgh-jklmnopqrstuvwxyz1234567890abcdefgh-567890abcde__\u0026Key-Pair-Id=ABCDEFGHIJKLMNOPQRST"}
  1. If the book contains audio (the previous step returns something), create a folder based on JavaScript tag e.g."reader:book:title:changed", "Bad Blood" on the local drive.

  2. Decode to proper URL, (replace \u0026 with &), e.g.: https://abcdefgh12345.cloudfront.net/5c28f2fc6cee070008e7a3d7/5c28f3296cee070007b46369.m4a?Expires=1234567890&Signature=abcdefghijklmnopqrstuvwxyz1234-567890abcde-fghi~jklmnopqrstuvwxyz1234567890abcdefgh~jklmnopqrstuvwxyz1234567890abcdefgh-abcd~abcdefghijklmnopqrstuvwxyz1234-567890abcde~jklmnopqrstuvwxyz1234567890abcdefgh-jklmnopqrstuvwxyz1234567890abcdefgh-567890abcde__&Key-Pair-Id=ABCDEFGHIJKLMNOPQRST

  3. Download the chapter using the above link as the m4a file. Filename will be based on data-chapterNo and stored in the book title folder, e.g.: Bad Blood/000.m4a, Bad Blood/001.m4a, Bad Blood/002.m4a,... .

Stats

Item Size
Categories 27
Books 1,771
Books with Audio 1,576
Books missing Audio 195
No. of m4a files 14,646
All files size 26,473,732,000 B (25.2GB)
Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected].