All Projects → dipu-bd → Lightnovel Crawler

dipu-bd / Lightnovel Crawler

Licence: apache-2.0
Download and generate e-books from online sources.

Programming Languages

python
94365 projects - #7 most used programming language

Projects that are alternatives of or similar to Lightnovel Crawler

Easybook
Book publishing as easy as it should be (built with Symfony components)
Stars: ✭ 744 (+116.28%)
Mutual labels:  mobi, pdf, epub
Readteractive
Tool for writing and generating interactive books.
Stars: ✭ 23 (-93.31%)
Mutual labels:  mobi, pdf, epub
Koodo Reader
A modern ebook manager and reader with sync and backup capacities for Windows, macOS, Linux and Web
Stars: ✭ 2,938 (+754.07%)
Mutual labels:  mobi, pdf, epub
Owasp Masvs
The Mobile Application Security Verification Standard (MASVS) is a standard for mobile app security.
Stars: ✭ 1,030 (+199.42%)
Mutual labels:  mobi, pdf, epub
Redux Offline Docs
Redux documentation in PDF, ePub and MOBI formats for offline reading.
Stars: ✭ 292 (-15.12%)
Mutual labels:  mobi, pdf, epub
Open Publisher
Using Jekyll to create outputs that can be used as Pandoc inputs. In short - input markdown, output mobi, epub, pdf, and print-ready pdf. With a focus on fiction.
Stars: ✭ 242 (-29.65%)
Mutual labels:  mobi, pdf, epub
Ethereum Development With Go Book
📖 A little book on Ethereum Development with Go (golang)
Stars: ✭ 754 (+119.19%)
Mutual labels:  mobi, pdf, epub
Kotlin Reference Chinese
Kotlin 官方文档(参考部分)中文版
Stars: ✭ 85 (-75.29%)
Mutual labels:  mobi, pdf, epub
The Economist Ebooks
经济学人(含音频)、纽约客、自然、新科学人、卫报、科学美国人、连线、大西洋月刊、新闻周刊、国家地理等英语杂志免费下载、订阅(kindle推送),支持epub、mobi、pdf格式, 每周更新. The Economist 、The New Yorker 、Nature、The Atlantic 、New Scientist、The Guardian、Scientific American、Wired、Newsweek magazines, free download and subscription for kindle, mobi、epub、pdf format.
Stars: ✭ 3,471 (+909.01%)
Mutual labels:  mobi, pdf, epub
Laravel Book
Up to date Epub, Mobi and PDF versions from the official Laravel Docs
Stars: ✭ 221 (-35.76%)
Mutual labels:  mobi, pdf, epub
Ruby Hacking Guide.github.com
Ruby Hacking Guide Translation
Stars: ✭ 305 (-11.34%)
Mutual labels:  mobi, pdf, epub
fiction-dl
A content downloader, capable of retrieving works of (fan)fiction from the web and saving them in a few common file formats.
Stars: ✭ 22 (-93.6%)
Mutual labels:  mobi, epub
Loritta
💁 A multipurpose, multilanguage, customizable, modular, and very cute bot for Discord using JDA! ~Making your server more awesome~
Stars: ✭ 253 (-26.45%)
Mutual labels:  bot, discord-bot
Narou
Narou.rb - 小説家になろうのダウンローダ&縦書き整形&管理アプリ。Kindle(などの電子書籍端末)でなろうを読む場合に超便利です!
Stars: ✭ 256 (-25.58%)
Mutual labels:  mobi, epub
Nostrum
Elixir Discord Library
Stars: ✭ 274 (-20.35%)
Mutual labels:  bot, discord-bot
ebook-converter
Commandline tool for converting between several e-books formats, based on Calibre project.
Stars: ✭ 21 (-93.9%)
Mutual labels:  mobi, epub
Livebot
An app that allows you to be inside a bot!
Stars: ✭ 271 (-21.22%)
Mutual labels:  bot, discord-bot
Starter Book
A book starter to kickstart your writing journey 🎉
Stars: ✭ 277 (-19.48%)
Mutual labels:  pdf, epub
Mantarobot
Multipurpose Discord bot made in Java using JDA.
Stars: ✭ 282 (-18.02%)
Mutual labels:  bot, discord-bot
Percollate
A command-line tool to turn web pages into beautiful, readable PDF, EPUB, or HTML docs.
Stars: ✭ 3,535 (+927.62%)
Mutual labels:  pdf, epub

Lightnovel Crawler pip package download win download linux

Build Status Python version PyPI version License GitHub stars

An app to download novels from online sources and generate e-books.

Discord: https://discord.gg/wMECG2Q

Telegram: https://t.me/epub_smelter_bot

Table of contents

(A) Installation

This application uses Calibre to convert ebooks.
Install it from https://calibre-ebook.com/download
Without it, you will only get output in epub, text, and web formats.

A1. Standalone Bundle (Windows, Linux)

Windows: lightnovel-crawler v2.24.5 ~ 23MB

In Windows 8, 10 or later versions, it might say that lncrawl.exe is not safe to dowload or execute. You should bypass/ignore this security check to execute this program.

To get older versions visit the Releases page

Linux: lightnovel-crawler v2.24.5 ~ 24MB

Copy it to /usr/bin or ~/.local/bin to make it accessible in terminal. Or, you can create a launcher to easily access the app. However, it is recommended to use pip version directly if you are on Linux

A2. PIP (Windows, Mac, and Linux)

📦 A python package named lightnovel-crawler is available at pypi.

Make sure you have installed Python v3.5 or higher and have pip enabled. Visit these links to install python with pip in Windows, Linux and MacOS. Feel free to ask on the Discord server if you are stuck.

To install this app or to update installed one via pip, just run:

$ pip install --user -U lightnovel-crawler

In some cases you have to use python3 -m pip or pip3 or python -m pip. And you do not need --user option, if you are running from root.

Next, open your terminal and enter:

$ lightnovel-crawler

# Or, a shortcut:
$ lncrawl

To view extra logs, use: lncrawl -lll

A3. Termux (Android)

Mobile platforms are unpredictable. It is not guaranteed that the app will run on all devices. It is recommended to use the bots on either Discord or Telegram if you are on mobile.

📱 Using Termux, you can run this app in your android phones too. Follow this instructions:

  • Install Termux from playstore.
  • Open the app and run these commands one by one:
    • apt update && apt upgrade
    • termux-setup-storage
    • pkg install ndk-sysroot make python zlib clang
    • pkg install libxml2 libxslt libiconv libcrypt libffi zlib libjpeg-turbo
    • pkg install nodejs-lts
    • pip install -U lightnovel-crawler to install the latest version of this app.
  • Now exit the console and relaunch it.
  • Type cd ~/storage/downloads to store novels there.
  • Type lncrawl to start.
  • You navigate up using Volume UP + W and down using Volume UP + S.
  • Run pip install -U lightnovel-crawler again to install the latest updates.

A4. Chatbots

A4.1 Discord

Join our server: https://discord.gg/7A5Hktx

Or, visit this link to install discord bot to your own server: https://discordapp.com/oauth2/authorize?client_id=537526751170002946&permissions=51264&scope=bot

A4.2 Telegram

Visit this link to get started with the telegram bot: https://t.me/epub_smelter_bot

Send !help to open the bot help message.

A5. Run from source

  • First clone the repository:
$ git clone https://github.com/dipu-bd/lightnovel-crawler
  • Open command prompt inside of the project folder and install requirements:
$ pip install --user -r requirements.txt
  • Run the program (use python v3.5 or higher):
$ python __main__.py

# Or, in short,
$ python .

A6. Heroku Deployment

Simply fill out the environment variables and you get a running instance.

Deploy

(B) General Usage

B1. Available options

$ lncrawl -h
================================================================================
                           📒 Lightnovel Crawler 🍀2.24.0
                  https://github.com/dipu-bd/lightnovel-crawler
--------------------------------------------------------------------------------
usage: lncrawl [options...]
       lightnovel-crawler [options...]

optional arguments:
  -h, --help            show this help message and exit

  -v, --version         show program's version number and exit
  -l                    Set log levels. (-l = warn, -ll = info, -lll = debug).
  --list-sources        Display a list of available sources.
  -s URL, --source URL  Profile page url of the novel.
  -q STR, --query STR   Novel query followed by list of source sites.
  -x, --sources         Display the source selection menu while searching.
  --login USER PASSWD   User name/email address and password for login.
  --format E [E ...]    Define which formats to output. Default: all.
  --add-source-url      Add source url at the end of each chapter.
  --single              Put everything in a single book.
  --multi               Build separate books by volumes.
  -o PATH, --output PATH
                        Path where the downloads to be stored.
  --filename NAME       Set the output file name
  --filename-only       Skip appending chapter range with file name
  -f, --force           Force replace any existing folder.
  -i, --ignore          Ignore any existing folder (do not replace).
  --all                 Download all chapters.
  --first [COUNT]       Download first few chapters (default: 10).
  --last [COUNT]        Download last few chapters (default: 10).
  --page START STOP.    The start and final chapter urls.
  --range FROM TO.      The start and final chapter indexes.
  --volumes [N [N ...]]
                        The list of volume numbers to download.
  --chapters [URL [URL ...]]
                        A list of specific chapter urls.
  --bot {console,telegram,discord,test}
                        Select a bot. Default: console.
  --shard-id [SHARD_ID]
                        Discord bot shard id (default: 0)
  --shard-count [SHARD_COUNT]
                        Discord bot shard counts (default: 1)
  --suppress            Suppress all input prompts and use defaults.
  ENV                   [chatbots only] Pass query string at the end of all options. It will be use instead of .env
                        file. Sample: "BOT=discord&DISCORD_TOKEN=***&LOG_LEVEL=DEBUG"

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

B2. Example Usage

Open your console and type lncrawl --version first to check if you have installed it properly. Here are some example usage of the app:

  • To start an interactive session: lncrawl

  • To download using an url: lncrawl -s https://boxnovel.com/novel/reincarnation-of-the-strongest-sword-god/

  • To search novels: lncrawl -q "Strongest Sword God"

  • To search novels from selected sources: lncrawl -q "Strongest Sword God" --sources

  • To download all chapters: lncrawl --all

  • To download first 25 chapters: lncrawl --first 25

  • To download all between two chapters: lncrawl --range 10 30

  • To download all between two chapter links: lncrawl https://boxnovel.com/novel/reincarnation-of-the-strongest-sword-god/chapter-2205 https://boxnovel.com/novel/reincarnation-of-the-strongest-sword-god/chapter-2211

  • To download a specific volumes: lncrawl --volumes 2 3

  • To define output path: lncrawl -o "D:\Lightnovels\reincarnation-of-the-strongest-sword-god"

  • To delete the output folder if exists: lncrawl -f

  • To ignore the output folder if exists: lncrawl -i

  • To resume download where is has been left previously: lncrawl -i

  • To specify output formats: lncrawl --format epub pdf mobi

  • To display list of supported sources: lncrawl ---list-sources

  • If you provide an option in the argument, it will skip it in the interactive session. If you want to disable all interactive prompts, pass --suppress at the end.

  • You can stack up options like this: lncrawl -s https://boxnovel.com/novel/reincarnation-of-the-strongest-sword-god/ -o "D:\Lightnovels\reincarnation-of-the-strongest-sword-god" --last 50 -i --format pdf --suppress

B3. Running the bot

There are two chatbots available at this moment: Telegram and Discord. To run your own bot server, follow these instructions:

# Clone this repository
$ git clone https://github.com/dipu-bd/lightnovel-crawler
# Install requirements
$ pip3 install --user -r requirements.txt
$ pip3 install --user -r bot_requirements.txt
# Edit the environment variables
# You should give your API keys and log info here
# Also specify which bot server you want to run
$ cp .env.example .env
$ vim .env
# Run the server using:
$ python3 .

There is a server.sh script to run a bot in ubuntu servers. It will basically execute the python __main__.py and send the task to run in background. I use it to run my discord bot in the server.

(C) Development

You are very welcome to contribute in this project. You can:

  • create new issues pointing out the bugs.
  • solve existing issues.
  • add your own sources.
  • add new output formats.
  • create new bots.

C1. Adding new source

  • Create new crawler using the sources/_template_.py as template.
  • Update Supported sources section in README.md
  • Add some test inputs to test_user_inputs variable in lncrawl/bots/test/test_inputs.py

C2. Adding new Bot

C3. Supported sources

Request new one by creating a new issue.

Available Sources Can Search Can Login Maintainer
http://boxnovel.org @dipu-bd
http://liberspark.com
http://novelfull.com @dipu-bd
http://tiknovel.com
http://www.fujitranslation.com
http://www.hanyunovels.site @SirGryphin
http://www.machinenoveltranslation.com
http://www.tiknovel.com
http://zenithnovels.com
http://zhi-end.blogspot.co.id @SirGryphin
http://zhi-end.blogspot.com @SirGryphin
https://4scanlation.xyz
https://88tangeatdrinkread.wordpress.com @SirGryphin
https://9kqw.com
https://anythingnovel.com
https://arangscans.com @SirGryphin
https://asadatranslations.com @SirGryphin
https://automtl.wordpress.com @SirGryphin
https://babelnovel.com
https://bestlightnovel.com
https://bestoflightnovels.com @SirGryphin
https://book.qidian.com
https://boxnovel.com @dipu-bd
https://clicknovel.net @SirGryphin
https://creativenovels.com
https://crescentmoon.blog
https://daonovel.com @SirGryphin
https://darktranslation.com @SirGryphin
https://demontranslations.com @SirGryphin
https://dmtranslationscn.com @SirGryphin
https://dsrealmtranslations.com @SirGryphin
https://es.mtlnovel.com
https://exiledrebelsscanlations.com @SirGryphin
https://fanstranslations.com @SirGryphin
https://fastnovel.net @SirGryphin
https://foxaholic.com
https://fr.mtlnovel.com
https://fujitranslation.com @SirGryphin
https://hui3r.wordpress.com @SirGryphin
https://id.mtlnovel.com
https://inadequatetranslations.wordpress.com @SirGryphin
https://indowebnovel.id
https://infinitenoveltranslations.net @SirGryphin
https://instadoses.com
https://jpmtl.com @dipu-bd
https://jstranslations1.com @SirGryphin
https://justatranslatortranslations.com @SirGryphin
https://kiss-novel.com
https://kisslightnovels.info
https://lemontreetranslations.wordpress.com @SirGryphin
https://light-novel.online
https://lightnovelheaven.com @SirGryphin
https://lightnovelsonl.com @SirGryphin
https://listnovel.com
https://litnet.com
https://lnmtl.com @dipu-bd
https://m.chinesefantasynovels.com
https://m.mywuxiaworld.com
https://m.novelspread.com
https://m.readlightnovel.cc
https://m.romanticlovebooks.com
https://m.wuxiaworld.co @dipu-bd
https://mangatoon.mobi
https://meionovel.id
https://moonstonetranslation.com @SirGryphin
https://morenovel.net @SirGryphin
https://myoniyonitranslations.com
https://mysticalmerries.com @SirGryphin
https://novel27.com @SirGryphin
https://novelcrush.com @SirGryphin
https://novelfull.com @dipu-bd
https://novelgo.id/
https://novelonlinefree.com @SirGryphin
https://novelonlinefull.com
https://novelraw.blogspot.com
https://novelsrock.com
https://noveltranslate.com @SirGryphin
https://noveltrench.com @SirGryphin
https://pery.info/ @SirGryphin
https://ranobelib.me
https://readwebnovels.net @SirGryphin
https://readlightnovels.net @PreownedFIN
https://reincarnationpalace.com @SirGryphin
https://rewayat.club
https://shalvationtranslations.wordpress.com @SirGryphin
https://skynovel.org/ @SirGryphin
https://sleepytranslations.com @SirGryphin
https://tomotranslations.com
https://viewnovel.net @SirGryphin
https://vipnovel.com @SirGryphin
https://vistranslations.wordpress.com @SirGryphin
https://volarenovels.com
https://wbnovel.com
https://webnovel.online
https://webnovelindonesia.com
https://webnovelonline.com
https://woopread.com
https://wordexcerpt.com @dipu-bd, @SirGryphin
https://wordexcerpt.org @dipu-bd, @SirGryphin
https://writerupdates.com @SirGryphin
https://wuxiaworld.io @SirGryphin
https://wuxiaworld.live @SirGryphin
https://wuxiaworld.online @dipu-bd
https://wuxiaworld.site @dipu-bd
https://www.aixdzs.com
https://www.asianhobbyist.com
https://www.centinni.com @SirGryphin
https://www.daocaorenshuwu.com @SirGryphin
https://www.f-w-o.com @SirGryphin
https://www.fuyuneko.org @SirGryphin
https://www.idqidian.us
https://www.lightnovelworld.com
https://www.machine-translation.org
https://www.mtlnovel.com
https://www.mywuxiaworld.com
https://www.novelall.com
https://www.novelcool.com @SirGryphin
https://www.novelhall.com
https://www.novelhunters.com @SirGryphin
https://www.novelringan.com
https://www.novelspread.com
https://www.novelupdates.cc @SirGryphin
https://www.oppatranslations.com @SirGryphin
https://www.qidian.com
https://www.readlightnovel.cc
https://www.readlightnovel.org
https://www.readnovelfull.com
https://www.romanticlovebooks.com
https://www.royalroad.com
https://www.scribblehub.com
https://www.shinsori.com
https://www.tapread.com
https://www.translateindo.com
https://www.virlyce.com @SirGryphin
https://www.wattpad.com @dipu-bd
https://www.webnovel.com @dipu-bd
https://www.webnovelover.com @SirGryphin
https://www.worldnovel.online
https://www.wuxialeague.com
https://www.wuxiaworld.co @dipu-bd
https://www.wuxiaworld.com @dipu-bd
https://www.x81zw.com @SirGryphin
https://www.xiainovel.com @SirGryphin
https://www.xsbiquge.com @SirGryphin
https://yukinovel.id

C4. Rejected sources

Rejected Sources Reason
http://fullnovel.live 403 - Forbidden: Access is denied
http://gravitytales.com Redirects to webnovel.com
http://moonbunnycafe.com Does not follow uniform format
https://anythingnovel.com Site broken
https://chrysanthemumgarden.com Removed on request of the owner #649
https://indomtl.com Does not like to be crawled
https://lnindo.org Does not like to be crawled
https://mtled-novels.com Domain is expired
https://novelplanet.com Site is closed
https://www.flying-lines.com Obfuscated content
https://www.jieruihao.cn Unavailable
https://www.noveluniverse.com Site is down
https://www.novelupdates.com Does not host any novels
https://www.novelv.com Site is down
https://www.rebirth.online Site moved

C5. Supported output formats

  • JSON (default)
  • EPUB (default)
  • TEXT
  • WEB
  • DOCX
  • MOBI
  • PDF
  • RTF
  • TXT
  • AZW3
  • FB2
  • LIT
  • LRF
  • OEB
  • PDB
  • PML
  • RB
  • SNB
  • TCR
  • HTML

C6. Supported bots

  • Console Bot
  • Telegram Bot
  • Discord Bot
Note that the project description data, including the texts, logos, images, and/or trademarks, for each open source project belongs to its rightful owner. If you wish to add or remove any projects, please contact us at [email protected]