c4software / Python Sitemap
Licence: gpl-3.0
Mini website crawler to make sitemap from a website.
Stars: ✭ 246
Programming Languages
python
139335 projects - #7 most used programming language
Labels
Projects that are alternatives of or similar to Python Sitemap
Templates Using Bootstrap4
🌆 Here I've aggregated some of the most commonly used web-page templates made using Bootstrap4 🛒
Stars: ✭ 60 (-75.61%)
Mutual labels: sitemap
Laravel Seo Tools
Laravel Seo package for Content writer/admin/web master who do not know programming but want to edit/update SEO tags from dashboard
Stars: ✭ 99 (-59.76%)
Mutual labels: sitemap
Sitemap Generator Crawler
Script that generates a sitemap by crawling a given URL
Stars: ✭ 169 (-31.3%)
Mutual labels: sitemap
Sitemap
Sitemap is the easiest way to generate Sitemaps in Elixir.
Stars: ✭ 90 (-63.41%)
Mutual labels: sitemap
Blog Generator
static blog generator for my blog at https://zupzup.org/
Stars: ✭ 57 (-76.83%)
Mutual labels: sitemap
Sitemap Generator Cli
Creates an XML-Sitemap by crawling a given site.
Stars: ✭ 214 (-13.01%)
Mutual labels: sitemap
Laravel Sitemap
Create and generate sitemaps with ease
Stars: ✭ 1,325 (+438.62%)
Mutual labels: sitemap
Go Sitemap Generator
go-sitemap-generator is the easiest way to generate Sitemaps in Go
Stars: ✭ 152 (-38.21%)
Mutual labels: sitemap
Laravel Sitemap
Laravelium Sitemap generator for Laravel.
Stars: ✭ 1,231 (+400.41%)
Mutual labels: sitemap
Android Sitemap
👓 Every link ever to Android Developer site.
Stars: ✭ 61 (-75.2%)
Mutual labels: sitemap
React Router Sitemap
Generate sitemap.xml by React Router configuration
Stars: ✭ 189 (-23.17%)
Mutual labels: sitemap
Vue Router Sitemap
Generate sitemap.xml by vue-router configuration
Stars: ✭ 102 (-58.54%)
Mutual labels: sitemap
Seo
SEO utilities including a unique field type, sitemap & redirect manager
Stars: ✭ 210 (-14.63%)
Mutual labels: sitemap
Craft Seomatic
SEOmatic facilitates modern SEO best practices & implementation for Craft CMS 3. It is a turnkey SEO system that is comprehensive, powerful, and flexible.
Stars: ✭ 135 (-45.12%)
Mutual labels: sitemap
Python-Sitemap
Simple script to crawl websites and create a sitemap.xml of all public link in it.
Warning : This script only works with Python3
Simple usage
>>> python main.py --domain http://blog.lesite.us --output sitemap.xml
Advanced usage
Read a config file to set parameters: You can overide (or add for list) any parameters define in the config.json
>>> python main.py --config config/config.json
Enable debug:
$ python main.py --domain https://blog.lesite.us --output sitemap.xml --debug
Enable verbose output:
$ python main.py --domain https://blog.lesite.us --output sitemap.xml --verbose
Enable Image Sitemap
More informations here https://support.google.com/webmasters/answer/178636?hl=en
$ python main.py --domain https://blog.lesite.us --output sitemap.xml --images
Enable report for print summary of the crawl:
$ python main.py --domain https://blog.lesite.us --output sitemap.xml --report
Skip url (by extension) (skip pdf AND xml url):
$ python main.py --domain https://blog.lesite.us --output sitemap.xml --skipext pdf --skipext xml
Drop a part of an url via regexp :
$ python main.py --domain https://blog.lesite.us --output sitemap.xml --drop "id=[0-9]{5}"
Exclude url by filter a part of it :
$ python main.py --domain https://blog.lesite.us --output sitemap.xml --exclude "action=edit"
Read the robots.txt to ignore some url:
$ python main.py --domain https://blog.lesite.us --output sitemap.xml --parserobots
Human readable XML
$ python3 main.py --domain https://blog.lesite.us --images --parserobots | xmllint --format -
Multithreaded
$ python3 main.py --domain https://blog.lesite.us --num-workers 4
with basic auth
You need to configure username
and password
in your config.py
before
$ python3 main.py --domain https://blog.lesite.us --auth
Output sitemap index file
Sitemaps with over 50,000 URLs should be split into an index file that points to sitemap files that each contain 50,000 URLs or fewer. Outputting as an index requires specifying an output file. An index will only be output if a crawl has more than 50,000 URLs:
$ python3 main.py --domain https://blog.lesite.us --as-index --output sitemap.xml
Docker usage
Build the Docker image:
$ docker build -t python-sitemap:latest .
Run with default domain :
$ docker run -it python-sitemap
Run with custom domain :
$ docker run -it python-sitemap --domain https://www.graylog.fr
Run with config file and output :
You need to configure config.json file before
$ docker run -it -v `pwd`/config/:/config/ -v `pwd`:/home/python-sitemap/ python-sitemap --config config/config.json
Note that the project description data, including the texts, logos, images, and/or trademarks,
for each open source project belongs to its rightful owner.
If you wish to add or remove any projects, please contact us at [email protected].