mediacloud / ultimate-sitemap-parser

Website sitemap parser for Python 3.5+.

Features

• Supports all sitemap formats:
  • XML sitemaps
  • Google News sitemaps
  • plain text sitemaps
  • RSS 2.0 / Atom 0.3 / Atom 1.0 sitemaps
  • Sitemaps linked from robots.txt
• Field-tested with ~1 million URLs as part of the Media Cloud project
• Error-tolerant with more common sitemap bugs
• Tries to find sitemaps not listed in robots.txt
• Uses fast and memory efficient Expat XML parsing
• Doesn't consume much memory even with massive sitemap hierarchies
• Provides a generated sitemap tree as easy to use object tree
• Supports using a custom web client
• Uses a small number of actively maintained third-party modules
• Reasonably tested

Installation

pip install ultimate-sitemap-parser

Usage

from usp.tree import sitemap_tree_for_homepage

tree = sitemap_tree_for_homepage('https://www.nytimes.com/')
print(tree)

sitemap_tree_for_homepage() will return a tree of AbstractSitemap subclass objects that represent the sitemap hierarchy found on the website; see a reference of AbstractSitemap subclasses.

If you'd like to just list all the pages found in all of the sitemaps within the website, consider using all_pages() method:

# all_pages() returns an Iterator
for page in tree.all_pages():
    print(page)

all_pages() method will return an iterator yielding SitemapPage objects; see a reference of SitemapPage.