python-sitemap 0

Mini website crawler to make sitemap from a website.

3 years after

Python-Sitemap

Simple script to crawl a website and create a sitemap.xml of all public link in a website

Warning : This script is designed to works with Python3

Simple usage

>>> python main.py --domain http://blog.lesite.us --output sitemap.xml

Advanced usage

Read a config file to set parameters: You can overide (or add for list) any parameters define in the config.json

>>> python main.py --config config.json

Enable debug :

>>> python main.py --domain http://blog.lesite.us --output sitemap.xml --debug

Enable report for print summary of the crawl:

>>> python main.py --domain http://blog.lesite.us --output sitemap.xml --report

Skip url (by extension) (skip pdf AND xml url):

>>> python main.py --domain http://blog.lesite.us --output sitemap.xml --skipext pdf --skipext xml 

Drop a part of an url via regexp :

>>> python main.py --domain http://blog.lesite.us --output sitemap.xml --drop "id=[0-9]{5}"

Exclude url by filter a part of it :

>>> python main.py --domain http://blog.lesite.us --output sitemap.xml --exclude "action=edit"

Read the robots.txt to ignore some url:

>>> python main.py --domain http://blog.lesite.us --output sitemap.xml --parserobots

Related Repositories

Cactus

Cactus

Static site generator for designers. Uses Python and Django templates. ...

acrylamid

acrylamid

(unmaintained) static blog generator in python with incremental rendering ...

sitemapgen4j

sitemapgen4j

SitemapGen4j is a library to generate XML sitemaps in Java. ...

flask-sitemap

flask-sitemap

Flask-Sitemap is a Flask extension helping with sitemap generation. ...

Simple-Site-Checker

Simple-Site-Checker

Check the links in your website XML sitemap ...


Top Contributors

c4software sebclick Garrett-R ghuntley