python-sitemap 0

Mini website crawler to make sitemap from a website.

Python-Sitemap

Simple script to crawl a website and create a sitemap.xml of all public link in a website

Warning : This script is designed to works with Python3

Simple usage

>>> python main.py --domain http://blog.lesite.us --output sitemap.xml

Advanced usage

Read a config file to set parameters: You can overide (or add for list) any parameters define in the config.json

>>> python main.py --config config.json

Enable debug :

>>> python main.py --domain http://blog.lesite.us --output sitemap.xml --debug

Enable report for print summary of the crawl:

>>> python main.py --domain http://blog.lesite.us --output sitemap.xml --report

Skip url (by extension) (skip pdf AND xml url):

>>> python main.py --domain http://blog.lesite.us --output sitemap.xml --skipext pdf --skipext xml 

Drop a part of an url via regexp :

>>> python main.py --domain http://blog.lesite.us --output sitemap.xml --drop "id=[0-9]{5}"

Exclude url by filter a part of it :

>>> python main.py --domain http://blog.lesite.us --output sitemap.xml --exclude "action=edit"

Read the robots.txt to ignore some url:

>>> python main.py --domain http://blog.lesite.us --output sitemap.xml --parserobots

Related Repositories

python-sitemap

python-sitemap

Mini website crawler to make sitemap from a website. ...

python-sitemap

python-sitemap

Python library for parsing & generating sitemaps ...

varela-python-sitemap-parser

varela-python-sitemap-parser

Python 2.6 sitemap parser ...

pythonsitemap

pythonsitemap

Python SiteMap Generator ...

sitemapserpent

sitemapserpent

A Sitemap generator written in Python that uses the lxml library ...


Top Contributors

c4software sebclick Garrett-R ghuntley