big_sitemap 0 ruby Rubygems

A Sitemap library suitable for large sites. Compatible with most frameworks, including Rails and Merb.

7 years after

= BigSitemap

BigSitemap is a {Sitemap}[] generator suitable for applications with greater than 50,000 URLs. It splits large Sitemaps into multiple files, gzips the files to minimize bandwidth usage, batches database queries to minimize memory usage, supports increment updates, can be set up with just a few lines of code and is compatible with just about any framework.

BigSitemap is best run periodically through a Rake/Thor task.

require 'big_sitemap'

sitemap = :url_options => {:host => ''}, :document_root => "#{APP_ROOT}/public" )

Add a model

sitemap.add Product

Add another model with some options

sitemap.add(Post, :conditions => {:published => true}, :path => 'articles', :change_frequency => 'daily', :priority => 0.5 )

Add a static resource

sitemap.add_static('',, 'monthly', 0.1)

Generate the files


The code above will create a minimum of four files:

  1. public/sitemaps/sitemap_index.xml.gz
  2. public/sitemaps/sitemap_products.xml.gz
  3. public/sitemaps/sitemap_posts.xml.gz
  4. public/sitemaps/sitemap_static.xml.gz

If your sitemaps grow beyond 50,000 URLs (this limit can be overridden with the :max_per_sitemap option), the sitemap files will be partitioned into multiple files (sitemap_products_1.xml.gz, sitemap_products_2.xml.gz, ...).

=== Framework-specific Classes

Use the framework-specific classes to take advantage of built-in shortcuts.

==== Rails

BigSiteMapRails includes UrlWriter (useful for making use of your Rails routes - see the Location URLs section) and deals with setting the :document_root and :url_options initialization options.

==== Merb

BigSitemapMerb deals with setting the :document_root initialization option.

== Install

Via gem:

sudo gem install big_sitemap

== Advanced

=== Initialization Options

  • :url_options -- hash with :host, optionally :port and :protocol
  • :base_url -- string alternative to :url_options, e.g. ''
  • :document_root -- string
  • :path -- string defaults to 'sitemaps', which places sitemap files under the /sitemaps directory
  • :max_per_sitemap -- 50000, which is the limit dictated by Google but can be less
  • :batch_size -- 1001 (not 1000 due to a bug in DataMapper)
  • :gzip -- true
  • :ping_google -- true
  • :ping_yahoo -- false, needs :yahoo_app_id
  • :ping_bing -- false
  • :ping_ask -- false
  • :partial_update -- false

=== Chaining

You can chain methods together: => {:host => ''}).add(Post).generate

With the Rails-specific class, you could even get away with as little code as:

=== Pinging Search Engines

To ping search engines, call ping_search_engines after you generate the sitemap:


=== Location URLs

By default, URLs for the "loc" values are generated in the form:


Alternatively, you can pass a lambda. For example, to make use of your Rails route helper:

sitemap.add(Post, :location => lambda { |post| post_url(post) } )

=== Change Frequency, Priority and Last Modified

You can control "changefreq", "priority" and "lastmod" values for each record individually by passing lambdas instead of fixed values:

sitemap.add(Post, :change_frequency => lambda { |post| ... }, :priority => lambda { |post| ... }, :last_modified => lambda { |post| ... } )

=== Find Methods

Your models must provide either a find_for_sitemap or all class method that returns the instances that are to be included in the sitemap.

Additionally, you models must provide a count_for_sitemap or count class method that returns a count of the instances to be included.

If you're using ActiveRecord (Rails) or DataMapper then all and count are already provided and you can make use of any supported parameter: (:conditions, :limit, :joins, :select, :order, :include, :group)

sitemap.add(Track, :select => "id, permalink, user_id, updated_at", :include => :user, :conditions => "public = 1 AND state = 'finished' AND user_id IS NOT NULL", :order => "id ASC" )

If you provide your own find_for_sitemap or all method then it should be able to handle the :offset and :limit options, in the same way that ActiveRecord and DataMapper handle them. This is especially important if you have more than 50,000 URLs.

=== Partial Update

If you enable :partial_update, the filename will include an id smaller than the id of the first entry. This is perfect to update just the last file with new entries without the need to re-generate files being already there.

=== Lock Generation Process

To prevent another process overwriting from the generated files, use the with_lock method:

sitemap.with_lock do sitemap.generate end

=== Cleaning the Sitemaps Directory

Calling the clean method will remove all files from the Sitemaps directory.

== Limitations

If your database is likely to shrink during the time it takes to create the sitemap then you might run into problems (the final, batched SQL select will overrun by setting a limit that is too large since it is calculated from the count, which is queried at the very beginning). In this case and your database uses incremental primary IDs then you might want to use the :partial_update option, which looks at the last ID instead of paginating.


Tests for framework-specific components.

== Credits

Thanks to Alastair Brunton and Harry Love, who's work provided a starting point for this library.

Thanks also to those who have contributed patches:

== Copyright

Copyright (c) 2010 Stateless Systems ( See LICENSE for details.

Related Repositories



sitemap-generator is a Rails plugin that makes it easy to generate sitemaps. ...



Super DRY Sitemaps for Rails and Sinatra Apps ...



BigSitemap is a Sitemapgenerator suitable for applications with greater than 50, ...

Top Contributors

alexrabarts sbecker dalibor christianhellsten jschoolcraft mislav


-   v0.8.2 zip tar
-   v0.8.1 zip tar
-   v0.4.0 zip tar
-   v0.3.5 zip tar
-   v0.3.4 zip tar
-   v0.3.3 zip tar
-   v0.3.2 zip tar
-   v0.2.1 zip tar
-   v0.2.0 zip tar
-   v0.1.3 zip tar
-   v0.1.1 zip tar