. . . in cache. Separate caches for text and media results. Admin configurable. Follow sitemap files If available, sitemap.xml as well as gzip compressed files will be used to follow the links of a site. If is detected, also multiple sitemap files are processed. Periodical Re-index Re-indexing could be performed automatically, repeated every . . .
. . . not followed during Re-index, only main URL is indexed (option 1). It is not a bug, it is a feature. If 'Follow sitemap.xml' is activated in Admin settings, links will only be followed if: - 'last modified' date in sitemap.xml is newer than Sphiders 'last indexed' date. - New link that is not yet known in Sphiders link table. The main URL will . . .