Sphider-plus

Sphider-plus is a search engine based on the scripts of original Sphider. [ About Sphider-plus] More than 400 new features (additional mods functions template designs and debugging) have been added to the original Sphider. For details about all the improvements and changes please read the Documentation section. [ Main features ] Item . . .
. . .
section. [ Main features ] Item Description UTF-8 and UTF-16 support Indexation and search procedure for Chinese Cyrillic Georgian Hebrew etc. charsets. UNICODE support including astral symbols. Support for non-ASCII domains 'Internationalized Domain Names' (IDN) like 'http://президент.рф/' and 'http://müller.de/' are accepted and . . .
. . .
'http://müller.de/' are accepted and processed. Responsive design Automatically adapting the size of search form result listing and addurl form to display size of computers tablets smartphones etc. Media support Index and search for images (incl. Open Graph images) audio and video (incl. Youtube videos). EXIF and ID3 information are also . . .
. . .
player software. Multiple database support Individual configuration and activation of databases for 'Admin' 'Search User' and 'Suggest URL'. Support of multiple table sets in each db MySQL query cache individual index for each db individual or bulk search in predefined databases. Individual Admin settings for each db and each set of . . .
. . .
always in cache. Separate caches for text and media results. Admin configurable. Follow sitemap files If available sitemap.xml as well as gzip compressed files will be used to follow the links of a site. If <sitemapindex . . . > is detected also multiple sitemap files are processed. Periodical Re-index Re-indexing could be performed . . .
. . .
also multiple sitemap files are processed. Periodical Re-index Re-indexing could be performed automatically repeated every selected time interval. Admin selectable intervals for 3 hours 12 hours 1 day 1 week or 1 month. Multithreaded indexing In order to reduce the time for indexing 1-10 parallel running threads might be activated . . .
. . .
1-10 parallel running threads might be activated in Admin settings. Preferred re-index While invoking this option the admin may select a suitable level for the next index procedure. Thus only those URLs containing the according level will be re-indexed.. Erase Re-index and Continue suspended index procedures Individual (site specific) or . . .
. . .
(site specific) or bulk update of database. Support of XML product feeds Index and search of feed content inclusive formatting the search results. RDF RSD RSS and Atom feed support Index and search of feed content inclusive RDF 'Dublin Core' tags. Obey / ignore 'preferred' tags in RSD feeds. Follow / ignore CDATA directives. Various . . .
. . .
ignore 'preferred' tags in RSD feeds. Follow / ignore CDATA directives. Various search modes Search with wildcards Tolerant search Search strict Search only in one domain Search all links of a site Search for media (link-specific). Add thumbnails to each page presented in text results Admin selectable this feature will present a web shot . . .
. . .
suspected to contain malware or phishing content. 11 different modes of sorting the text results Admin selectable: -By relevance (weight % ) -By hit counts in full text -Most popular links on top -By indexdate -By URL names -By file suffix -Main URL (domain) on top -Like Google (Top 2 per URL) - Promoted domain on top - Links holding promoted . . .
. . .
on top - Links holding promoted catchwords on top. 5 different modes of sorting the media results Admin selectable: -By title(alphabetic) -By file suffix -By image size -By 'Last queried' -By 'Most popular'. Same results for queries typed with pure vowels or with accents Will deliver the same results for queries like: caf e and caf é . To be . . .
. . .
in Admin backend. Same results for queries with and without quotes Will deliver the same results for queries like: d'information <-> information dei'largi <-> largi Also Admin selectabe: Equalize the different quotes like: ' ` ´ Same text results for queries with and without ligatures Admin selectable; will deliver the same results . . .
. . .
results for queries with and without ligatures Admin selectable; will deliver the same results for queries like: cœur and coeur . Worked out for Latin ligatures in Unicode (Latin-derived alphabets) and also ligatures used only in phonetic transcription but not taking into consideration medieval ligatures. Present all results for singular and . . .
. . .
all will become searchable. Dictionaries with 106.800 radicals. Segmentation of Japanese words Segmentation of 5.724 kanji (new old and half width) hiragana katakana and jinmeiyo Japanese character writing systems. Transliterate Latin characters into their Greek equivalents Transforms query input alla to find ἀλλὰ and baptismatos to find . . .
. . .
redirections caused by HTTP 301 302 303 and 307 status codes. Also obeying JavaScript sent as HTML content like: <SCRIPT language="javascript"> win.loc="mp.php?mcv=59";</SCRIPT> Follow header redirections refresh tags and canonical links Automatical forwarding for the indexer. Follow links found in JavaScript and index also the . . .
. . .
and index also the content of document.write Will index JavaScript commands. Detect and follow links like: document.write(' <a href="new12.pdf">All news 2012</a> '); and index the content of: document.write(' this content '); Not indexing content created in real-time by JavaScript. Accept gzip formatted transmission In order . . .
. . .
Converter included for PDF DOCX XLSX ODT ODS CSV PPTX and XLS files Converting also non-Latin text like: Arabic Cyrillic Chinese Greece and Hebrew. Links found in the converted files will be followed. Debug mode Offering detailed information during index/re-index: New links keywords frames and media found per link. To be . . .
. . .
Included for 33 languages. Common word lists holding stop words. Included for 25 languages Admin selectable for: Arabic Bengali Bulgarian Catalan Chinese Cyrillic Czech Danish Dutch English Farsi Finnish French Greek German Hindi Hungarian Italian Norwegian Polish Portuguese Romanian Spanish Swedish and Turkish. . . .
. . .
links outside are followed. Multiple and nested divs will be attended. Do not index parts of a page defined by HTML5 elements <tag> . . . </tag> Foreseen to cooperate with the HTML5 elements like: section nav aside hgroup article header footer Vice versa function also included in order to index only parts of a page between . . .
. . .
Extension implemented SQLi connector implemented between PHP and a MySQL database. Performed by OOP also PHP v.5.5 is supported. Compatible with MySQL and MariaDB Proven up to: - MySQL version 8.0.32 - MariaDB version 10.4.28 sp_executesql Ready to run in PHP 8 environment Latest version of Sphider-plus version 4.2024a is proven up to PHP . . .
. . .
8.3.2 [ Proven ] Successfully implemented as search engine on a customer site with a database capacity such as: 25.206 sites 324.595 page links 1.260.698 keywords 169.251 media links. Imprint Private Notice Private Policy . . .


Sphider-plus is a search engine, based on the scripts of original Sphider. [ About Sphider-plus] More than 400 new features (additional mods, functions, template designs and debugging) have been added to the original Sphider. For details about all the improvements and changes, please read the Documentation section. [ Main features ] Item . . .
. . .
section. [ Main features ] Item Description UTF-8 and UTF-16 support Indexation and search procedure for Chinese, Cyrillic, Georgian, Hebrew etc. charsets. UNICODE support including astral symbols. Support for non-ASCII domains 'Internationalized Domain Names' (IDN) like 'http://президент.рф/' and 'http://müller.de/' are accepted and . . .
. . .
'http://müller.de/' are accepted and processed. Responsive design Automatically adapting the size of search form, result listing and addurl form to display size of computers, tablets, smartphones, etc. Media support Index and search for images (incl. Open Graph images), audio and video (incl. Youtube videos). EXIF and ID3 information are also . . .
. . .
player software. Multiple database support Individual configuration and activation of databases for 'Admin', 'Search User' and 'Suggest URL'. Support of multiple table sets in each db, MySQL query cache, individual index for each db, individual or bulk search in predefined databases. Individual Admin settings for each db and each set of . . .
. . .
always in cache. Separate caches for text and media results. Admin configurable. Follow sitemap files If available, sitemap.xml as well as gzip compressed files will be used to follow the links of a site. If <sitemapindex . . . > is detected, also multiple sitemap files are processed. Periodical Re-index Re-indexing could be performed . . .
. . .
also multiple sitemap files are processed. Periodical Re-index Re-indexing could be performed automatically, repeated every selected time interval. Admin selectable intervals for 3 hours, 12 hours, 1 day, 1 week, or 1 month. Multithreaded indexing In order to reduce the time for indexing, 1-10 parallel running threads might be activated . . .
. . .
1-10 parallel running threads might be activated in Admin settings. Preferred re-index While invoking this option, the admin may select a suitable level for the next index procedure. Thus, only those URLs, containing the according level, will be re-indexed.. Erase Re-index and Continue suspended index procedures Individual (site specific) or . . .
. . .
(site specific) or bulk update of database. Support of XML product feeds Index and search of feed content, inclusive formatting the search results. RDF, RSD, RSS and Atom feed support Index and search of feed content, inclusive RDF 'Dublin Core' tags. Obey / ignore 'preferred' tags in RSD feeds. Follow / ignore CDATA directives. Various . . .
. . .
ignore 'preferred' tags in RSD feeds. Follow / ignore CDATA directives. Various search modes Search with wildcards, Tolerant search, Search strict, Search only in one domain, Search all links of a site, Search for media (link-specific). Add thumbnails to each page presented in text results Admin selectable, this feature will present a web shot . . .
. . .
suspected to contain malware or phishing content. 11 different modes of sorting the text results Admin selectable: -By relevance (weight % ) -By hit counts in full text -Most popular links on top -By indexdate -By URL names -By file suffix -Main URL (domain) on top -Like Google (Top 2 per URL) - Promoted domain on top - Links holding promoted . . .
. . .
on top - Links holding promoted catchwords on top. 5 different modes of sorting the media results Admin selectable: -By title(alphabetic) -By file suffix -By image size -By 'Last queried' -By 'Most popular'. Same results for queries typed with pure vowels, or with accents Will deliver the same results for queries like: caf e and caf é . To be . . .
. . .
in Admin backend. Same results for queries with and without quotes Will deliver the same results for queries like: d'information <-> information dei'largi <-> largi Also Admin selectabe: Equalize the different quotes like: ' ` ´ Same text results for queries with and without ligatures Admin selectable; will deliver the same results . . .
. . .
results for queries with and without ligatures Admin selectable; will deliver the same results for queries like: cœur and coeur . Worked out for Latin ligatures in Unicode (Latin-derived alphabets) and also ligatures used only in phonetic transcription, but not taking into consideration medieval ligatures. Present all results for singular and . . .
. . .
all will become searchable. Dictionaries with 106.800 radicals. Segmentation of Japanese words Segmentation of 5.724 kanji (new, old and half width), hiragana, katakana and jinmeiyo Japanese character writing systems. Transliterate Latin characters into their Greek equivalents Transforms query input alla to find ἀλλὰ and baptismatos to find . . .
. . .
redirections caused by HTTP 301, 302, 303 and 307 status codes. Also obeying JavaScript, sent as HTML content like: <SCRIPT language="javascript"> win.loc="mp.php?mcv=59";</SCRIPT> Follow header redirections, refresh tags and canonical links Automatical forwarding for the indexer. Follow links found in JavaScript and index also the . . .
. . .
and index also the content of document.write Will index JavaScript commands. Detect and follow links like: document.write(' <a href="new12.pdf">All news 2012</a> '); and index the content of: document.write(' this content '); Not indexing content created in real-time by JavaScript. Accept gzip formatted transmission In order . . .
. . .
Converter included for PDF, DOCX, XLSX, ODT, ODS, CSV, PPTX and XLS files Converting also non-Latin text like: Arabic, Cyrillic, Chinese, Greece and Hebrew. Links found in the converted files will be followed. Debug mode Offering detailed information during index/re-index: New links, keywords, frames and media found per link. To be . . .
. . .
Included for 33 languages. Common word lists holding stop words. Included for 25 languages Admin selectable for: Arabic, Bengali, Bulgarian, Catalan, Chinese, Cyrillic, Czech, Danish, Dutch, English, Farsi, Finnish, French, Greek, German, Hindi, Hungarian, Italian, Norwegian, Polish, Portuguese, Romanian, Spanish, Swedish and Turkish. . . .
. . .
links outside are followed. Multiple and nested divs will be attended. Do not index parts of a page defined by HTML5 elements <tag> . . . </tag> Foreseen to cooperate with the HTML5 elements like: section, nav, aside, hgroup, article, header, footer Vice versa function also included in order to index only parts of a page between . . .
. . .
Extension implemented SQLi connector implemented between PHP and a MySQL database. Performed by OOP, also PHP v.5.5 is supported. Compatible with MySQL and MariaDB Proven up to: - MySQL version 8.0.32 - MariaDB version 10.4.28 sp_executesql Ready to run in PHP 8 environment Latest version of Sphider-plus version 4.2024a is proven up to PHP . . .
. . .
8.3.2 [ Proven ] Successfully implemented as search engine on a customer site with a database capacity such as: 25.206 sites 32459595 page links 1.260.698 keywords 169.251 media links. Imprint Private Notice Private Policy . . .


Sphider-plus is a search engine, based on the scripts of original Sphider. [ About Sphider-plus] More than 400 new features (additional mods, functions, template designs and debugging) have been added to the original Sphider. For details about all the improvements and changes, please read the Documentation section. [ Main features ] Item . . .
. . .
section. [ Main features ] Item Description UTF-8 and UTF-16 support Indexation and search procedure for Chinese, Cyrillic, Georgian, Hebrew etc. charsets. UNICODE support including astral symbols. Support for non-ASCII domains 'Internationalized Domain Names' (IDN) like 'http://президент.рф/' and 'http://müller.de/' are accepted and . . .
. . .
'http://müller.de/' are accepted and processed. Responsive design Automatically adapting the size of search form, result listing and addurl form to display size of computers, tablets, smartphones, etc. Media support Index and search for images (incl. Open Graph images), audio and video (incl. Youtube videos). EXIF and ID3 information are also . . .
. . .
player software. Multiple database support Individual configuration and activation of databases for 'Admin', 'Search User' and 'Suggest URL'. Support of multiple table sets in each db, MySQL query cache, individual index for each db, individual or bulk search in predefined databases. Individual Admin settings for each db and each set of . . .
. . .
always in cache. Separate caches for text and media results. Admin configurable. Follow sitemap files If available, sitemap.xml as well as gzip compressed files will be used to follow the links of a site. If <sitemapindex . . . > is detected, also multiple sitemap files are processed. Periodical Re-index Re-indexing could be performed . . .
. . .
also multiple sitemap files are processed. Periodical Re-index Re-indexing could be performed automatically, repeated every selected time interval. Admin selectable intervals for 3 hours, 12 hours, 1 day, 1 week, or 1 month. Multithreaded indexing In order to reduce the time for indexing, 1-10 parallel running threads might be activated . . .
. . .
1-10 parallel running threads might be activated in Admin settings. Preferred re-index While invoking this option, the admin may select a suitable level for the next index procedure. Thus, only those URLs, containing the according level, will be re-indexed.. Erase Re-index and Continue suspended index procedures Individual (site specific) or . . .
. . .
(site specific) or bulk update of database. Support of XML product feeds Index and search of feed content, inclusive formatting the search results. RDF, RSD, RSS and Atom feed support Index and search of feed content, inclusive RDF 'Dublin Core' tags. Obey / ignore 'preferred' tags in RSD feeds. Follow / ignore CDATA directives. Various . . .
. . .
ignore 'preferred' tags in RSD feeds. Follow / ignore CDATA directives. Various search modes Search with wildcards, Tolerant search, Search strict, Search only in one domain, Search all links of a site, Search for media (link-specific). Add thumbnails to each page presented in text results Admin selectable, this feature will present a web shot . . .
. . .
suspected to contain malware or phishing content. 11 different modes of sorting the text results Admin selectable: -By relevance (weight % ) -By hit counts in full text -Most popular links on top -By indexdate -By URL names -By file suffix -Main URL (domain) on top -Like Google (Top 2 per URL) - Promoted domain on top - Links holding promoted . . .
. . .
on top - Links holding promoted catchwords on top. 5 different modes of sorting the media results Admin selectable: -By title(alphabetic) -By file suffix -By image size -By 'Last queried' -By 'Most popular'. Same results for queries typed with pure vowels, or with accents Will deliver the same results for queries like: caf e and caf é . To be . . .
. . .
in Admin backend. Same results for queries with and without quotes Will deliver the same results for queries like: d'information <-> information dei'largi <-> largi Also Admin selectabe: Equalize the different quotes like: ' ` ´ Same text results for queries with and without ligatures Admin selectable; will deliver the same results . . .
. . .
results for queries with and without ligatures Admin selectable; will deliver the same results for queries like: cœur and coeur . Worked out for Latin ligatures in Unicode (Latin-derived alphabets) and also ligatures used only in phonetic transcription, but not taking into consideration medieval ligatures. Present all results for singular and . . .
. . .
all will become searchable. Dictionaries with 106.800 radicals. Segmentation of Japanese words Segmentation of 5.724 kanji (new, old and half width), hiragana, katakana and jinmeiyo Japanese character writing systems. Transliterate Latin characters into their Greek equivalents Transforms query input alla to find ἀλλὰ and baptismatos to find . . .
. . .
redirections caused by HTTP 301, 302, 303 and 307 status codes. Also obeying JavaScript, sent as HTML content like: <SCRIPT language="javascript"> win.loc="mp.php?mcv=59";</SCRIPT> Follow header redirections, refresh tags and canonical links Automatical forwarding for the indexer. Follow links found in JavaScript and index also the . . .
. . .
and index also the content of document.write Will index JavaScript commands. Detect and follow links like: document.write(' <a href="new12.pdf">All news 2012</a> '); and index the content of: document.write(' this content '); Not indexing content created in real-time by JavaScript. Accept gzip formatted transmission In order . . .
. . .
Converter included for PDF, DOCX, XLSX, ODT, ODS, CSV, PPTX and XLS files Converting also non-Latin text like: Arabic, Cyrillic, Chinese, Greece and Hebrew. Links found in the converted files will be followed. Debug mode Offering detailed information during index/re-index: New links, keywords, frames and media found per link. To be . . .
. . .
Included for 33 languages. Common word lists holding stop words. Included for 25 languages Admin selectable for: Arabic, Bengali, Bulgarian, Catalan, Chinese, Cyrillic, Czech, Danish, Dutch, English, Farsi, Finnish, French, Greek, German, Hindi, Hungarian, Italian, Norwegian, Polish, Portuguese, Romanian, Spanish, Swedish and Turkish. . . .
. . .
links outside are followed. Multiple and nested divs will be attended. Do not index parts of a page defined by HTML5 elements <tag> . . . </tag> Foreseen to cooperate with the HTML5 elements like: section, nav, aside, hgroup, article, header, footer Vice versa function also included in order to index only parts of a page between . . .
. . .
Extension implemented SQLi connector implemented between PHP and a MySQL database. Performed by OOP, also PHP v.55 is supported. Compatible with MySQL and MariaDB Proven up to: - MySQL version 8.0.32 - MariaDB version 10.4.28 sp_executesql Ready to run in PHP 8 environment Latest version of Sphider-plus version 4.2024a is proven up to PHP . . .
. . .
8.3.2 [ Proven ] Successfully implemented as search engine on a customer site with a database capacity such as: 25.206 sites 324.595 page links 1.260.698 keywords 169.251 media links. Imprint Private Notice Private Policy . . .


Sphider-plus is a search engine, based on the scripts of original Sphider. [ About Sphider-plus] More than 400 new features (additional mods, functions, template designs and debugging) have been added to the original Sphider. For details about all the improvements and changes, please read the Documentation section. [ Main features ] Item . . .
. . .
section. [ Main features ] Item Description UTF-8 and UTF-16 support Indexation and search procedure for Chinese, Cyrillic, Georgian, Hebrew etc. charsets. UNICODE support including astral symbols. Support for non-ASCII domains 'Internationalized Domain Names' (IDN) like 'http://президент.рф/' and 'http://müller.de/' are accepted and . . .
. . .
'http://müller.de/' are accepted and processed. Responsive design Automatically adapting the size of search form, result listing and addurl form to display size of computers, tablets, smartphones, etc. Media support Index and search for images (incl. Open Graph images), audio and video (incl. Youtube videos). EXIF and ID3 information are also . . .
. . .
player software. Multiple database support Individual configuration and activation of databases for 'Admin', 'Search User' and 'Suggest URL'. Support of multiple table sets in each db, MySQL query cache, individual index for each db, individual or bulk search in predefined databases. Individual Admin settings for each db and each set of . . .
. . .
always in cache. Separate caches for text and media results. Admin configurable. Follow sitemap files If available, sitemap.xml as well as gzip compressed files will be used to follow the links of a site. If <sitemapindex . . . > is detected, also multiple sitemap files are processed. Periodical Re-index Re-indexing could be performed . . .
. . .
also multiple sitemap files are processed. Periodical Re-index Re-indexing could be performed automatically, repeated every selected time interval. Admin selectable intervals for 3 hours, 12 hours, 1 day, 1 week, or 1 month. Multithreaded indexing In order to reduce the time for indexing, 1-10 parallel running threads might be activated . . .
. . .
1-10 parallel running threads might be activated in Admin settings. Preferred re-index While invoking this option, the admin may select a suitable level for the next index procedure. Thus, only those URLs, containing the according level, will be re-indexed.. Erase Re-index and Continue suspended index procedures Individual (site specific) or . . .
. . .
(site specific) or bulk update of database. Support of XML product feeds Index and search of feed content, inclusive formatting the search results. RDF, RSD, RSS and Atom feed support Index and search of feed content, inclusive RDF 'Dublin Core' tags. Obey / ignore 'preferred' tags in RSD feeds. Follow / ignore CDATA directives. Various . . .
. . .
ignore 'preferred' tags in RSD feeds. Follow / ignore CDATA directives. Various search modes Search with wildcards, Tolerant search, Search strict, Search only in one domain, Search all links of a site, Search for media (link-specific). Add thumbnails to each page presented in text results Admin selectable, this feature will present a web shot . . .
. . .
suspected to contain malware or phishing content. 11 different modes of sorting the text results Admin selectable: -By relevance (weight % ) -By hit counts in full text -Most popular links on top -By indexdate -By URL names -By file suffix -Main URL (domain) on top -Like Google (Top 2 per URL) - Promoted domain on top - Links holding promoted . . .
. . .
on top - Links holding promoted catchwords on top. 5 different modes of sorting the media results Admin selectable: -By title(alphabetic) -By file suffix -By image size -By 'Last queried' -By 'Most popular'. Same results for queries typed with pure vowels, or with accents Will deliver the same results for queries like: caf e and caf é . To be . . .
. . .
in Admin backend. Same results for queries with and without quotes Will deliver the same results for queries like: d'information <-> information dei'largi <-> largi Also Admin selectabe: Equalize the different quotes like: ' ` ´ Same text results for queries with and without ligatures Admin selectable; will deliver the same results . . .
. . .
results for queries with and without ligatures Admin selectable; will deliver the same results for queries like: cœur and coeur . Worked out for Latin ligatures in Unicode (Latin-derived alphabets) and also ligatures used only in phonetic transcription, but not taking into consideration medieval ligatures. Present all results for singular and . . .
. . .
all will become searchable. Dictionaries with 106.800 radicals. Segmentation of Japanese words Segmentation of 572424 kanji (new, old and half width), hiragana, katakana and jinmeiyo Japanese character writing systems. Transliterate Latin characters into their Greek equivalents Transforms query input alla to find ἀλλὰ and baptismatos to find . . .
. . .
redirections caused by HTTP 301, 302, 303 and 307 status codes. Also obeying JavaScript, sent as HTML content like: <SCRIPT language="javascript"> win.loc="mp.php?mcv=59";</SCRIPT> Follow header redirections, refresh tags and canonical links Automatical forwarding for the indexer. Follow links found in JavaScript and index also the . . .
. . .
and index also the content of document.write Will index JavaScript commands. Detect and follow links like: document.write(' <a href="new12.pdf">All news 2012</a> '); and index the content of: document.write(' this content '); Not indexing content created in real-time by JavaScript. Accept gzip formatted transmission In order . . .
. . .
Converter included for PDF, DOCX, XLSX, ODT, ODS, CSV, PPTX and XLS files Converting also non-Latin text like: Arabic, Cyrillic, Chinese, Greece and Hebrew. Links found in the converted files will be followed. Debug mode Offering detailed information during index/re-index: New links, keywords, frames and media found per link. To be . . .
. . .
Included for 33 languages. Common word lists holding stop words. Included for 25 languages Admin selectable for: Arabic, Bengali, Bulgarian, Catalan, Chinese, Cyrillic, Czech, Danish, Dutch, English, Farsi, Finnish, French, Greek, German, Hindi, Hungarian, Italian, Norwegian, Polish, Portuguese, Romanian, Spanish, Swedish and Turkish. . . .
. . .
links outside are followed. Multiple and nested divs will be attended. Do not index parts of a page defined by HTML5 elements <tag> . . . </tag> Foreseen to cooperate with the HTML5 elements like: section, nav, aside, hgroup, article, header, footer Vice versa function also included in order to index only parts of a page between . . .
. . .
Extension implemented SQLi connector implemented between PHP and a MySQL database. Performed by OOP, also PHP v.5.5 is supported. Compatible with MySQL and MariaDB Proven up to: - MySQL version 8.0.32 - MariaDB version 10.4.28 sp_executesql Ready to run in PHP 8 environment Latest version of Sphider-plus version 4.2024a is proven up to PHP . . .
. . .
8.3.2 [ Proven ] Successfully implemented as search engine on a customer site with a database capacity such as: 25.206 sites 324.595 page links 1.260.698 keywords 169.251 media links. Imprint Private Notice Private Policy . . .


Sphider-plus is a search engine, based on the scripts of original Sphider. [ About Sphider-plus] More than 400 new features (additional mods, functions, template designs and debugging) have been added to the original Sphider. For details about all the improvements and changes, please read the Documentation section. [ Main features ] Item . . .
. . .
section. [ Main features ] Item Description UTF-8 and UTF-16 support Indexation and search procedure for Chinese, Cyrillic, Georgian, Hebrew etc. charsets. UNICODE support including astral symbols. Support for non-ASCII domains 'Internationalized Domain Names' (IDN) like 'http://президент.рф/' and 'http://müller.de/' are accepted and . . .
. . .
'http://müller.de/' are accepted and processed. Responsive design Automatically adapting the size of search form, result listing and addurl form to display size of computers, tablets, smartphones, etc. Media support Index and search for images (incl. Open Graph images), audio and video (incl. Youtube videos). EXIF and ID3 information are also . . .
. . .
player software. Multiple database support Individual configuration and activation of databases for 'Admin', 'Search User' and 'Suggest URL'. Support of multiple table sets in each db, MySQL query cache, individual index for each db, individual or bulk search in predefined databases. Individual Admin settings for each db and each set of . . .
. . .
always in cache. Separate caches for text and media results. Admin configurable. Follow sitemap files If available, sitemap.xml as well as gzip compressed files will be used to follow the links of a site. If <sitemapindex . . . > is detected, also multiple sitemap files are processed. Periodical Re-index Re-indexing could be performed . . .
. . .
also multiple sitemap files are processed. Periodical Re-index Re-indexing could be performed automatically, repeated every selected time interval. Admin selectable intervals for 3 hours, 12 hours, 1 day, 1 week, or 1 month. Multithreaded indexing In order to reduce the time for indexing, 1-10 parallel running threads might be activated . . .
. . .
1-10 parallel running threads might be activated in Admin settings. Preferred re-index While invoking this option, the admin may select a suitable level for the next index procedure. Thus, only those URLs, containing the according level, will be re-indexed.. Erase Re-index and Continue suspended index procedures Individual (site specific) or . . .
. . .
(site specific) or bulk update of database. Support of XML product feeds Index and search of feed content, inclusive formatting the search results. RDF, RSD, RSS and Atom feed support Index and search of feed content, inclusive RDF 'Dublin Core' tags. Obey / ignore 'preferred' tags in RSD feeds. Follow / ignore CDATA directives. Various . . .
. . .
ignore 'preferred' tags in RSD feeds. Follow / ignore CDATA directives. Various search modes Search with wildcards, Tolerant search, Search strict, Search only in one domain, Search all links of a site, Search for media (link-specific). Add thumbnails to each page presented in text results Admin selectable, this feature will present a web shot . . .
. . .
suspected to contain malware or phishing content. 11 different modes of sorting the text results Admin selectable: -By relevance (weight % ) -By hit counts in full text -Most popular links on top -By indexdate -By URL names -By file suffix -Main URL (domain) on top -Like Google (Top 2 per URL) - Promoted domain on top - Links holding promoted . . .
. . .
on top - Links holding promoted catchwords on top. 5 different modes of sorting the media results Admin selectable: -By title(alphabetic) -By file suffix -By image size -By 'Last queried' -By 'Most popular'. Same results for queries typed with pure vowels, or with accents Will deliver the same results for queries like: caf e and caf é . To be . . .
. . .
in Admin backend. Same results for queries with and without quotes Will deliver the same results for queries like: d'information <-> information dei'largi <-> largi Also Admin selectabe: Equalize the different quotes like: ' ` ´ Same text results for queries with and without ligatures Admin selectable; will deliver the same results . . .
. . .
results for queries with and without ligatures Admin selectable; will deliver the same results for queries like: cœur and coeur . Worked out for Latin ligatures in Unicode (Latin-derived alphabets) and also ligatures used only in phonetic transcription, but not taking into consideration medieval ligatures. Present all results for singular and . . .
. . .
all will become searchable. Dictionaries with 106.800 radicals. Segmentation of Japanese words Segmentation of 5724 kanji (new, old and half width), hiragana, katakana and jinmeiyo Japanese character writing systems. Transliterate Latin characters into their Greek equivalents Transforms query input alla to find ἀλλὰ and baptismatos to find . . .
. . .
redirections caused by HTTP 301, 302, 303 and 307 status codes. Also obeying JavaScript, sent as HTML content like: <SCRIPT language="javascript"> win.loc="mp.php?mcv=59";</SCRIPT> Follow header redirections, refresh tags and canonical links Automatical forwarding for the indexer. Follow links found in JavaScript and index also the . . .
. . .
and index also the content of document.write Will index JavaScript commands. Detect and follow links like: document.write(' <a href="new12.pdf">All news 2012</a> '); and index the content of: document.write(' this content '); Not indexing content created in real-time by JavaScript. Accept gzip formatted transmission In order . . .
. . .
Converter included for PDF, DOCX, XLSX, ODT, ODS, CSV, PPTX and XLS files Converting also non-Latin text like: Arabic, Cyrillic, Chinese, Greece and Hebrew. Links found in the converted files will be followed. Debug mode Offering detailed information during index/re-index: New links, keywords, frames and media found per link. To be . . .
. . .
Included for 33 languages. Common word lists holding stop words. Included for 25 languages Admin selectable for: Arabic, Bengali, Bulgarian, Catalan, Chinese, Cyrillic, Czech, Danish, Dutch, English, Farsi, Finnish, French, Greek, German, Hindi, Hungarian, Italian, Norwegian, Polish, Portuguese, Romanian, Spanish, Swedish and Turkish. . . .
. . .
links outside are followed. Multiple and nested divs will be attended. Do not index parts of a page defined by HTML5 elements <tag> . . . </tag> Foreseen to cooperate with the HTML5 elements like: section, nav, aside, hgroup, article, header, footer Vice versa function also included in order to index only parts of a page between . . .
. . .
Extension implemented SQLi connector implemented between PHP and a MySQL database. Performed by OOP, also PHP v.5.5 is supported. Compatible with MySQL and MariaDB Proven up to: - MySQL version 8.0.32 - MariaDB version 10.4.28 sp_executesql Ready to run in PHP 8 environment Latest version of Sphider-plus version 4.2024a is proven up to PHP . . .
. . .
8.3.2 [ Proven ] Successfully implemented as search engine on a customer site with a database capacity such as: 25.206 sites 324.595 page links 1.260.698 keywords 169.251 media links. Imprint Private Notice Private Policy . . .


Sphider-plus is a search engine, based on the scripts of original Sphider. [ About Sphider-plus] More than 400 new features (additional mods, functions, template designs and debugging) have been added to the original Sphider. For details about all the improvements and changes, please read the Documentation section. [ Main features ] Item . . .
. . .
section. [ Main features ] Item Description UTF-8 and UTF-16 support Indexation and search procedure for Chinese, Cyrillic, Georgian, Hebrew etc. charsets. UNICODE support including astral symbols. Support for non-ASCII domains 'Internationalized Domain Names' (IDN) like 'http://президент.рф/' and 'http://müller.de/' are accepted and . . .
. . .
'http://müller.de/' are accepted and processed. Responsive design Automatically adapting the size of search form, result listing and addurl form to display size of computers, tablets, smartphones, etc. Media support Index and search for images (incl. Open Graph images), audio and video (incl. Youtube videos). EXIF and ID3 information are also . . .
. . .
player software. Multiple database support Individual configuration and activation of databases for 'Admin', 'Search User' and 'Suggest URL'. Support of multiple table sets in each db, MySQL query cache, individual index for each db, individual or bulk search in predefined databases. Individual Admin settings for each db and each set of . . .
. . .
always in cache. Separate caches for text and media results. Admin configurable. Follow sitemap files If available, sitemap.xml as well as gzip compressed files will be used to follow the links of a site. If <sitemapindex . . . > is detected, also multiple sitemap files are processed. Periodical Re-index Re-indexing could be performed . . .
. . .
also multiple sitemap files are processed. Periodical Re-index Re-indexing could be performed automatically, repeated every selected time interval. Admin selectable intervals for 3 hours, 12 hours, 1 day, 1 week, or 1 month. Multithreaded indexing In order to reduce the time for indexing, 1-10 parallel running threads might be activated . . .
. . .
1-10 parallel running threads might be activated in Admin settings. Preferred re-index While invoking this option, the admin may select a suitable level for the next index procedure. Thus, only those URLs, containing the according level, will be re-indexed.. Erase Re-index and Continue suspended index procedures Individual (site specific) or . . .
. . .
(site specific) or bulk update of database. Support of XML product feeds Index and search of feed content, inclusive formatting the search results. RDF, RSD, RSS and Atom feed support Index and search of feed content, inclusive RDF 'Dublin Core' tags. Obey / ignore 'preferred' tags in RSD feeds. Follow / ignore CDATA directives. Various . . .
. . .
ignore 'preferred' tags in RSD feeds. Follow / ignore CDATA directives. Various search modes Search with wildcards, Tolerant search, Search strict, Search only in one domain, Search all links of a site, Search for media (link-specific). Add thumbnails to each page presented in text results Admin selectable, this feature will present a web shot . . .
. . .
suspected to contain malware or phishing content. 11 different modes of sorting the text results Admin selectable: -By relevance (weight % ) -By hit counts in full text -Most popular links on top -By indexdate -By URL names -By file suffix -Main URL (domain) on top -Like Google (Top 2 per URL) - Promoted domain on top - Links holding promoted . . .
. . .
on top - Links holding promoted catchwords on top. 5 different modes of sorting the media results Admin selectable: -By title(alphabetic) -By file suffix -By image size -By 'Last queried' -By 'Most popular'. Same results for queries typed with pure vowels, or with accents Will deliver the same results for queries like: caf e and caf é . To be . . .
. . .
in Admin backend. Same results for queries with and without quotes Will deliver the same results for queries like: d'information <-> information dei'largi <-> largi Also Admin selectabe: Equalize the different quotes like: ' ` ´ Same text results for queries with and without ligatures Admin selectable; will deliver the same results . . .
. . .
results for queries with and without ligatures Admin selectable; will deliver the same results for queries like: cœur and coeur . Worked out for Latin ligatures in Unicode (Latin-derived alphabets) and also ligatures used only in phonetic transcription, but not taking into consideration medieval ligatures. Present all results for singular and . . .
. . .
all will become searchable. Dictionaries with 106.800 radicals. Segmentation of Japanese words Segmentation of 5.724 kanji (new, old and half width), hiragana, katakana and jinmeiyo Japanese character writing systems. Transliterate Latin characters into their Greek equivalents Transforms query input alla to find ἀλλὰ and baptismatos to find . . .
. . .
redirections caused by HTTP 301, 302, 303 and 307 status codes. Also obeying JavaScript, sent as HTML content like: <SCRIPT language="javascript"> win.loc="mp.php?mcv=59";</SCRIPT> Follow header redirections, refresh tags and canonical links Automatical forwarding for the indexer. Follow links found in JavaScript and index also the . . .
. . .
and index also the content of document.write Will index JavaScript commands. Detect and follow links like: document.write(' <a href="new12.pdf">All news 2012</a> '); and index the content of: document.write(' this content '); Not indexing content created in real-time by JavaScript. Accept gzip formatted transmission In order . . .
. . .
Converter included for PDF, DOCX, XLSX, ODT, ODS, CSV, PPTX and XLS files Converting also non-Latin text like: Arabic, Cyrillic, Chinese, Greece and Hebrew. Links found in the converted files will be followed. Debug mode Offering detailed information during index/re-index: New links, keywords, frames and media found per link. To be . . .
. . .
Included for 33 languages. Common word lists holding stop words. Included for 25 languages Admin selectable for: Arabic, Bengali, Bulgarian, Catalan, Chinese, Cyrillic, Czech, Danish, Dutch, English, Farsi, Finnish, French, Greek, German, Hindi, Hungarian, Italian, Norwegian, Polish, Portuguese, Romanian, Spanish, Swedish and Turkish. . . .
. . .
links outside are followed. Multiple and nested divs will be attended. Do not index parts of a page defined by HTML5 elements <tag> . . . </tag> Foreseen to cooperate with the HTML5 elements like: section, nav, aside, hgroup, article, header, footer Vice versa function also included in order to index only parts of a page between . . .
. . .
Extension implemented SQLi connector implemented between PHP and a MySQL database. Performed by OOP, also PHP v.5.5 is supported. Compatible with MySQL and MariaDB Proven up to: - MySQL version 8.0.32 - MariaDB version 10.4.28 sp_executesql Ready to run in PHP 8 environment Latest version of Sphider-plus version 4.2024a is proven up to PHP . . .
. . .
8.3.2 [ Proven ] Successfully implemented as search engine on a customer site with a database capacity such as: 25206 sites 324.595 page links 1.260.698 keywords 169.251 media links. Imprint Private Notice Private Policy . . .


Sphider-plus is a search engine, based on the scripts of original Sphider. [ About Sphider-plus] More than 400 new features (additional mods, functions, template designs and debugging) have been added to the original Sphider. For details about all the improvements and changes, please read the Documentation section. [ Main features ] Item . . .
. . .
section. [ Main features ] Item Description UTF-8 and UTF-16 support Indexation and search procedure for Chinese, Cyrillic, Georgian, Hebrew etc. charsets. UNICODE support including astral symbols. Support for non-ASCII domains 'Internationalized Domain Names' (IDN) like 'http://президент.рф/' and 'http://müller.de/' are accepted and . . .
. . .
'http://müller.de/' are accepted and processed. Responsive design Automatically adapting the size of search form, result listing and addurl form to display size of computers, tablets, smartphones, etc. Media support Index and search for images (incl. Open Graph images), audio and video (incl. Youtube videos). EXIF and ID3 information are also . . .
. . .
player software. Multiple database support Individual configuration and activation of databases for 'Admin', 'Search User' and 'Suggest URL'. Support of multiple table sets in each db, MySQL query cache, individual index for each db, individual or bulk search in predefined databases. Individual Admin settings for each db and each set of . . .
. . .
always in cache. Separate caches for text and media results. Admin configurable. Follow sitemap files If available, sitemap.xml as well as gzip compressed files will be used to follow the links of a site. If <sitemapindex . . . > is detected, also multiple sitemap files are processed. Periodical Re-index Re-indexing could be performed . . .
. . .
also multiple sitemap files are processed. Periodical Re-index Re-indexing could be performed automatically, repeated every selected time interval. Admin selectable intervals for 3 hours, 12 hours, 1 day, 1 week, or 1 month. Multithreaded indexing In order to reduce the time for indexing, 1-10 parallel running threads might be activated . . .
. . .
1-10 parallel running threads might be activated in Admin settings. Preferred re-index While invoking this option, the admin may select a suitable level for the next index procedure. Thus, only those URLs, containing the according level, will be re-indexed.. Erase Re-index and Continue suspended index procedures Individual (site specific) or . . .
. . .
(site specific) or bulk update of database. Support of XML product feeds Index and search of feed content, inclusive formatting the search results. RDF, RSD, RSS and Atom feed support Index and search of feed content, inclusive RDF 'Dublin Core' tags. Obey / ignore 'preferred' tags in RSD feeds. Follow / ignore CDATA directives. Various . . .
. . .
ignore 'preferred' tags in RSD feeds. Follow / ignore CDATA directives. Various search modes Search with wildcards, Tolerant search, Search strict, Search only in one domain, Search all links of a site, Search for media (link-specific). Add thumbnails to each page presented in text results Admin selectable, this feature will present a web shot . . .
. . .
suspected to contain malware or phishing content. 11 different modes of sorting the text results Admin selectable: -By relevance (weight % ) -By hit counts in full text -Most popular links on top -By indexdate -By URL names -By file suffix -Main URL (domain) on top -Like Google (Top 2 per URL) - Promoted domain on top - Links holding promoted . . .
. . .
on top - Links holding promoted catchwords on top. 5 different modes of sorting the media results Admin selectable: -By title(alphabetic) -By file suffix -By image size -By 'Last queried' -By 'Most popular'. Same results for queries typed with pure vowels, or with accents Will deliver the same results for queries like: caf e and caf é . To be . . .
. . .
in Admin backend. Same results for queries with and without quotes Will deliver the same results for queries like: d'information <-> information dei'largi <-> largi Also Admin selectabe: Equalize the different quotes like: ' ` ´ Same text results for queries with and without ligatures Admin selectable; will deliver the same results . . .
. . .
results for queries with and without ligatures Admin selectable; will deliver the same results for queries like: cœur and coeur . Worked out for Latin ligatures in Unicode (Latin-derived alphabets) and also ligatures used only in phonetic transcription, but not taking into consideration medieval ligatures. Present all results for singular and . . .
. . .
all will become searchable. Dictionaries with 106.800 radicals. Segmentation of Japanese words Segmentation of 5.724 kanji (new, old and half width), hiragana, katakana and jinmeiyo Japanese character writing systems. Transliterate Latin characters into their Greek equivalents Transforms query input alla to find ἀλλὰ and baptismatos to find . . .
. . .
redirections caused by HTTP 301, 302, 303 and 307 status codes. Also obeying JavaScript, sent as HTML content like: <SCRIPT language="javascript"> win.loc="mp.php?mcv=59";</SCRIPT> Follow header redirections, refresh tags and canonical links Automatical forwarding for the indexer. Follow links found in JavaScript and index also the . . .
. . .
and index also the content of document.write Will index JavaScript commands. Detect and follow links like: document.write(' <a href="new12.pdf">All news 2012</a> '); and index the content of: document.write(' this content '); Not indexing content created in real-time by JavaScript. Accept gzip formatted transmission In order . . .
. . .
Converter included for PDF, DOCX, XLSX, ODT, ODS, CSV, PPTX and XLS files Converting also non-Latin text like: Arabic, Cyrillic, Chinese, Greece and Hebrew. Links found in the converted files will be followed. Debug mode Offering detailed information during index/re-index: New links, keywords, frames and media found per link. To be . . .
. . .
Included for 33 languages. Common word lists holding stop words. Included for 25 languages Admin selectable for: Arabic, Bengali, Bulgarian, Catalan, Chinese, Cyrillic, Czech, Danish, Dutch, English, Farsi, Finnish, French, Greek, German, Hindi, Hungarian, Italian, Norwegian, Polish, Portuguese, Romanian, Spanish, Swedish and Turkish. . . .
. . .
links outside are followed. Multiple and nested divs will be attended. Do not index parts of a page defined by HTML5 elements <tag> . . . </tag> Foreseen to cooperate with the HTML5 elements like: section, nav, aside, hgroup, article, header, footer Vice versa function also included in order to index only parts of a page between . . .
. . .
Extension implemented SQLi connector implemented between PHP and a MySQL database. Performed by OOP, also PHP v.5.5 is supported. Compatible with MySQL and MariaDB Proven up to: - MySQL version 8.0.32 - MariaDB version 10.4.28 sp_executesql Ready to run in PHP 8 environment Latest version of Sphider-plus version 4.2024a is proven up to PHP . . .
. . .
8.3.2 [ Proven ] Successfully implemented as search engine on a customer site with a database capacity such as: 25.206 sites 324.595 page links 1.260.698 keywords 169251 media links. Imprint Private Notice Private Policy . . .


Sphider-plus is a search engine, based on the scripts of original Sphider. [ About Sphider-plus] More than 400 new features (additional mods, functions, template designs and debugging) have been added to the original Sphider. For details about all the improvements and changes, please read the Documentation section. [ Main features ] Item . . .
. . .
section. [ Main features ] Item Description UTF-8 and UTF-16 support Indexation and search procedure for Chinese, Cyrillic, Georgian, Hebrew etc. charsets. UNICODE support including astral symbols. Support for non-ASCII domains 'Internationalized Domain Names' (IDN) like 'http://президент.рф/' and 'http://müller.de/' are accepted and . . .
. . .
'http://müller.de/' are accepted and processed. Responsive design Automatically adapting the size of search form, result listing and addurl form to display size of computers, tablets, smartphones, etc. Media support Index and search for images (incl. Open Graph images), audio and video (incl. Youtube videos). EXIF and ID3 information are also . . .
. . .
player software. Multiple database support Individual configuration and activation of databases for 'Admin', 'Search User' and 'Suggest URL'. Support of multiple table sets in each db, MySQL query cache, individual index for each db, individual or bulk search in predefined databases. Individual Admin settings for each db and each set of . . .
. . .
always in cache. Separate caches for text and media results. Admin configurable. Follow sitemap files If available, sitemap.xml as well as gzip compressed files will be used to follow the links of a site. If <sitemapindex . . . > is detected, also multiple sitemap files are processed. Periodical Re-index Re-indexing could be performed . . .
. . .
also multiple sitemap files are processed. Periodical Re-index Re-indexing could be performed automatically, repeated every selected time interval. Admin selectable intervals for 3 hours, 12 hours, 1 day, 1 week, or 1 month. Multithreaded indexing In order to reduce the time for indexing, 1-10 parallel running threads might be activated . . .
. . .
1-10 parallel running threads might be activated in Admin settings. Preferred re-index While invoking this option, the admin may select a suitable level for the next index procedure. Thus, only those URLs, containing the according level, will be re-indexed.. Erase Re-index and Continue suspended index procedures Individual (site specific) or . . .
. . .
(site specific) or bulk update of database. Support of XML product feeds Index and search of feed content, inclusive formatting the search results. RDF, RSD, RSS and Atom feed support Index and search of feed content, inclusive RDF 'Dublin Core' tags. Obey / ignore 'preferred' tags in RSD feeds. Follow / ignore CDATA directives. Various . . .
. . .
ignore 'preferred' tags in RSD feeds. Follow / ignore CDATA directives. Various search modes Search with wildcards, Tolerant search, Search strict, Search only in one domain, Search all links of a site, Search for media (link-specific). Add thumbnails to each page presented in text results Admin selectable, this feature will present a web shot . . .
. . .
suspected to contain malware or phishing content. 11 different modes of sorting the text results Admin selectable: -By relevance (weight % ) -By hit counts in full text -Most popular links on top -By indexdate -By URL names -By file suffix -Main URL (domain) on top -Like Google (Top 2 per URL) - Promoted domain on top - Links holding promoted . . .
. . .
on top - Links holding promoted catchwords on top. 5 different modes of sorting the media results Admin selectable: -By title(alphabetic) -By file suffix -By image size -By 'Last queried' -By 'Most popular'. Same results for queries typed with pure vowels, or with accents Will deliver the same results for queries like: caf e and caf é . To be . . .
. . .
in Admin backend. Same results for queries with and without quotes Will deliver the same results for queries like: d'information <-> information dei'largi <-> largi Also Admin selectabe: Equalize the different quotes like: ' ` ´ Same text results for queries with and without ligatures Admin selectable; will deliver the same results . . .
. . .
results for queries with and without ligatures Admin selectable; will deliver the same results for queries like: cœur and coeur . Worked out for Latin ligatures in Unicode (Latin-derived alphabets) and also ligatures used only in phonetic transcription, but not taking into consideration medieval ligatures. Present all results for singular and . . .
. . .
all will become searchable. Dictionaries with 106.800 radicals. Segmentation of Japanese words Segmentation of 5.724 kanji (new, old and half width), hiragana, katakana and jinmeiyo Japanese character writing systems. Transliterate Latin characters into their Greek equivalents Transforms query input alla to find ἀλλὰ and baptismatos to find . . .
. . .
redirections caused by HTTP 301, 302, 303 and 307 status codes. Also obeying JavaScript, sent as HTML content like: <SCRIPT language="javascript"> win.loc="mp.php?mcv=59";</SCRIPT> Follow header redirections, refresh tags and canonical links Automatical forwarding for the indexer. Follow links found in JavaScript and index also the . . .
. . .
and index also the content of document.write Will index JavaScript commands. Detect and follow links like: document.write(' <a href="new12.pdf">All news 2012</a> '); and index the content of: document.write(' this content '); Not indexing content created in real-time by JavaScript. Accept gzip formatted transmission In order . . .
. . .
Converter included for PDF, DOCX, XLSX, ODT, ODS, CSV, PPTX and XLS files Converting also non-Latin text like: Arabic, Cyrillic, Chinese, Greece and Hebrew. Links found in the converted files will be followed. Debug mode Offering detailed information during index/re-index: New links, keywords, frames and media found per link. To be . . .
. . .
Included for 33 languages. Common word lists holding stop words. Included for 25 languages Admin selectable for: Arabic, Bengali, Bulgarian, Catalan, Chinese, Cyrillic, Czech, Danish, Dutch, English, Farsi, Finnish, French, Greek, German, Hindi, Hungarian, Italian, Norwegian, Polish, Portuguese, Romanian, Spanish, Swedish and Turkish. . . .
. . .
links outside are followed. Multiple and nested divs will be attended. Do not index parts of a page defined by HTML5 elements <tag> . . . </tag> Foreseen to cooperate with the HTML5 elements like: section, nav, aside, hgroup, article, header, footer Vice versa function also included in order to index only parts of a page between . . .
. . .
Extension implemented SQLi connector implemented between PHP and a MySQL database. Performed by OOP, also PHP v.5.5 is supported. Compatible with MySQL and MariaDB Proven up to: - MySQL version 8.0.32 - MariaDB version 10.4.28 sp_executesql Ready to run in PHP 8 environment Latest version of Sphider-plus version 4.2024a is proven up to PHP . . .
. . .
8.3.2 [ Proven ] Successfully implemented as search engine on a customer site with a database capacity such as: 25.206 sites 324595 page links 1.260.698 keywords 169.251 media links. Imprint Private Notice Private Policy . . .


Sphider-plus is a search engine, based on the scripts of original Sphider. [ About Sphider-plus] More than 400 new features (additional mods, functions, template designs and debugging) have been added to the original Sphider. For details about all the improvements and changes, please read the Documentation section. [ Main features ] Item . . .
. . .
section. [ Main features ] Item Description UTF-8 and UTF-16 support Indexation and search procedure for Chinese, Cyrillic, Georgian, Hebrew etc. charsets. UNICODE support including astral symbols. Support for non-ASCII domains 'Internationalized Domain Names' (IDN) like 'http//президентрф andhttp//президент.рф/' and 'http//президентрф andhttp//müller.de/' are accepted and . . .
. . .
'http//президентрф andhttp//müller.de/' are accepted and processed. Responsive design Automatically adapting the size of search form, result listing and addurl form to display size of computers, tablets, smartphones, etc. Media support Index and search for images (incl. Open Graph images), audio and video (incl. Youtube videos). EXIF and ID3 information are also . . .
. . .
player software. Multiple database support Individual configuration and activation of databases for 'Admin', 'Search User' and 'Suggest URL'. Support of multiple table sets in each db, MySQL query cache, individual index for each db, individual or bulk search in predefined databases. Individual Admin settings for each db and each set of . . .
. . .
always in cache. Separate caches for text and media results. Admin configurable. Follow sitemap files If available, sitemap.xml as well as gzip compressed files will be used to follow the links of a site. If <sitemapindex . . . > is detected, also multiple sitemap files are processed. Periodical Re-index Re-indexing could be performed . . .
. . .
also multiple sitemap files are processed. Periodical Re-index Re-indexing could be performed automatically, repeated every selected time interval. Admin selectable intervals for 3 hours, 12 hours, 1 day, 1 week, or 1 month. Multithreaded indexing In order to reduce the time for indexing, 1-10 parallel running threads might be activated . . .
. . .
1-10 parallel running threads might be activated in Admin settings. Preferred re-index While invoking this option, the admin may select a suitable level for the next index procedure. Thus, only those URLs, containing the according level, will be re-indexed.. Erase Re-index and Continue suspended index procedures Individual (site specific) or . . .
. . .
(site specific) or bulk update of database. Support of XML product feeds Index and search of feed content, inclusive formatting the search results. RDF, RSD, RSS and Atom feed support Index and search of feed content, inclusive RDF 'Dublin Core' tags. Obey / ignore 'preferred' tags in RSD feeds. Follow / ignore CDATA directives. Various . . .
. . .
ignore 'preferred' tags in RSD feeds. Follow / ignore CDATA directives. Various search modes Search with wildcards, Tolerant search, Search strict, Search only in one domain, Search all links of a site, Search for media (link-specific). Add thumbnails to each page presented in text results Admin selectable, this feature will present a web shot . . .
. . .
suspected to contain malware or phishing content. 11 different modes of sorting the text results Admin selectable//президентрф andhttp -By relevance (weight % ) -By hit counts in full text -Most popular links on top -By indexdate -By URL names -By file suffix -Main URL (domain) on top -Like Google (Top 2 per URL) - Promoted domain on top - Links holding promoted . . .
. . .
on top - Links holding promoted catchwords on top. 5 different modes of sorting the media results Admin selectable//президентрф andhttp -By title(alphabetic) -By file suffix -By image size -By 'Last queried' -By 'Most popular'. Same results for queries typed with pure vowels, or with accents Will deliver the same results for queries like//президентрф andhttp caf e and caf é . To be . . .
. . .
in Admin backend. Same results for queries with and without quotes Will deliver the same results for queries like//президентрф andhttp d'information <-> information dei'largi <-> largi Also Admin selectabe//президентрф andhttp Equalize the different quotes like//президентрф andhttp ' ` ´ Same text results for queries with and without ligatures Admin selectable; will deliver the same results . . .
. . .
results for queries with and without ligatures Admin selectable; will deliver the same results for queries like//президентрф andhttp cœur and coeur . Worked out for Latin ligatures in Unicode (Latin-derived alphabets) and also ligatures used only in phonetic transcription, but not taking into consideration medieval ligatures. Present all results for singular and . . .
. . .
all will become searchable. Dictionaries with 106.800 radicals. Segmentation of Japanese words Segmentation of 5.724 kanji (new, old and half width), hiragana, katakana and jinmeiyo Japanese character writing systems. Transliterate Latin characters into their Greek equivalents Transforms query input alla to find ἀλλὰ and baptismatos to find . . .
. . .
redirections caused by HTTP 301, 302, 303 and 307 status codes. Also obeying JavaScript, sent as HTML content like//президентрф andhttp <SCRIPT language="javascript"> win.loc="mp.php?mcv=59";</SCRIPT> Follow header redirections, refresh tags and canonical links Automatical forwarding for the indexer. Follow links found in JavaScript and index also the . . .
. . .
and index also the content of document.write Will index JavaScript commands. Detect and follow links like//президентрф andhttp document.write(' <a href="new12.pdf">All news 2012</a> '); and index the content of//президентрф andhttp document.write(' this content '); Not indexing content created in real-time by JavaScript. Accept gzip formatted transmission In order . . .
. . .
Converter included for PDF, DOCX, XLSX, ODT, ODS, CSV, PPTX and XLS files Converting also non-Latin text like//президентрф andhttp Arabic, Cyrillic, Chinese, Greece and Hebrew. Links found in the converted files will be followed. Debug mode Offering detailed information during index/re-index//президентрф andhttp New links, keywords, frames and media found per link. To be . . .
. . .
Included for 33 languages. Common word lists holding stop words. Included for 25 languages Admin selectable for//президентрф andhttp Arabic, Bengali, Bulgarian, Catalan, Chinese, Cyrillic, Czech, Danish, Dutch, English, Farsi, Finnish, French, Greek, German, Hindi, Hungarian, Italian, Norwegian, Polish, Portuguese, Romanian, Spanish, Swedish and Turkish. . . .
. . .
links outside are followed. Multiple and nested divs will be attended. Do not index parts of a page defined by HTML5 elements <tag> . . . </tag> Foreseen to cooperate with the HTML5 elements like//президентрф andhttp section, nav, aside, hgroup, article, header, footer Vice versa function also included in order to index only parts of a page between . . .
. . .
Extension implemented SQLi connector implemented between PHP and a MySQL database. Performed by OOP, also PHP v.5.5 is supported. Compatible with MySQL and MariaDB Proven up to//президентрф andhttp - MySQL version 8.0.32 - MariaDB version 10.4.28 sp_executesql Ready to run in PHP 8 environment Latest version of Sphider-plus version 4.2024a is proven up to PHP . . .
. . .
8.3.2 [ Proven ] Successfully implemented as search engine on a customer site with a database capacity such as//президентрф andhttp 25.206 sites 324.595 page links 1.260.698 keywords 169.251 media links. Imprint Private Notice Private Policy . . .


Sphider-plus is a search engine, based on the scripts of original Sphider. [ About Sphider-plus] More than 400 new features (additional mods, functions, template designs and debugging) have been added to the original Sphider. For details about all the improvements and changes, please read the Documentation section. [ Main features ] Item . . .
. . .
section. [ Main features ] Item Description UTF-8 and UTF-16 support Indexation and search procedure for Chinese, Cyrillic, Georgian, Hebrew etc. charsets. UNICODE support including astral symbols. Support for non-ASCII domains 'Internationalized Domain Names' (IDN) like 'http//президентрф andhttp//президент.рф/' and 'http//президентрф andhttp//müller.de/' are accepted and . . .
. . .
'http//президентрф andhttp//müller.de/' are accepted and processed. Responsive design Automatically adapting the size of search form, result listing and addurl form to display size of computers, tablets, smartphones, etc. Media support Index and search for images (incl. Open Graph images), audio and video (incl. Youtube videos). EXIF and ID3 information are also . . .
. . .
player software. Multiple database support Individual configuration and activation of databases for 'Admin', 'Search User' and 'Suggest URL'. Support of multiple table sets in each db, MySQL query cache, individual index for each db, individual or bulk search in predefined databases. Individual Admin settings for each db and each set of . . .
. . .
always in cache. Separate caches for text and media results. Admin configurable. Follow sitemap files If available, sitemap.xml as well as gzip compressed files will be used to follow the links of a site. If <sitemapindex . . . > is detected, also multiple sitemap files are processed. Periodical Re-index Re-indexing could be performed . . .
. . .
also multiple sitemap files are processed. Periodical Re-index Re-indexing could be performed automatically, repeated every selected time interval. Admin selectable intervals for 3 hours, 12 hours, 1 day, 1 week, or 1 month. Multithreaded indexing In order to reduce the time for indexing, 1-10 parallel running threads might be activated . . .
. . .
1-10 parallel running threads might be activated in Admin settings. Preferred re-index While invoking this option, the admin may select a suitable level for the next index procedure. Thus, only those URLs, containing the according level, will be re-indexed.. Erase Re-index and Continue suspended index procedures Individual (site specific) or . . .
. . .
(site specific) or bulk update of database. Support of XML product feeds Index and search of feed content, inclusive formatting the search results. RDF, RSD, RSS and Atom feed support Index and search of feed content, inclusive RDF 'Dublin Core' tags. Obey / ignore 'preferred' tags in RSD feeds. Follow / ignore CDATA directives. Various . . .
. . .
ignore 'preferred' tags in RSD feeds. Follow / ignore CDATA directives. Various search modes Search with wildcards, Tolerant search, Search strict, Search only in one domain, Search all links of a site, Search for media (link-specific). Add thumbnails to each page presented in text results Admin selectable, this feature will present a web shot . . .
. . .
suspected to contain malware or phishing content. 11 different modes of sorting the text results Admin selectable//президентрф andhttp -By relevance (weight % ) -By hit counts in full text -Most popular links on top -By indexdate -By URL names -By file suffix -Main URL (domain) on top -Like Google (Top 2 per URL) - Promoted domain on top - Links holding promoted . . .
. . .
on top - Links holding promoted catchwords on top. 5 different modes of sorting the media results Admin selectable//президентрф andhttp -By title(alphabetic) -By file suffix -By image size -By 'Last queried' -By 'Most popular'. Same results for queries typed with pure vowels, or with accents Will deliver the same results for queries like//президентрф andhttp caf e and caf é . To be . . .
. . .
in Admin backend. Same results for queries with and without quotes Will deliver the same results for queries like//президентрф andhttp d'information <-> information dei'largi <-> largi Also Admin selectabe//президентрф andhttp Equalize the different quotes like//президентрф andhttp ' ` ´ Same text results for queries with and without ligatures Admin selectable; will deliver the same results . . .
. . .
results for queries with and without ligatures Admin selectable; will deliver the same results for queries like//президентрф andhttp cœur and coeur . Worked out for Latin ligatures in Unicode (Latin-derived alphabets) and also ligatures used only in phonetic transcription, but not taking into consideration medieval ligatures. Present all results for singular and . . .
. . .
all will become searchable. Dictionaries with 106.800 radicals. Segmentation of Japanese words Segmentation of 5.724 kanji (new, old and half width), hiragana, katakana and jinmeiyo Japanese character writing systems. Transliterate Latin characters into their Greek equivalents Transforms query input alla to find ἀλλὰ and baptismatos to find . . .
. . .
redirections caused by HTTP 301, 302, 303 and 307 status codes. Also obeying JavaScript, sent as HTML content like//президентрф andhttp <SCRIPT language="javascript"> win.loc="mp.php?mcv=59";</SCRIPT> Follow header redirections, refresh tags and canonical links Automatical forwarding for the indexer. Follow links found in JavaScript and index also the . . .
. . .
and index also the content of document.write Will index JavaScript commands. Detect and follow links like//президентрф andhttp document.write(' <a href="new12.pdf">All news 2012</a> '); and index the content of//президентрф andhttp document.write(' this content '); Not indexing content created in real-time by JavaScript. Accept gzip formatted transmission In order . . .
. . .
Converter included for PDF, DOCX, XLSX, ODT, ODS, CSV, PPTX and XLS files Converting also non-Latin text like//президентрф andhttp Arabic, Cyrillic, Chinese, Greece and Hebrew. Links found in the converted files will be followed. Debug mode Offering detailed information during index/re-index//президентрф andhttp New links, keywords, frames and media found per link. To be . . .
. . .
Included for 33 languages. Common word lists holding stop words. Included for 25 languages Admin selectable for//президентрф andhttp Arabic, Bengali, Bulgarian, Catalan, Chinese, Cyrillic, Czech, Danish, Dutch, English, Farsi, Finnish, French, Greek, German, Hindi, Hungarian, Italian, Norwegian, Polish, Portuguese, Romanian, Spanish, Swedish and Turkish. . . .
. . .
links outside are followed. Multiple and nested divs will be attended. Do not index parts of a page defined by HTML5 elements <tag> . . . </tag> Foreseen to cooperate with the HTML5 elements like//президентрф andhttp section, nav, aside, hgroup, article, header, footer Vice versa function also included in order to index only parts of a page between . . .
. . .
Extension implemented SQLi connector implemented between PHP and a MySQL database. Performed by OOP, also PHP v.5.5 is supported. Compatible with MySQL and MariaDB Proven up to//президентрф andhttp - MySQL version 8.0.32 - MariaDB version 10.4.28 sp_executesql Ready to run in PHP 8 environment Latest version of Sphider-plus version 4.2024a is proven up to PHP . . .
. . .
8.3.2 [ Proven ] Successfully implemented as search engine on a customer site with a database capacity such as//президентрф andhttp 25.206 sites 324.595 page links 1.260.698 keywords 169.251 media links. Imprint Private Notice Private Policy . . .


Sphider-plus is a search engine, based on the scripts of original Sphider. [ About Sphider-plus] More than 400 new features (additional mods, functions, template designs and debugging) have been added to the original Sphider. For details about all the improvements and changes, please read the Documentation section. [ Main features ] Item . . .
. . .
section. [ Main features ] Item Description UTF-8 and UTF-16 support Indexation and search procedure for Chinese, Cyrillic, Georgian, Hebrew etc. charsets. UNICODE support including astral symbols. Support for non-ASCII domains 'Internationalized Domain Names' (IDN) like 'http://президент.рф/' and 'http://müller.de/' are accepted and . . .
. . .
'http://müller.de/' are accepted and processed. Responsive design Automatically adapting the size of search form, result listing and addurl form to display size of computers, tablets, smartphones, etc. Media support Index and search for images (incl. Open Graph images), audio and video (incl. Youtube videos). EXIF and ID3 information are also . . .
. . .
player software. Multiple database support Individual configuration and activation of databases for 'Admin', 'Search User' and 'Suggest URL'. Support of multiple table sets in each db, MySQL query cache, individual index for each db, individual or bulk search in predefined databases. Individual Admin settings for each db and each set of . . .
. . .
always in cache. Separate caches for text and media results. Admin configurable. Follow sitemap files If available, sitemap.xml as well as gzip compressed files will be used to follow the links of a site. If <sitemapindex . . . > is detected, also multiple sitemap files are processed. Periodical Re-index Re-indexing could be performed . . .
. . .
also multiple sitemap files are processed. Periodical Re-index Re-indexing could be performed automatically, repeated every selected time interval. Admin selectable intervals for 3 hours, 12 hours, 1 day, 1 week, or 1 month. Multithreaded indexing In order to reduce the time for indexing, 1-10 parallel running threads might be activated . . .
. . .
1-10 parallel running threads might be activated in Admin settings. Preferred re-index While invoking this option, the admin may select a suitable level for the next index procedure. Thus, only those URLs, containing the according level, will be re-indexed.. Erase Re-index and Continue suspended index procedures Individual (site specific) or . . .
. . .
(site specific) or bulk update of database. Support of XML product feeds Index and search of feed content, inclusive formatting the search results. RDF, RSD, RSS and Atom feed support Index and search of feed content, inclusive RDF 'Dublin Core' tags. Obey / ignore 'preferred' tags in RSD feeds. Follow / ignore CDATA directives. Various . . .
. . .
ignore 'preferred' tags in RSD feeds. Follow / ignore CDATA directives. Various search modes Search with wildcards, Tolerant search, Search strict, Search only in one domain, Search all links of a site, Search for media (link-specific). Add thumbnails to each page presented in text results Admin selectable, this feature will present a web shot . . .
. . .
suspected to contain malware or phishing content. 11 different modes of sorting the text results Admin selectable: -By relevance (weight % ) -By hit counts in full text -Most popular links on top -By indexdate -By URL names -By file suffix -Main URL (domain) on top -Like Google (Top 2 per URL) - Promoted domain on top - Links holding promoted . . .
. . .
on top - Links holding promoted catchwords on top. 5 different modes of sorting the media results Admin selectable: -By title(alphabetic) -By file suffix -By image size -By 'Last queried' -By 'Most popular'. Same results for queries typed with pure vowels, or with accents Will deliver the same results for queries like: caf e and caf é . To be . . .
. . .
in Admin backend. Same results for queries with and without quotes Will deliver the same results for queries like: d'information <-> information dei'largi <-> largi Also Admin selectabe: Equalize the different quotes like: ' ` ´ Same text results for queries with and without ligatures Admin selectable; will deliver the same results . . .
. . .
results for queries with and without ligatures Admin selectable; will deliver the same results for queries like: cœur and coeur . Worked out for Latin ligatures in Unicode (Latin-derived alphabets) and also ligatures used only in phonetic transcription, but not taking into consideration medieval ligatures. Present all results for singular and . . .
. . .
all will become searchable. Dictionaries with 106.800 radicals. Segmentation of Japanese words Segmentation of 5.724 kanji (new, old and half width), hiragana, katakana and jinmeiyo Japanese character writing systems. Transliterate Latin characters into their Greek equivalents Transforms query input alla to find ἀλλὰ and baptismatos to find . . .
. . .
redirections caused by HTTP 301, 302, 303 and 307 status codes. Also obeying JavaScript, sent as HTML content like: <SCRIPT language="javascript"> win.loc="mp.php?mcv=59";</SCRIPT> Follow header redirections, refresh tags and canonical links Automatical forwarding for the indexer. Follow links found in JavaScript and index also the . . .
. . .
and index also the content of document.write Will index JavaScript commands. Detect and follow links like: document.write(' <a href="new12.pdf">All news 2012</a> '); and index the content of: document.write(' this content '); Not indexing content created in real-time by JavaScript. Accept gzip formatted transmission In order . . .
. . .
Converter included for PDF, DOCX, XLSX, ODT, ODS, CSV, PPTX and XLS files Converting also non-Latin text like: Arabic, Cyrillic, Chinese, Greece and Hebrew. Links found in the converted files will be followed. Debug mode Offering detailed information during index/re-index: New links, keywords, frames and media found per link. To be . . .
. . .
Included for 33 languages. Common word lists holding stop words. Included for 25 languages Admin selectable for: Arabic, Bengali, Bulgarian, Catalan, Chinese, Cyrillic, Czech, Danish, Dutch, English, Farsi, Finnish, French, Greek, German, Hindi, Hungarian, Italian, Norwegian, Polish, Portuguese, Romanian, Spanish, Swedish and Turkish. . . .
. . .
links outside are followed. Multiple and nested divs will be attended. Do not index parts of a page defined by HTML5 elements <tag> . . . </tag> Foreseen to cooperate with the HTML5 elements like: section, nav, aside, hgroup, article, header, footer Vice versa function also included in order to index only parts of a page between . . .
. . .
Extension implemented SQLi connector implemented between PHP and a MySQL database. Performed by OOP, also PHP v.5.5 is supported. Compatible with MySQL and MariaDB Proven up to: - MySQL version 8.0.32 - MariaDB version 10.4.28 sp_executesql Ready to run in PHP 8 environment Latest version of Sphider-plus version 4.2024a is proven up to PHP . . .
. . .
8.3.2 [ Proven ] Successfully implemented as search engine on a customer site with a database capacity such as: 25.206 sites 324.595 page links 1.260.698 keywords 169.251 media links. Imprint Private Notice Private Policy . . .


Sphider-plus is a search engine, based on the scripts of original Sphider. [ About Sphider-plus] More than 400 new features (additional mods, functions, template designs and debugging) have been added to the original Sphider. For details about all the improvements and changes, please read the Documentation section. [ Main features ] Item . . .
. . .
section. [ Main features ] Item Description UTF-8 and UTF-16 support Indexation and search procedure for Chinese, Cyrillic, Georgian, Hebrew etc. charsets. UNICODE support including astral symbols. Support for non-ASCII domains 'Internationalized Domain Names' (IDN) like 'http://президент.рф/' and 'http://müller.de/' are accepted and . . .
. . .
'http://müller.de/' are accepted and processed. Responsive design Automatically adapting the size of search form, result listing and addurl form to display size of computers, tablets, smartphones, etc. Media support Index and search for images (incl. Open Graph images), audio and video (incl. Youtube videos). EXIF and ID3 information are also . . .
. . .
player software. Multiple database support Individual configuration and activation of databases for 'Admin', 'Search User' and 'Suggest URL'. Support of multiple table sets in each db, MySQL query cache, individual index for each db, individual or bulk search in predefined databases. Individual Admin settings for each db and each set of . . .
. . .
always in cache. Separate caches for text and media results. Admin configurable. Follow sitemap files If available, sitemap.xml as well as gzip compressed files will be used to follow the links of a site. If <sitemapindex . . . > is detected, also multiple sitemap files are processed. Periodical Re-index Re-indexing could be performed . . .
. . .
also multiple sitemap files are processed. Periodical Re-index Re-indexing could be performed automatically, repeated every selected time interval. Admin selectable intervals for 3 hours, 12 hours, 1 day, 1 week, or 1 month. Multithreaded indexing In order to reduce the time for indexing, 1-10 parallel running threads might be activated . . .
. . .
1-10 parallel running threads might be activated in Admin settings. Preferred re-index While invoking this option, the admin may select a suitable level for the next index procedure. Thus, only those URLs, containing the according level, will be re-indexed.. Erase Re-index and Continue suspended index procedures Individual (site specific) or . . .
. . .
(site specific) or bulk update of database. Support of XML product feeds Index and search of feed content, inclusive formatting the search results. RDF, RSD, RSS and Atom feed support Index and search of feed content, inclusive RDF 'Dublin Core' tags. Obey / ignore 'preferred' tags in RSD feeds. Follow / ignore CDATA directives. Various . . .
. . .
ignore 'preferred' tags in RSD feeds. Follow / ignore CDATA directives. Various search modes Search with wildcards, Tolerant search, Search strict, Search only in one domain, Search all links of a site, Search for media (link-specific). Add thumbnails to each page presented in text results Admin selectable, this feature will present a web shot . . .
. . .
suspected to contain malware or phishing content. 11 different modes of sorting the text results Admin selectable: -By relevance (weight % ) -By hit counts in full text -Most popular links on top -By indexdate -By URL names -By file suffix -Main URL (domain) on top -Like Google (Top 2 per URL) - Promoted domain on top - Links holding promoted . . .
. . .
on top - Links holding promoted catchwords on top. 5 different modes of sorting the media results Admin selectable: -By title(alphabetic) -By file suffix -By image size -By 'Last queried' -By 'Most popular'. Same results for queries typed with pure vowels, or with accents Will deliver the same results for queries like: caf e and caf é . To be . . .
. . .
in Admin backend. Same results for queries with and without quotes Will deliver the same results for queries like: d'information <-> information dei'largi <-> largi Also Admin selectabe: Equalize the different quotes like: ' ` ´ Same text results for queries with and without ligatures Admin selectable; will deliver the same results . . .
. . .
results for queries with and without ligatures Admin selectable; will deliver the same results for queries like: cœur and coeur . Worked out for Latin ligatures in Unicode (Latin-derived alphabets) and also ligatures used only in phonetic transcription, but not taking into consideration medieval ligatures. Present all results for singular and . . .
. . .
all will become searchable. Dictionaries with 106.800 radicals. Segmentation of Japanese words Segmentation of 5.724 kanji (new, old and half width), hiragana, katakana and jinmeiyo Japanese character writing systems. Transliterate Latin characters into their Greek equivalents Transforms query input alla to find ἀλλὰ and baptismatos to find . . .
. . .
redirections caused by HTTP 301, 302, 303 and 307 status codes. Also obeying JavaScript, sent as HTML content like: <SCRIPT language=javascript>"javascript"> win.loc="mp.php?mcv=59";</SCRIPT> Follow header redirections, refresh tags and canonical links Automatical forwarding for the indexer. Follow links found in JavaScript and index also the . . .
. . .
and index also the content of document.write Will index JavaScript commands. Detect and follow links like: document.write(' <a href="new12.pdf">All news 2012</a> '); and index the content of: document.write(' this content '); Not indexing content created in real-time by JavaScript. Accept gzip formatted transmission In order . . .
. . .
Converter included for PDF, DOCX, XLSX, ODT, ODS, CSV, PPTX and XLS files Converting also non-Latin text like: Arabic, Cyrillic, Chinese, Greece and Hebrew. Links found in the converted files will be followed. Debug mode Offering detailed information during index/re-index: New links, keywords, frames and media found per link. To be . . .
. . .
Included for 33 languages. Common word lists holding stop words. Included for 25 languages Admin selectable for: Arabic, Bengali, Bulgarian, Catalan, Chinese, Cyrillic, Czech, Danish, Dutch, English, Farsi, Finnish, French, Greek, German, Hindi, Hungarian, Italian, Norwegian, Polish, Portuguese, Romanian, Spanish, Swedish and Turkish. . . .
. . .
links outside are followed. Multiple and nested divs will be attended. Do not index parts of a page defined by HTML5 elements <tag> . . . </tag> Foreseen to cooperate with the HTML5 elements like: section, nav, aside, hgroup, article, header, footer Vice versa function also included in order to index only parts of a page between . . .
. . .
Extension implemented SQLi connector implemented between PHP and a MySQL database. Performed by OOP, also PHP v.5.5 is supported. Compatible with MySQL and MariaDB Proven up to: - MySQL version 8.0.32 - MariaDB version 10.4.28 sp_executesql Ready to run in PHP 8 environment Latest version of Sphider-plus version 4.2024a is proven up to PHP . . .
. . .
8.3.2 [ Proven ] Successfully implemented as search engine on a customer site with a database capacity such as: 25.206 sites 324.595 page links 1.260.698 keywords 169.251 media links. Imprint Private Notice Private Policy . . .


Sphider-plus is a search engine, based on the scripts of original Sphider. [ About Sphider-plus] More than 400 new features (additional mods, functions, template designs and debugging) have been added to the original Sphider. For details about all the improvements and changes, please read the Documentation section. [ Main features ] Item . . .
. . .
section. [ Main features ] Item Description UTF-8 and UTF-16 support Indexation and search procedure for Chinese, Cyrillic, Georgian, Hebrew etc. charsets. UNICODE support including astral symbols. Support for non-ASCII domains 'Internationalized Domain Names' (IDN) like 'http://президент.рф/' and 'http://müller.de/' are accepted and . . .
. . .
'http://müller.de/' are accepted and processed. Responsive design Automatically adapting the size of search form, result listing and addurl form to display size of computers, tablets, smartphones, etc. Media support Index and search for images (incl. Open Graph images), audio and video (incl. Youtube videos). EXIF and ID3 information are also . . .
. . .
player software. Multiple database support Individual configuration and activation of databases for 'Admin', 'Search User' and 'Suggest URL'. Support of multiple table sets in each db, MySQL query cache, individual index for each db, individual or bulk search in predefined databases. Individual Admin settings for each db and each set of . . .
. . .
always in cache. Separate caches for text and media results. Admin configurable. Follow sitemap files If available, sitemap.xml as well as gzip compressed files will be used to follow the links of a site. If <sitemapindex . . . > is detected, also multiple sitemap files are processed. Periodical Re-index Re-indexing could be performed . . .
. . .
also multiple sitemap files are processed. Periodical Re-index Re-indexing could be performed automatically, repeated every selected time interval. Admin selectable intervals for 3 hours, 12 hours, 1 day, 1 week, or 1 month. Multithreaded indexing In order to reduce the time for indexing, 1-10 parallel running threads might be activated . . .
. . .
1-10 parallel running threads might be activated in Admin settings. Preferred re-index While invoking this option, the admin may select a suitable level for the next index procedure. Thus, only those URLs, containing the according level, will be re-indexed.. Erase Re-index and Continue suspended index procedures Individual (site specific) or . . .
. . .
(site specific) or bulk update of database. Support of XML product feeds Index and search of feed content, inclusive formatting the search results. RDF, RSD, RSS and Atom feed support Index and search of feed content, inclusive RDF 'Dublin Core' tags. Obey / ignore 'preferred' tags in RSD feeds. Follow / ignore CDATA directives. Various . . .
. . .
ignore 'preferred' tags in RSD feeds. Follow / ignore CDATA directives. Various search modes Search with wildcards, Tolerant search, Search strict, Search only in one domain, Search all links of a site, Search for media (link-specific). Add thumbnails to each page presented in text results Admin selectable, this feature will present a web shot . . .
. . .
suspected to contain malware or phishing content. 11 different modes of sorting the text results Admin selectable: -By relevance (weight % ) -By hit counts in full text -Most popular links on top -By indexdate -By URL names -By file suffix -Main URL (domain) on top -Like Google (Top 2 per URL) - Promoted domain on top - Links holding promoted . . .
. . .
on top - Links holding promoted catchwords on top. 5 different modes of sorting the media results Admin selectable: -By title(alphabetic) -By file suffix -By image size -By 'Last queried' -By 'Most popular'. Same results for queries typed with pure vowels, or with accents Will deliver the same results for queries like: caf e and caf é . To be . . .
. . .
in Admin backend. Same results for queries with and without quotes Will deliver the same results for queries like: d'information <-> information dei'largi <-> largi Also Admin selectabe: Equalize the different quotes like: ' ` ´ Same text results for queries with and without ligatures Admin selectable; will deliver the same results . . .
. . .
results for queries with and without ligatures Admin selectable; will deliver the same results for queries like: cœur and coeur . Worked out for Latin ligatures in Unicode (Latin-derived alphabets) and also ligatures used only in phonetic transcription, but not taking into consideration medieval ligatures. Present all results for singular and . . .
. . .
all will become searchable. Dictionaries with 106.800 radicals. Segmentation of Japanese words Segmentation of 5.724 kanji (new, old and half width), hiragana, katakana and jinmeiyo Japanese character writing systems. Transliterate Latin characters into their Greek equivalents Transforms query input alla to find ἀλλὰ and baptismatos to find . . .
. . .
redirections caused by HTTP 301, 302, 303 and 307 status codes. Also obeying JavaScript, sent as HTML content like: <SCRIPT language="javascript"> win.loc="mpphpmcv=59</SCRIPT>";</SCRIPT> Follow header redirections, refresh tags and canonical links Automatical forwarding for the indexer. Follow links found in JavaScript and index also the . . .
. . .
and index also the content of document.write Will index JavaScript commands. Detect and follow links like: document.write(' <a href="new12.pdf">All news 2012</a> '); and index the content of: document.write(' this content '); Not indexing content created in real-time by JavaScript. Accept gzip formatted transmission In order . . .
. . .
Converter included for PDF, DOCX, XLSX, ODT, ODS, CSV, PPTX and XLS files Converting also non-Latin text like: Arabic, Cyrillic, Chinese, Greece and Hebrew. Links found in the converted files will be followed. Debug mode Offering detailed information during index/re-index: New links, keywords, frames and media found per link. To be . . .
. . .
Included for 33 languages. Common word lists holding stop words. Included for 25 languages Admin selectable for: Arabic, Bengali, Bulgarian, Catalan, Chinese, Cyrillic, Czech, Danish, Dutch, English, Farsi, Finnish, French, Greek, German, Hindi, Hungarian, Italian, Norwegian, Polish, Portuguese, Romanian, Spanish, Swedish and Turkish. . . .
. . .
links outside are followed. Multiple and nested divs will be attended. Do not index parts of a page defined by HTML5 elements <tag> . . . </tag> Foreseen to cooperate with the HTML5 elements like: section, nav, aside, hgroup, article, header, footer Vice versa function also included in order to index only parts of a page between . . .
. . .
Extension implemented SQLi connector implemented between PHP and a MySQL database. Performed by OOP, also PHP v.5.5 is supported. Compatible with MySQL and MariaDB Proven up to: - MySQL version 8.0.32 - MariaDB version 10.4.28 sp_executesql Ready to run in PHP 8 environment Latest version of Sphider-plus version 4.2024a is proven up to PHP . . .
. . .
8.3.2 [ Proven ] Successfully implemented as search engine on a customer site with a database capacity such as: 25.206 sites 324.595 page links 1.260.698 keywords 169.251 media links. Imprint Private Notice Private Policy . . .


Sphider-plus is a search engine, based on the scripts of original Sphider. [ About Sphider-plus] More than 400 new features (additional mods, functions, template designs and debugging) have been added to the original Sphider. For details about all the improvements and changes, please read the Documentation section. [ Main features ] Item . . .
. . .
section. [ Main features ] Item Description UTF-8 and UTF-16 support Indexation and search procedure for Chinese, Cyrillic, Georgian, Hebrew etc. charsets. UNICODE support including astral symbols. Support for non-ASCII domains 'Internationalized Domain Names' (IDN) like 'http://президент.рф/' and 'http://müller.de/' are accepted and . . .
. . .
'http://müller.de/' are accepted and processed. Responsive design Automatically adapting the size of search form, result listing and addurl form to display size of computers, tablets, smartphones, etc. Media support Index and search for images (incl. Open Graph images), audio and video (incl. Youtube videos). EXIF and ID3 information are also . . .
. . .
player software. Multiple database support Individual configuration and activation of databases for 'Admin', 'Search User' and 'Suggest URL'. Support of multiple table sets in each db, MySQL query cache, individual index for each db, individual or bulk search in predefined databases. Individual Admin settings for each db and each set of . . .
. . .
always in cache. Separate caches for text and media results. Admin configurable. Follow sitemap files If available, sitemap.xml as well as gzip compressed files will be used to follow the links of a site. If <sitemapindex . . . > is detected, also multiple sitemap files are processed. Periodical Re-index Re-indexing could be performed . . .
. . .
also multiple sitemap files are processed. Periodical Re-index Re-indexing could be performed automatically, repeated every selected time interval. Admin selectable intervals for 3 hours, 12 hours, 1 day, 1 week, or 1 month. Multithreaded indexing In order to reduce the time for indexing, 1-10 parallel running threads might be activated . . .
. . .
1-10 parallel running threads might be activated in Admin settings. Preferred re-index While invoking this option, the admin may select a suitable level for the next index procedure. Thus, only those URLs, containing the according level, will be re-indexed.. Erase Re-index and Continue suspended index procedures Individual (site specific) or . . .
. . .
(site specific) or bulk update of database. Support of XML product feeds Index and search of feed content, inclusive formatting the search results. RDF, RSD, RSS and Atom feed support Index and search of feed content, inclusive RDF 'Dublin Core' tags. Obey / ignore 'preferred' tags in RSD feeds. Follow / ignore CDATA directives. Various . . .
. . .
ignore 'preferred' tags in RSD feeds. Follow / ignore CDATA directives. Various search modes Search with wildcards, Tolerant search, Search strict, Search only in one domain, Search all links of a site, Search for media (link-specific). Add thumbnails to each page presented in text results Admin selectable, this feature will present a web shot . . .
. . .
suspected to contain malware or phishing content. 11 different modes of sorting the text results Admin selectable: -By relevance (weight % ) -By hit counts in full text -Most popular links on top -By indexdate -By URL names -By file suffix -Main URL (domain) on top -Like Google (Top 2 per URL) - Promoted domain on top - Links holding promoted . . .
. . .
on top - Links holding promoted catchwords on top. 5 different modes of sorting the media results Admin selectable: -By title(alphabetic) -By file suffix -By image size -By 'Last queried' -By 'Most popular'. Same results for queries typed with pure vowels, or with accents Will deliver the same results for queries like: caf e and caf é . To be . . .
. . .
in Admin backend. Same results for queries with and without quotes Will deliver the same results for queries like: d'information <-> information dei'largi <-> largi Also Admin selectabe: Equalize the different quotes like: ' ` ´ Same text results for queries with and without ligatures Admin selectable; will deliver the same results . . .
. . .
results for queries with and without ligatures Admin selectable; will deliver the same results for queries like: cœur and coeur . Worked out for Latin ligatures in Unicode (Latin-derived alphabets) and also ligatures used only in phonetic transcription, but not taking into consideration medieval ligatures. Present all results for singular and . . .
. . .
all will become searchable. Dictionaries with 106.800 radicals. Segmentation of Japanese words Segmentation of 5.724 kanji (new, old and half width), hiragana, katakana and jinmeiyo Japanese character writing systems. Transliterate Latin characters into their Greek equivalents Transforms query input alla to find ἀλλὰ and baptismatos to find . . .
. . .
redirections caused by HTTP 301, 302, 303 and 307 status codes. Also obeying JavaScript, sent as HTML content like: <SCRIPT language="javascript"> win.loc="mp.php?mcv=59";</SCRIPT> Follow header redirections, refresh tags and canonical links Automatical forwarding for the indexer. Follow links found in JavaScript and index also the . . .
. . .
and index also the content of document.write Will index JavaScript commands. Detect and follow links like: document.write(' <a href="new12.pdf">All news 2012</a> '); and index the content of: document.write(' this content '); Not indexing content created in real-time by JavaScript. Accept gzip formatted transmission In order . . .
. . .
Converter included for PDF, DOCX, XLSX, ODT, ODS, CSV, PPTX and XLS files Converting also non-Latin text like: Arabic, Cyrillic, Chinese, Greece and Hebrew. Links found in the converted files will be followed. Debug mode Offering detailed information during index/re-index: New links, keywords, frames and media found per link. To be . . .
. . .
Included for 33 languages. Common word lists holding stop words. Included for 25 languages Admin selectable for: Arabic, Bengali, Bulgarian, Catalan, Chinese, Cyrillic, Czech, Danish, Dutch, English, Farsi, Finnish, French, Greek, German, Hindi, Hungarian, Italian, Norwegian, Polish, Portuguese, Romanian, Spanish, Swedish and Turkish. . . .
. . .
links outside are followed. Multiple and nested divs will be attended. Do not index parts of a page defined by HTML5 elements <tag> . . . </tag> Foreseen to cooperate with the HTML5 elements like: section, nav, aside, hgroup, article, header, footer Vice versa function also included in order to index only parts of a page between . . .
. . .
Extension implemented SQLi connector implemented between PHP and a MySQL database. Performed by OOP, also PHP v55 is supported. Compatible with MySQL and MariaDB Proven up to: - MySQL version 8.0.32 - MariaDB version 10.4.28 sp_executesql Ready to run in PHP 8 environment Latest version of Sphider-plus version 4.2024a is proven up to PHP . . .
. . .
8.3.2 [ Proven ] Successfully implemented as search engine on a customer site with a database capacity such as: 25.206 sites 324.595 page links 1.260.698 keywords 169.251 media links. Imprint Private Notice Private Policy . . .

All required information. Introduction Release and Legal Info Installation Documentation Change Log [ Release ] Name: Sphider-plus Version: 4.2024a Released: February 12 2024 Based on original Sphider version 1.3.5 released 2009-12-13 [ Legal Info ] This program is licensed under the GNU GPL v.3 by Rolf Kellner [Tec] tec(a t)sphider-plus.eu . . .
. . .
the GNU GPL v.3 by Rolf Kellner [Tec] tec(a t)sphider-plus.eu Original Sphider GNU GPL licence by Ando Saabas ando(a t)cs.ioc.ee We distribute software in the hope that it will be useful but without any warranty. No author or distributor of this software accepts responsibility to anyone for the consequences of using it or for whether it . . .
. . .
to anyone for the consequences of using it or for whether it serves any particular purpose or works at all unless he says so in writing. This is exactly the same warranty that proprietary software companies offer: none. Imprint Privacy Policy [ Donation ] If you want to use Sphider-plus and also want to promote further development your . . .


All required information. Introduction Release and Legal Info Installation Documentation Change Log [ Release ] Name: Sphider-plus Version: 4.2024a Released: February 12, 2024 Based on original Sphider version 135, released 2009-12-13 [ Legal Info ] This program is licensed under the GNU GPL v.3 by Rolf Kellner [Tec], tec(a t)sphider-plus.eu . . .
. . .
the GNU GPL v.3 by Rolf Kellner [Tec], tec(a t)sphider-plus.eu Original Sphider GNU GPL licence by Ando Saabas, ando(a t)cs.ioc.ee We distribute software in the hope that it will be useful, but without any warranty. No author or distributor of this software accepts responsibility to anyone for the consequences of using it or for whether it . . .
. . .
to anyone for the consequences of using it or for whether it serves any particular purpose or works at all, unless he says so in writing. This is exactly the same warranty that proprietary software companies offer: none. Imprint Privacy Policy [ Donation ] If you want to use Sphider-plus and also want to promote further development, your . . .


All required information. Introduction Release and Legal Info Installation Documentation Change Log [ Release ] Name Sphider-plus Version 4.2024a Released February 12, 2024 Based on original Sphider version 1.3.5, released 2009-12-13 [ Legal Info ] This program is licensed under the GNU GPL v.3 by Rolf Kellner [Tec], tec(a t)sphider-plus.eu . . .
. . .
the GNU GPL v.3 by Rolf Kellner [Tec], tec(a t)sphider-plus.eu Original Sphider GNU GPL licence by Ando Saabas, ando(a t)cs.ioc.ee We distribute software in the hope that it will be useful, but without any warranty. No author or distributor of this software accepts responsibility to anyone for the consequences of using it or for whether it . . .
. . .
to anyone for the consequences of using it or for whether it serves any particular purpose or works at all, unless he says so in writing. This is exactly the same warranty that proprietary software companies offer none. Imprint Privacy Policy [ Donation ] If you want to use Sphider-plus and also want to promote further development, your . . .


All required information. Introduction Release and Legal Info Installation Documentation Change Log [ Release ] Name Sphider-plus Version 4.2024a Released February 12, 2024 Based on original Sphider version 1.3.5, released 2009-12-13 [ Legal Info ] This program is licensed under the GNU GPL v.3 by Rolf Kellner [Tec], tec(a t)sphider-plus.eu . . .
. . .
the GNU GPL v.3 by Rolf Kellner [Tec], tec(a t)sphider-plus.eu Original Sphider GNU GPL licence by Ando Saabas, ando(a t)cs.ioc.ee We distribute software in the hope that it will be useful, but without any warranty. No author or distributor of this software accepts responsibility to anyone for the consequences of using it or for whether it . . .
. . .
to anyone for the consequences of using it or for whether it serves any particular purpose or works at all, unless he says so in writing. This is exactly the same warranty that proprietary software companies offer none. Imprint Privacy Policy [ Donation ] If you want to use Sphider-plus and also want to promote further development, your . . .

5a2 All required information. Introduction Release and Legal Info Installation Documentation Change Log [ Documentation Summary ] Preamble: The info presented here is valid only for the latest release of Sphider-plus. At present version 4.2024a published February 12, 2024 is the actual release. 1. Settings, customizing and statistics 2. Indexing . . .
. . .
2. Indexing 2.1 Various options 2.2 Allow other hosts in same domain 2.3 Word stemming 2.4 Periodical Re-indexing 2.5 Preferred indexing 2.6 Multithreaded indexing 2.7 Create thumbnails during index procedure 2.8 Prevent indexing of known malware and pishing pages 2.9 Follow and create sitemap files 2.10 Use private sitemap instead of global . . .
. . .
Sitemap file 3. Using the indexer from command line 3.1 All options 3.2 Multithreaded indexing 4. Keeping pages, words and files from being indexed 4.1 robots.txt 4.2 Must include / must not include string list 4.3 Ignoring links 4.4 Ignoring parts of a page by <! sphider_noindex > 4.5 Ignoring parts of a page by <div id='abc'> . . .
. . .
charset' 6. Search modes 6.1 Search with wildcards * 6.2 Strict search ! 6.3 Tolerant search 6.4 Link search 6.5 Media search 6.6 Search only in one domain 6.7 Search in categories 6.8 Greek language support 6.9 Block queries 7. Chronological order for result listing 7.1 Text result listing 7.2 Media result listing 8. PDF converter 9. Clean . . .
. . .
of logging data 11. Error messages and Debug mode 12. Delete secondary characters 13. Media search for images, audio streams and videos 13.1 Media indexing 13.2 Not supported media content 13.3 Search for media content 13.4 Statistics for media content 14. Feed support 14.1 XML product feeds 14.2 RDF, RSD, RSS and Atom feeds 15. Result . . .
. . .
Overview 16.2 Definition and configuration 16.3 Activate / disable database 16.4 Backup & Restore of databases 16.5 Copy & and Move 16.6 Enhancing functionality of multiple database support 17. Search in categories 17.1 Hierachical structure 17.2 Parallel structure 18. User suggested sites 19. Vulnerability protection 19.1 Prevent queries . . .
. . .
templates 22.2 Embed the search engine into existing HTML code 22.3 The different style sheet files 23. JSON, XML and RSS result output [ Documentation ] 1. Settings, customizing and statistics If you want to change settings, behavior and design of Sphider-plus, you can do so by means of the Admin interface. There is a wide range of . . .
. . .
interface. There is a wide range of settings foreseen for Sphider-plus. Separated into different submenus like: Sites: - Add Site - Index only the new - Re-index all - Re-index only preferred URLs - Erase Re-index (available also for individual URLs) - Import/export URL list - Approve sites - Banned domains Categories: - Add, edit, delete - . . .
. . .
URL list - Approve sites - Banned domains Categories: - Add, edit, delete - Create new subcategory under Index: - Basic indexing options - Advanced options Clean: - Clean keywords not associated with any link - Clean links not associated with any site - Clean Category table not associated with any site - Clean Media links - Clear Temp table . . .
. . .
- Clear Search log - Clear 'Most Popular Page Links' log - Clear 'Most Popular Media Links' log - Clear Spider log, separate and bulk delete - Clear Thumbnail images, separate and bulk delete - Clear Text cache - Clear Media cache - Clear IDS log file - Clear flood attempts log file - Clear all entries in addurl or banned table - Truncate all . . .
. . .
flood attempts log file - Clear all entries in addurl or banned table - Truncate all tables in database Settings: - General Settings - Index Log Settings - Spider Settings - Search Settings - Order of Result listing - Suggest Options - Page Indexing Weights Database: - Configure up to 5 databases with unlimited number of table sets - Activate . . .
. . .
Database: - Configure up to 5 databases with unlimited number of table sets - Activate separately for 'Admin', 'Search' user and 'Suggest URL user' - Backup / Restore - Copy / Move - Optimize Templates: In order to enable customer's integration of Sphider-plus into existing sites, HTML templates are prepared for Search form Text result . . .
. . .
Search form Text result listing Media result listing Most popular queries etc. Three different designs are offered, which may be selected in submenu 'Settings'. If the layout does not fit the design of your site (which is normal), you may create new designs and modify the appropriate file /templates/My_template/adminstyle.css . . .
. . .
the appropriate file /templates/My_template/adminstyle.css /templates/My_template/userstyle.css Statistics output: - Top keywords (Top 50 with hit counter). - All indexed thumbnails w 3e80 ith ID3 and EXIF info. - Larges pages offering link URL and file size. - Most Popular Searches for text links offering: Link addr., total clicks, last . . .
. . .
Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Searches for media links offering: Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Links (click counter). - Search log offering: Query, Results, Queried at, Time taken, User IP, Country, Host name (Latest100) - Index log offering: . . .
. . .
Country, Host name (Latest100) - Index log offering: File-name, index date and delete option - sitemap log offering: sitemap.xml output sitemap list offering file/page suffixes - IDS log offering: IP, host, query, impact, involved tags, date and time of intrusion. - Flood attempts log offering: IP, query, date and time of flood attempt. - Auto . . .
. . .
attempts log offering: IP, query, date and time of flood attempt. - Auto Re-index log file - Server info offering: Server software, environment, MySQL, PDF-converter, image functions, php.ini file PHP integration, PHP security info. Each item holding lists of details. All text links, media links and thumbnails are active linked. As stated in . . .
. . .
individual Re-indexing, the periodical Re-indexer could be started and aborted in the "Options" menu of each site. 2.5 Preferred Re-indexing Each new URL added to the Admin backend, could be supplied with a priority level. This level will be used by the option 'Re-index only preferred sites'. Level 1 will be interpreted as most important, while . . .
. . .
less all index results will be stored in log files in sub folder /admin/log/ The names of the log files look like: db2_100524-21.47.56_1.html (log file of first thread) db2_100524-21.48.12_2.html (log file of second thread) and is build by the following items: db2 - Number of database. 100524 - Date (May 24, 2010) 21.47.56 - Time when this thread . . .
. . .
spider.php -new1 php spider.php -new2 etc. The IDs will be added to the name of the corresponding log files like: db2_100524-21.47.56_ID1.html (log file of first thread) db2_100524-21.48.12_ID2.html (log file of second thread) IDs could be defined by personal requirements, but the limitations for file names with respect to the OS should be taken . . .
. . .
but will not erase the content of all the other tables. So the check whether the content of a page has changed (MD5sum) is still available for a fast re-index procedure. Once prepared, multithreaded re-indexing could be invoked by starting several threads and adding individual IDs to the option parameter like: php spider.php -erased1 php . . .
. . .
4.4 Ignoring parts of a page Sphider-plus includes an option to exclude parts of pages from being indexed. Thi 775e s can for example be used to prevent search result flooding when certain keywords appear on certain part in most pages (like a header, footer or a menu). Any part of a page between <! sphider_noindex > and <! . . .
. . .
<! sphider_noindex > and <! /sphider_noindex > tags is not indexed, however links in it are followed. 4.5 Ignoring parts of a page by <div id='abc'> Ignoring parts of a page by the <! sphider_noindex > tags requires direct access to the page, because the tags need to be added (edited) to the page. A more flexible method, . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ menu[0-5] / 4.6 Indexing only parts of a page by <div id='abc'> If enabled in Admin settings, the values as defined in the list-file /include/common/divs_use.txt will be used to index only the content between <div . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ table[0-5] / 4.7 Ignore HTML elements defined by <tagname> . . </tagname> This option is foreseen to cooperate with the new HTML5 elements like section, nav, aside, hgroup, article, header, footer, etc. HTML elements . . .
. . .
contain a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */nav[0-5]/ Please keep in mind that element names placed in /include/common/elements_not.txt will be processed case-sensitive. 4.8 Index only HTML elements defined by <tagname> . . </tagname> This is the vice versa . . .
. . .
of all the duplicate content URLs: http://www.example.com/product.php?item=swedish-fish&category=gummy-candy http://www.example.com/product.php?item=swedish-fish&trackingid=1234&sessionid=5678 and Sphider-plus will understand that the duplicates all refer to the canonical URL: http://www.example.com/product.php?item=swedish-fish. The . . .
. . .
charset are supported by the ConvertCharset function and will be used to convert text into UTF-8 Unicode: WINDOWS windows-1250 - Central Europe windows-1251 - Cyrillic windows-1252 - Latin I windows-1253 - Greek windows-1254 - Turkish windows-1255 - Hebrew windows-1256 - Arabic windows-1257 - Baltic windows-1258 - Viet Nam cp874 - Thai - this . . .
. . .
- Baltic windows-1258 - Viet Nam cp874 - Thai - this file is also for DOS DOS cp437 - Latin US cp737 - Greek cp775 - BaltRim cp850 - Latin1 cp852 - Latin2 cp855 - Cyrylic cp857 - Turkish cp860 - Portuguese cp861 - Iceland cp862 - Hebrew cp863 - Canada cp864 - Arabic cp865 - Nordic cp866 - Cyrylic Russian (this is the one, used in IE . . .
. . .
IE Cyrillic (DOS) ) cp869 - Greek2 MAC (Apple) x-mac-cyrillic x-mac-greek x-mac-icelandic x-mac-ce x-mac-roman ISO iso-8859-1 iso-8859-2 iso-8859-3 iso-8859-4 iso-8859-5 iso-8859-6 iso-8859-7 iso-8859-8 iso-8859-9 iso-8859-10 iso-8859-11 iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 . . .
. . .
iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 cp500 cp856 cp875 cp1006 cp1026 koi8-r (Cyrillic) koi8-u (Cyrillic Ukrainian) nextstep us-ascii us-ascii-quotes DSP implementation for NeXT stdenc symbol zdingbat And specially for old Polish programs: mazovia This list is to be read . . .
. . .
Access denied; you need the RELOAD privilege. . . Top 10. Enable real-time output of logging data Up to version 1.5 of Sphider-plus, during index / re-index there was no printout available because: - Several servers, especially on Win32, buffer the output from the script until it terminates before transmitting the results to the browser. - . . .
. . .
is seen. - Some versions of Microsoft Internet Explorer only start to display the page after they have received 256 bytes of output. As progress was not presented during index / re- index procedure, waiting for results became a pain in the neck. Selectable in Admin setting together with the update interval (1 - 10 seconds), AJAX technology was . . .
. . .
characters in front of words Warning: This option should be used with special care and not be activated for non ISO-8859 charsets. Some special characters as part of the word ending might be erased by accidental. Top 13. Media search for images, audio streams and videos 13.1 Media indexing Index of media files is enabled by separated Admin . . .
. . .
and 'Found at' - Total clicks - Last clicked - Query input 'Indexed Image Thumbnails' presenting: - Thumbnail 150 x 100 pixel - Image details like title, filename size of original image, link- and thumb-id - Option to delete the thumbnail In order to open the media files all tables contain active links. Media results are also stored in . . .
. . .
Settings' menu of the admin backend: First option: For results of XML product feeds, present the text extract of 350 characters as part of product attributes. If this option is not activated, all products of the XML product feed will be presented in search results, and the hits of the query string will be highlighted in all involved products. . . .
. . .
content of the database tables (those with the same table prefix) will be destroyed by the restore procedure. 16.5 Copy & Move This section of the Database Management will present only those databases, which are correctly configured, assigned and do have a set of installed tables as described in chapter Definition and configuration. This . . .
. . .
results. Never the less to find these x results, the complete database had to been browsed. Starting with version 2.5, an additional clean option is offered as part of the Admin backend. Main advantage of this option is a significant reduction of the search time for any query, because the content of the db could be limited to offer only x . . .
. . .
<link>http://www.abc.de/images/warp.gif</link> <title>warp.gif</title> <xsize>635</xsize>; <y_size>98</y_size> </media_result> <media_result> <num>2</num> <type>audio</type> <url>http://www.abc.de/index.php</url> . . .
. . .
the following content: {"query":"colour","ip":"::1","host_name":"guard_007-hoster","query_time":"2016-03-18 10:56:23 AM","consumed":0.016,"total_results":2,"num_of_results":2,"from":1,"to":2,"text_results":[{"num":1,"weight":"100.0","url":"http:\/\/www.english.le-piaggie.info\/Info_eng.pdf","title":" Info_eng","fulltxt":" . . . . . .
. . .
placed on the ground floor, today's big kitchen with the heating fireplace for open and close mode of . . . ","page_size":"5,661.6kb","domain_name":"www.english.le-piaggie.info"},{"num":2,"weight":"50.0","url":"http:\/\/www.english.le-piaggie.info\/html\/description.html","title":" Description","fulltxt":" . . . cable. Additional active . . .
. . .
D.O.C.G. as far as to the Pratomagno mountains 160 olive-trees Detached house, about 800 m to reach the . . . ","page_size":"26.4 kb","domain_name":"www.english.le-piaggie.info"}]} Top . . .


5a2 All required information. Introduction Release and Legal Info Installation Documentation Change Log [ Documentation Summary ] Preamble: The info presented here is valid only for the latest release of Sphider-plus. At present version 4.2024a published February 12 2024 is the actual release. 1. Settings customizing and statistics 2. Indexing . . .
. . .
2. Indexing 2.1 Various options 2.2 Allow other hosts in same domain 2.3 Word stemming 2.4 Periodical Re-indexing 2.5 Preferred indexing 2.6 Multithreaded indexing 2.7 Create thumbnails during index procedure 2.8 Prevent indexing of known malware and pishing pages 2.9 Follow and create sitemap files 2.10 Use private sitemap instead of global . . .
. . .
Sitemap file 3. Using the indexer from command line 3.1 All options 3.2 Multithreaded indexing 4. Keeping pages words and files from being indexed 4.1 robots.txt 4.2 Must include / must not include string list 4.3 Ignoring links 4.4 Ignoring parts of a page by <! sphider_noindex > 4.5 Ignoring parts of a page by <div id='abc'> . . .
. . .
charset' 6. Search modes 6.1 Search with wildcards * 6.2 Strict search ! 6.3 Tolerant search 6.4 Link search 6.5 Media search 6.6 Search only in one domain 6.7 Search in categories 6.8 Greek language support 6.9 Block queries 7. Chronological order for result listing 7.1 Text result listing 7.2 Media result listing 8. PDF converter 9. Clean . . .
. . .
of logging data 11. Error messages and Debug mode 12. Delete secondary characters 13. Media search for images audio streams and videos 13.1 Media indexing 13.2 Not supported media content 13.3 Search for media content 13.4 Statistics for media content 14. Feed support 14.1 XML product feeds 14.2 RDF RSD RSS and Atom feeds 15. Result . . .
. . .
Overview 16.2 Definition and configuration 16.3 Activate / disable database 16.4 Backup & Restore of databases 16.5 Copy & and Move 16.6 Enhancing functionality of multiple database support 17. Search in categories 17.1 Hierachical structure 17.2 Parallel structure 18. User suggested sites 19. Vulnerability protection 19.1 Prevent queries . . .
. . .
templates 22.2 Embed the search engine into existing HTML code 22.3 The different style sheet files 23. JSON XML and RSS result output [ Documentation ] 1. Settings customizing and statistics If you want to change settings behavior and design of Sphider-plus you can do so by means of the Admin interface. There is a wide range of . . .
. . .
interface. There is a wide range of settings foreseen for Sphider-plus. Separated into different submenus like: Sites: - Add Site - Index only the new - Re-index all - Re-index only preferred URLs - Erase Re-index (available also for individual URLs) - Import/export URL list - Approve sites - Banned domains Categories: - Add edit delete - . . .
. . .
URL list - Approve sites - Banned domains Categories: - Add edit delete - Create new subcategory under Index: - Basic indexing options - Advanced options Clean: - Clean keywords not associated with any link - Clean links not associated with any site - Clean Category table not associated with any site - Clean Media links - Clear Temp table . . .
. . .
- Clear Search log - Clear 'Most Popular Page Links' log - Clear 'Most Popular Media Links' log - Clear Spider log separate and bulk delete - Clear Thumbnail images separate and bulk delete - Clear Text cache - Clear Media cache - Clear IDS log file - Clear flood attempts log file - Clear all entries in addurl or banned table - Truncate all . . .
. . .
flood attempts log file - Clear all entries in addurl or banned table - Truncate all tables in database Settings: - General Settings - Index Log Settings - Spider Settings - Search Settings - Order of Result listing - Suggest Options - Page Indexing Weights Database: - Configure up to 5 databases with unlimited number of table sets - Activate . . .
. . .
Database: - Configure up to 5 databases with unlimited number of table sets - Activate separately for 'Admin' 'Search' user and 'Suggest URL user' - Backup / Restore - Copy / Move - Optimize Templates: In order to enable customer's integration of Sphider-plus into existing sites HTML templates are prepared for Search form Text result . . .
. . .
Search form Text result listing Media result listing Most popular queries etc. Three different designs are offered which may be selected in submenu 'Settings'. If the layout does not fit the design of your site (which is normal) you may create new designs and modify the appropriate file /templates/My_template/adminstyle.css . . .
. . .
the appropriate file /templates/My_template/adminstyle.css /templates/My_template/userstyle.css Statistics output: - Top keywords (Top 50 with hit counter). - All indexed thumbnails w 3e80 ith ID3 and EXIF info. - Larges pages offering link URL and file size. - Most Popular Searches for text links offering: Link addr. total clicks last . . .
. . .
Link addr. total clicks last clicked last query (Top 50) - Most Popular Searches for media links offering: Link addr. total clicks last clicked last query (Top 50) - Most Popular Links (click counter). - Search log offering: Query Results Queried at Time taken User IP Country Host name (Latest100) - Index log offering: . . .
. . .
Country Host name (Latest100) - Index log offering: File-name index date and delete option - sitemap log offering: sitemap.xml output sitemap list offering file/page suffixes - IDS log offering: IP host query impact involved tags date and time of intrusion. - Flood attempts log offering: IP query date and time of flood attempt. - Auto . . .
. . .
attempts log offering: IP query date and time of flood attempt. - Auto Re-index log file - Server info offering: Server software environment MySQL PDF-converter image functions php.ini file PHP integration PHP security info. Each item holding lists of details. All text links media links and thumbnails are active linked. As stated in . . .
. . .
individual Re-indexing the periodical Re-indexer could be started and aborted in the "Options" menu of each site. 2.5 Preferred Re-indexing Each new URL added to the Admin backend could be supplied with a priority level. This level will be used by the option 'Re-index only preferred sites'. Level 1 will be interpreted as most important while . . .
. . .
less all index results will be stored in log files in sub folder /admin/log/ The names of the log files look like: db2_100524-21.47.56_1.html (log file of first thread) db2_100524-21.48.12_2.html (log file of second thread) and is build by the following items: db2 - Number of database. 100524 - Date (May 24 2010) 21.47.56 - Time when this thread . . .
. . .
spider.php -new1 php spider.php -new2 etc. The IDs will be added to the name of the corresponding log files like: db2_100524-21.47.56_ID1.html (log file of first thread) db2_100524-21.48.12_ID2.html (log file of second thread) IDs could be defined by personal requirements but the limitations for file names with respect to the OS should be taken . . .
. . .
but will not erase the content of all the other tables. So the check whether the content of a page has changed (MD5sum) is still available for a fast re-index procedure. Once prepared multithreaded re-indexing could be invoked by starting several threads and adding individual IDs to the option parameter like: php spider.php -erased1 php . . .
. . .
4.4 Ignoring parts of a page Sphider-plus includes an option to exclude parts of pages from being indexed. Thi 775e s can for example be used to prevent search result flooding when certain keywords appear on certain part in most pages (like a header footer or a menu). Any part of a page between <! sphider_noindex > and <! . . .
. . .
<! sphider_noindex > and <! /sphider_noindex > tags is not indexed however links in it are followed. 4.5 Ignoring parts of a page by <div id='abc'> Ignoring parts of a page by the <! sphider_noindex > tags requires direct access to the page because the tags need to be added (edited) to the page. A more flexible method . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ menu[0-5] / 4.6 Indexing only parts of a page by <div id='abc'> If enabled in Admin settings the values as defined in the list-file /include/common/divs_use.txt will be used to index only the content between <div . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ table[0-5] / 4.7 Ignore HTML elements defined by <tagname> . . </tagname> This option is foreseen to cooperate with the new HTML5 elements like section nav aside hgroup article header footer etc. HTML elements . . .
. . .
contain a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */nav[0-5]/ Please keep in mind that element names placed in /include/common/elements_not.txt will be processed case-sensitive. 4.8 Index only HTML elements defined by <tagname> . . </tagname> This is the vice versa . . .
. . .
of all the duplicate content URLs: http://www.example.com/product.php?item=swedish-fish&category=gummy-candy http://www.example.com/product.php?item=swedish-fish&trackingid=1234&sessionid=5678 and Sphider-plus will understand that the duplicates all refer to the canonical URL: http://www.example.com/product.php?item=swedish-fish. The . . .
. . .
charset are supported by the ConvertCharset function and will be used to convert text into UTF-8 Unicode: WINDOWS windows-1250 - Central Europe windows-1251 - Cyrillic windows-1252 - Latin I windows-1253 - Greek windows-1254 - Turkish windows-1255 - Hebrew windows-1256 - Arabic windows-1257 - Baltic windows-1258 - Viet Nam cp874 - Thai - this . . .
. . .
- Baltic windows-1258 - Viet Nam cp874 - Thai - this file is also for DOS DOS cp437 - Latin US cp737 - Greek cp775 - BaltRim cp850 - Latin1 cp852 - Latin2 cp855 - Cyrylic cp857 - Turkish cp860 - Portuguese cp861 - Iceland cp862 - Hebrew cp863 - Canada cp864 - Arabic cp865 - Nordic cp866 - Cyrylic Russian (this is the one used in IE . . .
. . .
IE Cyrillic (DOS) ) cp869 - Greek2 MAC (Apple) x-mac-cyrillic x-mac-greek x-mac-icelandic x-mac-ce x-mac-roman ISO iso-8859-1 iso-8859-2 iso-8859-3 iso-8859-4 iso-8859-5 iso-8859-6 iso-8859-7 iso-8859-8 iso-8859-9 iso-8859-10 iso-8859-11 iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 . . .
. . .
iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 cp500 cp856 cp875 cp1006 cp1026 koi8-r (Cyrillic) koi8-u (Cyrillic Ukrainian) nextstep us-ascii us-ascii-quotes DSP implementation for NeXT stdenc symbol zdingbat And specially for old Polish programs: mazovia This list is to be read . . .
. . .
Access denied; you need the RELOAD privilege. . . Top 10. Enable real-time output of logging data Up to version 1.5 of Sphider-plus during index / re-index there was no printout available because: - Several servers especially on Win32 buffer the output from the script until it terminates before transmitting the results to the browser. - . . .
. . .
is seen. - Some versions of Microsoft Internet Explorer only start to display the page after they have received 256 bytes of output. As progress was not presented during index / re- index procedure waiting for results became a pain in the neck. Selectable in Admin setting together with the update interval (1 - 10 seconds) AJAX technology was . . .
. . .
characters in front of words Warning: This option should be used with special care and not be activated for non ISO-8859 charsets. Some special characters as part of the word ending might be erased by accidental. Top 13. Media search for images audio streams and videos 13.1 Media indexing Index of media files is enabled by separated Admin . . .
. . .
and 'Found at' - Total clicks - Last clicked - Query input 'Indexed Image Thumbnails' presenting: - Thumbnail 150 x 100 pixel - Image details like title filename size of original image link- and thumb-id - Option to delete the thumbnail In order to open the media files all tables contain active links. Media results are also stored in . . .
. . .
Settings' menu of the admin backend: First option: For results of XML product feeds present the text extract of 350 characters as part of product attributes. If this option is not activated all products of the XML product feed will be presented in search results and the hits of the query string will be highlighted in all involved products. . . .
. . .
content of the database tables (those with the same table prefix) will be destroyed by the restore procedure. 16.5 Copy & Move This section of the Database Management will present only those databases which are correctly configured assigned and do have a set of installed tables as described in chapter Definition and configuration. This . . .
. . .
results. Never the less to find these x results the complete database had to been browsed. Starting with version 2.5 an additional clean option is offered as part of the Admin backend. Main advantage of this option is a significant reduction of the search time for any query because the content of the db could be limited to offer only x . . .
. . .
<link>http://www.abc.de/images/warp.gif</link> <title>warp.gif</title> <x_size>635</x_size> <y_size>98</y_size> </media_result> <media_result> <num>2</num> <type>audio</type> <url>http://www.abc.de/index.php</url> . . .
. . .
the following content: {"query":"colour""ip":"::1""host_name":"guard_007-hoster""query_time":"2016-03-18 10:56:23 AM""consumed":0.016"total_results":2"num_of_results":2"from":1"to":2"text_results":[{"num":1"weight":"100.0""url":"http:\/\/www.english.le-piaggie.info\/Info_eng.pdf""title":" Info_eng""fulltxt":" . . . . . .
. . .
placed on the ground floor today's big kitchen with the heating fireplace for open and close mode of . . . ""page_size":"5661.6kb""domain_name":"www.english.le-piaggie.info"}{"num":2"weight":"50.0""url":"http:\/\/www.english.le-piaggie.info\/html\/description.html""title":" Description""fulltxt":" . . . cable. Additional active . . .
. . .
D.O.C.G. as far as to the Pratomagno mountains 160 olive-trees Detached house about 800 m to reach the . . . ""page_size":"26.4 kb""domain_name":"www.english.le-piaggie.info"}]} Top . . .


5a2 All required information. Introduction Release and Legal Info Installation Documentation Change Log [ Documentation Summary ] Preamble: The info presented here is valid only for the latest release of Sphider-plus. At present version 4.2024a published February 12, 2024 is the actual release. 1. Settings, customizing and statistics 2. Indexing . . .
. . .
2. Indexing 2.1 Various options 2.2 Allow other hosts in same domain 2.3 Word stemming 2.4 Periodical Re-indexing 2.5 Preferred indexing 2.6 Multithreaded indexing 2.7 Create thumbnails during index procedure 2.8 Prevent indexing of known malware and pishing pages 2.9 Follow and create sitemap files 2.10 Use private sitemap instead of global . . .
. . .
Sitemap file 3. Using the indexer from command line 3.1 All options 3.2 Multithreaded indexing 4. Keeping pages, words and files from being indexed 4.1 robots.txt 4.2 Must include / must not include string list 4.3 Ignoring links 4.4 Ignoring parts of a page by <! sphider_noindex > 4.5 Ignoring parts of a page by <div id='abc'> . . .
. . .
charset' 6. Search modes 6.1 Search with wildcards * 6.2 Strict search ! 6.3 Tolerant search 6.4 Link search 6.5 Media search 6.6 Search only in one domain 6.7 Search in categories 6.8 Greek language support 6.9 Block queries 7. Chronological order for result listing 7.1 Text result listing 7.2 Media result listing 8. PDF converter 9. Clean . . .
. . .
of logging data 11. Error messages and Debug mode 12. Delete secondary characters 13. Media search for images, audio streams and videos 13.1 Media indexing 13.2 Not supported media content 13.3 Search for media content 13.4 Statistics for media content 14. Feed support 14.1 XML product feeds 14.2 RDF, RSD, RSS and Atom feeds 15. Result . . .
. . .
Overview 16.2 Definition and configuration 16.3 Activate / disable database 16.4 Backup & Restore of databases 16.5 Copy & and Move 16.6 Enhancing functionality of multiple database support 17. Search in categories 17.1 Hierachical structure 17.2 Parallel structure 18. User suggested sites 19. Vulnerability protection 19.1 Prevent queries . . .
. . .
templates 22.2 Embed the search engine into existing HTML code 22.3 The different style sheet files 23. JSON, XML and RSS result output [ Documentation ] 1. Settings, customizing and statistics If you want to change settings, behavior and design of Sphider-plus, you can do so by means of the Admin interface. There is a wide range of . . .
. . .
interface. There is a wide range of settings foreseen for Sphider-plus. Separated into different submenus like: Sites: - Add Site - Index only the new - Re-index all - Re-index only preferred URLs - Erase Re-index (available also for individual URLs) - Import/export URL list - Approve sites - Banned domains Categories: - Add, edit, delete - . . .
. . .
URL list - Approve sites - Banned domains Categories: - Add, edit, delete - Create new subcategory under Index: - Basic indexing options - Advanced options Clean: - Clean keywords not associated with any link - Clean links not associated with any site - Clean Category table not associated with any site - Clean Media links - Clear Temp table . . .
. . .
- Clear Search log - Clear 'Most Popular Page Links' log - Clear 'Most Popular Media Links' log - Clear Spider log, separate and bulk delete - Clear Thumbnail images, separate and bulk delete - Clear Text cache - Clear Media cache - Clear IDS log file - Clear flood attempts log file - Clear all entries in addurl or banned table - Truncate all . . .
. . .
flood attempts log file - Clear all entries in addurl or banned table - Truncate all tables in database Settings: - General Settings - Index Log Settings - Spider Settings - Search Settings - Order of Result listing - Suggest Options - Page Indexing Weights Database: - Configure up to 5 databases with unlimited number of table sets - Activate . . .
. . .
Database: - Configure up to 5 databases with unlimited number of table sets - Activate separately for 'Admin', 'Search' user and 'Suggest URL user' - Backup / Restore - Copy / Move - Optimize Templates: In order to enable customer's integration of Sphider-plus into existing sites, HTML templates are prepared for Search form Text result . . .
. . .
Search form Text result listing Media result listing Most popular queries etc. Three different designs are offered, which may be selected in submenu 'Settings'. If the layout does not fit the design of your site (which is normal), you may create new designs and modify the appropriate file /templates/My_template/adminstyle.css . . .
. . .
the appropriate file /templates/My_template/adminstyle.css /templates/My_template/userstyle.css Statistics output: - Top keywords (Top 50 with hit counter). - All indexed thumbnails w 3e80 ith ID3 and EXIF info. - Larges pages offering link URL and file size. - Most Popular Searches for text links offering: Link addr., total clicks, last . . .
. . .
Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Searches for media links offering: Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Links (click counter). - Search log offering: Query, Results, Queried at, Time taken, User IP, Country, Host name (Latest100) - Index log offering: . . .
. . .
Country, Host name (Latest100) - Index log offering: File-name, index date and delete option - sitemap log offering: sitemap.xml output sitemap list offering file/page suffixes - IDS log offering: IP, host, query, impact, involved tags, date and time of intrusion. - Flood attempts log offering: IP, query, date and time of flood attempt. - Auto . . .
. . .
attempts log offering: IP, query, date and time of flood attempt. - Auto Re-index log file - Server info offering: Server software, environment, MySQL, PDF-converter, image functions, php.ini file PHP integration, PHP security info. Each item holding lists of details. All text links, media links and thumbnails are active linked. As stated in . . .
. . .
individual Re-indexing, the periodical Re-indexer could be started and aborted in the "Options" menu of each site. 2.5 Preferred Re-indexing Each new URL added to the Admin backend, could be supplied with a priority level. This level will be used by the option 'Re-index only preferred sites'. Level 1 will be interpreted as most important, while . . .
. . .
less all index results will be stored in log files in sub folder /admin/log/ The names of the log files look like: db2_100524-21.47.56_1.html (log file of first thread) db2_100524-21.48.12_2.html (log file of second thread) and is build by the following items: db2 - Number of database. 100524 - Date (May 24, 2010) 21.47.56 - Time when this thread . . .
. . .
spider.php -new1 php spider.php -new2 etc. The IDs will be added to the name of the corresponding log files like: db2_100524-21.47.56_ID1.html (log file of first thread) db2_100524-21.48.12_ID2.html (log file of second thread) IDs could be defined by personal requirements, but the limitations for file names with respect to the OS should be taken . . .
. . .
but will not erase the content of all the other tables. So the check whether the content of a page has changed (MD5sum) is still available for a fast re-index procedure. Once prepared, multithreaded re-indexing could be invoked by starting several threads and adding individual IDs to the option parameter like: php spider.php -erased1 php . . .
. . .
4.4 Ignoring parts of a page Sphider-plus includes an option to exclude parts of pages from being indexed. Thi 775e s can for example be used to prevent search result flooding when certain keywords appear on certain part in most pages (like a header, footer or a menu). Any part of a page between <! sphider_noindex > and <! . . .
. . .
<! sphider_noindex > and <! /sphider_noindex > tags is not indexed, however links in it are followed. 4.5 Ignoring parts of a page by <div id='abc'> Ignoring parts of a page by the <! sphider_noindex > tags requires direct access to the page, because the tags need to be added (edited) to the page. A more flexible method, . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ menu[0-5] / 4.6 Indexing only parts of a page by <div id='abc'> If enabled in Admin settings, the values as defined in the list-file /include/common/divs_use.txt will be used to index only the content between <div . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ table[0-5] / 4.7 Ignore HTML elements defined by <tagname> . . </tagname> This option is foreseen to cooperate with the new HTML5 elements like section, nav, aside, hgroup, article, header, footer, etc. HTML elements . . .
. . .
contain a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */nav[0-5]/ Please keep in mind that element names placed in /include/common/elements_not.txt will be processed case-sensitive. 4.8 Index only HTML elements defined by <tagname> . . </tagname> This is the vice versa . . .
. . .
of all the duplicate content URLs: http://www.example.com/product.php?item=swedish-fish&category=gummy-candy http://www.example.com/product.php?item=swedish-fish&trackingid=1234&sessionid=5678 and Sphider-plus will understand that the duplicates all refer to the canonical URL: http://www.example.com/product.php?item=swedish-fish. The . . .
. . .
charset are supported by the ConvertCharset function and will be used to convert text into UTF-8 Unicode: WINDOWS windows-1250 - Central Europe windows-1251 - Cyrillic windows-1252 - Latin I windows-1253 - Greek windows-1254 - Turkish windows-1255 - Hebrew windows-1256 - Arabic windows-1257 - Baltic windows-1258 - Viet Nam cp874 - Thai - this . . .
. . .
- Baltic windows-1258 - Viet Nam cp874 - Thai - this file is also for DOS DOS cp437 - Latin US cp737 - Greek cp775 - BaltRim cp850 - Latin1 cp852 - Latin2 cp855 - Cyrylic cp857 - Turkish cp860 - Portuguese cp861 - Iceland cp862 - Hebrew cp863 - Canada cp864 - Arabic cp865 - Nordic cp866 - Cyrylic Russian (this is the one, used in IE . . .
. . .
IE Cyrillic (DOS) ) cp869 - Greek2 MAC (Apple) x-mac-cyrillic x-mac-greek x-mac-icelandic x-mac-ce x-mac-roman ISO iso-8859-1 iso-8859-2 iso-8859-3 iso-8859-4 iso-8859-5 iso-8859-6 iso-8859-7 iso-8859-8 iso-8859-9 iso-8859-10 iso-8859-11 iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 . . .
. . .
iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 cp500 cp856 cp875 cp1006 cp1026 koi8-r (Cyrillic) koi8-u (Cyrillic Ukrainian) nextstep us-ascii us-ascii-quotes DSP implementation for NeXT stdenc symbol zdingbat And specially for old Polish programs: mazovia This list is to be read . . .
. . .
Access denied; you need the RELOAD privilege. . . Top 10. Enable real-time output of logging data Up to version 15 of Sphider-plus, during index / re-index there was no printout available because: - Several servers, especially on Win32, buffer the output from the script until it terminates before transmitting the results to the browser. - . . .
. . .
is seen. - Some versions of Microsoft Internet Explorer only start to display the page after they have received 256 bytes of output. As progress was not presented during index / re- index procedure, waiting for results became a pain in the neck. Selectable in Admin setting together with the update interval (1 - 10 seconds), AJAX technology was . . .
. . .
characters in front of words Warning: This option should be used with special care and not be activated for non ISO-8859 charsets. Some special characters as part of the word ending might be erased by accidental. Top 13. Media search for images, audio streams and videos 13.1 Media indexing Index of media files is enabled by separated Admin . . .
. . .
and 'Found at' - Total clicks - Last clicked - Query input 'Indexed Image Thumbnails' presenting: - Thumbnail 150 x 100 pixel - Image details like title, filename size of original image, link- and thumb-id - Option to delete the thumbnail In order to open the media files all tables contain active links. Media results are also stored in . . .
. . .
Settings' menu of the admin backend: First option: For results of XML product feeds, present the text extract of 350 characters as part of product attributes. If this option is not activated, all products of the XML product feed will be presented in search results, and the hits of the query string will be highlighted in all involved products. . . .
. . .
content of the database tables (those with the same table prefix) will be destroyed by the restore procedure. 16.5 Copy & Move This section of the Database Management will present only those databases, which are correctly configured, assigned and do have a set of installed tables as described in chapter Definition and configuration. This . . .
. . .
results. Never the less to find these x results, the complete database had to been browsed. Starting with version 2.5, an additional clean option is offered as part of the Admin backend. Main advantage of this option is a significant reduction of the search time for any query, because the content of the db could be limited to offer only x . . .
. . .
<link>http://www.abc.de/images/warp.gif</link> <title>warp.gif</title> <x_size>635</x_size> <y_size>98</y_size> </media_result> <media_result> <num>2</num> <type>audio</type> <url>http://www.abc.de/index.php</url> . . .
. . .
the following content: {"query":"colour","ip":"::1","host_name":"guard_007-hoster","query_time":"2016-03-18 10:56:23 AM","consumed":0.016,"total_results":2,"num_of_results":2,"from":1,"to":2,"text_results":[{"num":1,"weight":"100.0","url":"http:\/\/www.english.le-piaggie.info\/Info_eng.pdf","title":" Info_eng","fulltxt":" . . . . . .
. . .
placed on the ground floor, today's big kitchen with the heating fireplace for open and close mode of . . . ","page_size":"5,661.6kb","domain_name":"www.english.le-piaggie.info"},{"num":2,"weight":"50.0","url":"http:\/\/www.english.le-piaggie.info\/html\/description.html","title":" Description","fulltxt":" . . . cable. Additional active . . .
. . .
D.O.C.G. as far as to the Pratomagno mountains 160 olive-trees Detached house, about 800 m to reach the . . . ","page_size":"26.4 kb","domain_name":"www.english.le-piaggie.info"}]} Top . . .


5a2 All required information. Introduction Release and Legal Info Installation Documentation Change Log [ Documentation Summary ] Preamble: The info presented here is valid only for the latest release of Sphider-plus. At present version 4.2024a published February 12, 2024 is the actual release. 1. Settings, customizing and statistics 2. Indexing . . .
. . .
2. Indexing 2.1 Various options 2.2 Allow other hosts in same domain 2.3 Word stemming 2.4 Periodical Re-indexing 2.5 Preferred indexing 2.6 Multithreaded indexing 2.7 Create thumbnails during index procedure 2.8 Prevent indexing of known malware and pishing pages 2.9 Follow and create sitemap files 2.10 Use private sitemap instead of global . . .
. . .
Sitemap file 3. Using the indexer from command line 3.1 All options 3.2 Multithreaded indexing 4. Keeping pages, words and files from being indexed 4.1 robots.txt 4.2 Must include / must not include string list 4.3 Ignoring links 4.4 Ignoring parts of a page by <! sphider_noindex > 4.5 Ignoring parts of a page by <div id='abc'> . . .
. . .
charset' 6. Search modes 6.1 Search with wildcards * 6.2 Strict search ! 6.3 Tolerant search 6.4 Link search 6.5 Media search 6.6 Search only in one domain 6.7 Search in categories 6.8 Greek language support 6.9 Block queries 7. Chronological order for result listing 7.1 Text result listing 7.2 Media result listing 8. PDF converter 9. Clean . . .
. . .
of logging data 11. Error messages and Debug mode 12. Delete secondary characters 13. Media search for images, audio streams and videos 13.1 Media indexing 13.2 Not supported media content 13.3 Search for media content 13.4 Statistics for media content 14. Feed support 14.1 XML product feeds 14.2 RDF, RSD, RSS and Atom feeds 15. Result . . .
. . .
Overview 16.2 Definition and configuration 16.3 Activate / disable database 16.4 Backup & Restore of databases 16.5 Copy & and Move 16.6 Enhancing functionality of multiple database support 17. Search in categories 17.1 Hierachical structure 17.2 Parallel structure 18. User suggested sites 19. Vulnerability protection 19.1 Prevent queries . . .
. . .
templates 22.2 Embed the search engine into existing HTML code 22.3 The different style sheet files 23. JSON, XML and RSS result output [ Documentation ] 1. Settings, customizing and statistics If you want to change settings, behavior and design of Sphider-plus, you can do so by means of the Admin interface. There is a wide range of . . .
. . .
interface. There is a wide range of settings foreseen for Sphider-plus. Separated into different submenus like: Sites: - Add Site - Index only the new - Re-index all - Re-index only preferred URLs - Erase Re-index (available also for individual URLs) - Import/export URL list - Approve sites - Banned domains Categories: - Add, edit, delete - . . .
. . .
URL list - Approve sites - Banned domains Categories: - Add, edit, delete - Create new subcategory under Index: - Basic indexing options - Advanced options Clean: - Clean keywords not associated with any link - Clean links not associated with any site - Clean Category table not associated with any site - Clean Media links - Clear Temp table . . .
. . .
- Clear Search log - Clear 'Most Popular Page Links' log - Clear 'Most Popular Media Links' log - Clear Spider log, separate and bulk delete - Clear Thumbnail images, separate and bulk delete - Clear Text cache - Clear Media cache - Clear IDS log file - Clear flood attempts log file - Clear all entries in addurl or banned table - Truncate all . . .
. . .
flood attempts log file - Clear all entries in addurl or banned table - Truncate all tables in database Settings: - General Settings - Index Log Settings - Spider Settings - Search Settings - Order of Result listing - Suggest Options - Page Indexing Weights Database: - Configure up to 5 databases with unlimited number of table sets - Activate . . .
. . .
Database: - Configure up to 5 databases with unlimited number of table sets - Activate separately for 'Admin', 'Search' user and 'Suggest URL user' - Backup / Restore - Copy / Move - Optimize Templates: In order to enable customer's integration of Sphider-plus into existing sites, HTML templates are prepared for Search form Text result . . .
. . .
Search form Text result listing Media result listing Most popular queries etc. Three different designs are offered, which may be selected in submenu 'Settings'. If the layout does not fit the design of your site (which is normal), you may create new designs and modify the appropriate file /templates/My_template/adminstyle.css . . .
. . .
the appropriate file /templates/My_template/adminstyle.css /templates/My_template/userstyle.css Statistics output: - Top keywords (Top 50 with hit counter). - All indexed thumbnails w 3e80 ith ID3 and EXIF info. - Larges pages offering link URL and file size. - Most Popular Searches for text links offering: Link addr., total clicks, last . . .
. . .
Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Searches for media links offering: Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Links (click counter). - Search log offering: Query, Results, Queried at, Time taken, User IP, Country, Host name (Latest100) - Index log offering: . . .
. . .
Country, Host name (Latest100) - Index log offering: File-name, index date and delete option - sitemap log offering: sitemap.xml output sitemap list offering file/page suffixes - IDS log offering: IP, host, query, impact, involved tags, date and time of intrusion. - Flood attempts log offering: IP, query, date and time of flood attempt. - Auto . . .
. . .
attempts log offering: IP, query, date and time of flood attempt. - Auto Re-index log file - Server info offering: Server software, environment, MySQL, PDF-converter, image functions, php.ini file PHP integration, PHP security info. Each item holding lists of details. All text links, media links and thumbnails are active linked. As stated in . . .
. . .
individual Re-indexing, the periodical Re-indexer could be started and aborted in the "Options" menu of each site. 2.5 Preferred Re-indexing Each new URL added to the Admin backend, could be supplied with a priority level. This level will be used by the option 'Re-index only preferred sites'. Level 1 will be interpreted as most important, while . . .
. . .
less all index results will be stored in log files in sub folder /admin/log/ The names of the log files look like: db2_100524-21.47.56_1.html (log file of first thread) db2_100524-21.48.12_2.html (log file of second thread) and is build by the following items: db2 - Number of database. 100524 - Date (May 24, 2010) 21.47.56 - Time when this thread . . .
. . .
spider.php -new1 php spider.php -new2 etc. The IDs will be added to the name of the corresponding log files like: db2_100524-21.47.56_ID1.html (log file of first thread) db2_100524-21.48.12_ID2.html (log file of second thread) IDs could be defined by personal requirements, but the limitations for file names with respect to the OS should be taken . . .
. . .
but will not erase the content of all the other tables. So the check whether the content of a page has changed (MD5sum) is still available for a fast re-index procedure. Once prepared, multithreaded re-indexing could be invoked by starting several threads and adding individual IDs to the option parameter like: php spider.php -erased1 php . . .
. . .
4.4 Ignoring parts of a page Sphider-plus includes an option to exclude parts of pages from being indexed. Thi 775e s can for example be used to prevent search result flooding when certain keywords appear on certain part in most pages (like a header, footer or a menu). Any part of a page between <! sphider_noindex > and <! . . .
. . .
<! sphider_noindex > and <! /sphider_noindex > tags is not indexed, however links in it are followed. 4.5 Ignoring parts of a page by <div id='abc'> Ignoring parts of a page by the <! sphider_noindex > tags requires direct access to the page, because the tags need to be added (edited) to the page. A more flexible method, . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ menu[0-5] / 4.6 Indexing only parts of a page by <div id='abc'> If enabled in Admin settings, the values as defined in the list-file /include/common/divs_use.txt will be used to index only the content between <div . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ table[0-5] / 4.7 Ignore HTML elements defined by <tagname> . . </tagname> This option is foreseen to cooperate with the new HTML5 elements like section, nav, aside, hgroup, article, header, footer, etc. HTML elements . . .
. . .
contain a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */nav[0-5]/ Please keep in mind that element names placed in /include/common/elements_not.txt will be processed case-sensitive. 4.8 Index only HTML elements defined by <tagname> . . </tagname> This is the vice versa . . .
. . .
of all the duplicate content URLs: http://www.example.com/product.php?item=swedish-fish&category=gummy-candy http://www.example.com/product.php?item=swedish-fish&trackingid=1234&sessionid=5678 and Sphider-plus will understand that the duplicates all refer to the canonical URL: http://www.example.com/product.php?item=swedish-fish. The . . .
. . .
charset are supported by the ConvertCharset function and will be used to convert text into UTF-8 Unicode: WINDOWS windows-1250 - Central Europe windows-1251 - Cyrillic windows-1252 - Latin I windows-1253 - Greek windows-1254 - Turkish windows-1255 - Hebrew windows-1256 - Arabic windows-1257 - Baltic windows-1258 - Viet Nam cp874 - Thai - this . . .
. . .
- Baltic windows-1258 - Viet Nam cp874 - Thai - this file is also for DOS DOS cp437 - Latin US cp737 - Greek cp775 - BaltRim cp850 - Latin1 cp852 - Latin2 cp855 - Cyrylic cp857 - Turkish cp860 - Portuguese cp861 - Iceland cp862 - Hebrew cp863 - Canada cp864 - Arabic cp865 - Nordic cp866 - Cyrylic Russian (this is the one, used in IE . . .
. . .
IE Cyrillic (DOS) ) cp869 - Greek2 MAC (Apple) x-mac-cyrillic x-mac-greek x-mac-icelandic x-mac-ce x-mac-roman ISO iso-8859-1 iso-8859-2 iso-8859-3 iso-8859-4 iso-8859-5 iso-8859-6 iso-8859-7 iso-8859-8 iso-8859-9 iso-8859-10 iso-8859-11 iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 . . .
. . .
iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 cp500 cp856 cp875 cp1006 cp1026 koi8-r (Cyrillic) koi8-u (Cyrillic Ukrainian) nextstep us-ascii us-ascii-quotes DSP implementation for NeXT stdenc symbol zdingbat And specially for old Polish programs: mazovia This list is to be read . . .
. . .
Access denied; you need the RELOAD privilege. . . Top 10. Enable real-time output of logging data Up to version 1.5 of Sphider-plus, during index / re-index there was no printout available because: - Several servers, especially on Win32, buffer the output from the script until it terminates before transmitting the results to the browser. - . . .
. . .
is seen. - Some versions of Microsoft Internet Explorer only start to display the page after they have received 256 bytes of output. As progress was not presented during index / re- index procedure, waiting for results became a pain in the neck. Selectable in Admin setting together with the update interval (1 - 10 seconds), AJAX technology was . . .
. . .
characters in front of words Warning: This option should be used with special care and not be activated for non ISO-8859 charsets. Some special characters as part of the word ending might be erased by accidental. Top 13. Media search for images, audio streams and videos 13.1 Media indexing Index of media files is enabled by separated Admin . . .
. . .
and 'Found at' - Total clicks - Last clicked - Query input 'Indexed Image Thumbnails' presenting: - Thumbnail 150 x 100 pixel - Image details like title, filename size of original image, link- and thumb-id - Option to delete the thumbnail In order to open the media files all tables contain active links. Media results are also stored in . . .
. . .
Settings' menu of the admin backend: First option: For results of XML product feeds, present the text extract of 350 characters as part of product attributes. If this option is not activated, all products of the XML product feed will be presented in search results, and the hits of the query string will be highlighted in all involved products. . . .
. . .
content of the database tables (those with the same table prefix) will be destroyed by the restore procedure. 16.5 Copy & Move This section of the Database Management will present only those databases, which are correctly configured, assigned and do have a set of installed tables as described in chapter Definition and configuration. This . . .
. . .
results. Never the less to find these x results, the complete database had to been browsed. Starting with version 2.5, an additional clean option is offered as part of the Admin backend. Main advantage of this option is a significant reduction of the search time for any query, because the content of the db could be limited to offer only x . . .
. . .
<link>http://www.abc.de/images/warp.gif</link> <title>warp.gif</title> <x_size>635</x_size> <y_size>98</y_size> </media_result> <media_result> <num>2</num> <type>audio</type> <url>http://www.abc.de/index.php</url> . . .
. . .
the following content: {"query":"colour","ip":"::1","host_name":"guard_007-hoster","query_time":"2016-03-18 105623 AM","consumed":0.016,"total_results":2,"num_of_results":2,"from":1,"to":2,"text_results":[{"num":1,"weight":"100.0","url":"http:\/\/www.english.le-piaggie.info\/Info_eng.pdf","title":" Info_eng","fulltxt":" . . . . . .
. . .
placed on the ground floor, today's big kitchen with the heating fireplace for open and close mode of . . . ","page_size":"5,661.6kb","domain_name":"www.english.le-piaggie.info"},{"num":2,"weight":"50.0","url":"http:\/\/www.english.le-piaggie.info\/html\/description.html","title":" Description","fulltxt":" . . . cable. Additional active . . .
. . .
D.O.C.G. as far as to the Pratomagno mountains 160 olive-trees Detached house, about 800 m to reach the . . . ","page_size":"26.4 kb","domain_name":"www.english.le-piaggie.info"}]} Top . . .


5a2 All required information. Introduction Release and Legal Info Installation Documentation Change Log [ Documentation Summary ] Preamble: The info presented here is valid only for the latest release of Sphider-plus. At present version 4.2024a published February 12, 2024 is the actual release. 1. Settings, customizing and statistics 2. Indexing . . .
. . .
2. Indexing 2.1 Various options 2.2 Allow other hosts in same domain 2.3 Word stemming 2.4 Periodical Re-indexing 25 Preferred indexing 2.6 Multithreaded indexing 2.7 Create thumbnails during index procedure 2.8 Prevent indexing of known malware and pishing pages 2.9 Follow and create sitemap files 2.10 Use private sitemap instead of global . . .
. . .
Sitemap file 3. Using the indexer from command line 3.1 All options 3.2 Multithreaded indexing 4. Keeping pages, words and files from being indexed 4.1 robots.txt 4.2 Must include / must not include string list 4.3 Ignoring links 4.4 Ignoring parts of a page by <! sphider_noindex > 4.5 Ignoring parts of a page by <div id='abc'> . . .
. . .
charset' 6. Search modes 6.1 Search with wildcards * 6.2 Strict search ! 6.3 Tolerant search 6.4 Link search 6.5 Media search 6.6 Search only in one domain 6.7 Search in categories 6.8 Greek language support 6.9 Block queries 7. Chronological order for result listing 7.1 Text result listing 7.2 Media result listing 8. PDF converter 9. Clean . . .
. . .
of logging data 11. Error messages and Debug mode 12. Delete secondary characters 13. Media search for images, audio streams and videos 13.1 Media indexing 13.2 Not supported media content 13.3 Search for media content 13.4 Statistics for media content 14. Feed support 14.1 XML product feeds 14.2 RDF, RSD, RSS and Atom feeds 15. Result . . .
. . .
Overview 16.2 Definition and configuration 16.3 Activate / disable database 16.4 Backup & Restore of databases 16.5 Copy & and Move 16.6 Enhancing functionality of multiple database support 17. Search in categories 17.1 Hierachical structure 17.2 Parallel structure 18. User suggested sites 19. Vulnerability protection 19.1 Prevent queries . . .
. . .
templates 22.2 Embed the search engine into existing HTML code 22.3 The different style sheet files 23. JSON, XML and RSS result output [ Documentation ] 1. Settings, customizing and statistics If you want to change settings, behavior and design of Sphider-plus, you can do so by means of the Admin interface. There is a wide range of . . .
. . .
interface. There is a wide range of settings foreseen for Sphider-plus. Separated into different submenus like: Sites: - Add Site - Index only the new - Re-index all - Re-index only preferred URLs - Erase Re-index (available also for individual URLs) - Import/export URL list - Approve sites - Banned domains Categories: - Add, edit, delete - . . .
. . .
URL list - Approve sites - Banned domains Categories: - Add, edit, delete - Create new subcategory under Index: - Basic indexing options - Advanced options Clean: - Clean keywords not associated with any link - Clean links not associated with any site - Clean Category table not associated with any site - Clean Media links - Clear Temp table . . .
. . .
- Clear Search log - Clear 'Most Popular Page Links' log - Clear 'Most Popular Media Links' log - Clear Spider log, separate and bulk delete - Clear Thumbnail images, separate and bulk delete - Clear Text cache - Clear Media cache - Clear IDS log file - Clear flood attempts log file - Clear all entries in addurl or banned table - Truncate all . . .
. . .
flood attempts log file - Clear all entries in addurl or banned table - Truncate all tables in database Settings: - General Settings - Index Log Settings - Spider Settings - Search Settings - Order of Result listing - Suggest Options - Page Indexing Weights Database: - Configure up to 5 databases with unlimited number of table sets - Activate . . .
. . .
Database: - Configure up to 5 databases with unlimited number of table sets - Activate separately for 'Admin', 'Search' user and 'Suggest URL user' - Backup / Restore - Copy / Move - Optimize Templates: In order to enable customer's integration of Sphider-plus into existing sites, HTML templates are prepared for Search form Text result . . .
. . .
Search form Text result listing Media result listing Most popular queries etc. Three different designs are offered, which may be selected in submenu 'Settings'. If the layout does not fit the design of your site (which is normal), you may create new designs and modify the appropriate file /templates/My_template/adminstyle.css . . .
. . .
the appropriate file /templates/My_template/adminstyle.css /templates/My_template/userstyle.css Statistics output: - Top keywords (Top 50 with hit counter). - All indexed thumbnails w 3e80 ith ID3 and EXIF info. - Larges pages offering link URL and file size. - Most Popular Searches for text links offering: Link addr., total clicks, last . . .
. . .
Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Searches for media links offering: Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Links (click counter). - Search log offering: Query, Results, Queried at, Time taken, User IP, Country, Host name (Latest100) - Index log offering: . . .
. . .
Country, Host name (Latest100) - Index log offering: File-name, index date and delete option - sitemap log offering: sitemap.xml output sitemap list offering file/page suffixes - IDS log offering: IP, host, query, impact, involved tags, date and time of intrusion. - Flood attempts log offering: IP, query, date and time of flood attempt. - Auto . . .
. . .
attempts log offering: IP, query, date and time of flood attempt. - Auto Re-index log file - Server info offering: Server software, environment, MySQL, PDF-converter, image functions, php.ini file PHP integration, PHP security info. Each item holding lists of details. All text links, media links and thumbnails are active linked. As stated in . . .
. . .
individual Re-indexing, the periodical Re-indexer could be started and aborted in the "Options" menu of each site. 25 Preferred Re-indexing Each new URL added to the Admin backend, could be supplied with a priority level. This level will be used by the option 'Re-index only preferred sites'. Level 1 will be interpreted as most important, while . . .
. . .
less all index results will be stored in log files in sub folder /admin/log/ The names of the log files look like: db2_100524-21.47.56_1.html (log file of first thread) db2_100524-21.48.12_2.html (log file of second thread) and is build by the following items: db2 - Number of database. 100524 - Date (May 24, 2010) 21.47.56 - Time when this thread . . .
. . .
spider.php -new1 php spider.php -new2 etc. The IDs will be added to the name of the corresponding log files like: db2_100524-21.47.56_ID1.html (log file of first thread) db2_100524-21.48.12_ID2.html (log file of second thread) IDs could be defined by personal requirements, but the limitations for file names with respect to the OS should be taken . . .
. . .
but will not erase the content of all the other tables. So the check whether the content of a page has changed (MD5sum) is still available for a fast re-index procedure. Once prepared, multithreaded re-indexing could be invoked by starting several threads and adding individual IDs to the option parameter like: php spider.php -erased1 php . . .
. . .
4.4 Ignoring parts of a page Sphider-plus includes an option to exclude parts of pages from being indexed. Thi 775e s can for example be used to prevent search result flooding when certain keywords appear on certain part in most pages (like a header, footer or a menu). Any part of a page between <! sphider_noindex > and <! . . .
. . .
<! sphider_noindex > and <! /sphider_noindex > tags is not indexed, however links in it are followed. 4.5 Ignoring parts of a page by <div id='abc'> Ignoring parts of a page by the <! sphider_noindex > tags requires direct access to the page, because the tags need to be added (edited) to the page. A more flexible method, . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ menu[0-5] / 4.6 Indexing only parts of a page by <div id='abc'> If enabled in Admin settings, the values as defined in the list-file /include/common/divs_use.txt will be used to index only the content between <div . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ table[0-5] / 4.7 Ignore HTML elements defined by <tagname> . . </tagname> This option is foreseen to cooperate with the new HTML5 elements like section, nav, aside, hgroup, article, header, footer, etc. HTML elements . . .
. . .
contain a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */nav[0-5]/ Please keep in mind that element names placed in /include/common/elements_not.txt will be processed case-sensitive. 4.8 Index only HTML elements defined by <tagname> . . </tagname> This is the vice versa . . .
. . .
of all the duplicate content URLs: http://www.example.com/product.php?item=swedish-fish&category=gummy-candy http://www.example.com/product.php?item=swedish-fish&trackingid=1234&sessionid=5678 and Sphider-plus will understand that the duplicates all refer to the canonical URL: http://www.example.com/product.php?item=swedish-fish. The . . .
. . .
charset are supported by the ConvertCharset function and will be used to convert text into UTF-8 Unicode: WINDOWS windows-1250 - Central Europe windows-1251 - Cyrillic windows-1252 - Latin I windows-1253 - Greek windows-1254 - Turkish windows-1255 - Hebrew windows-1256 - Arabic windows-1257 - Baltic windows-1258 - Viet Nam cp874 - Thai - this . . .
. . .
- Baltic windows-1258 - Viet Nam cp874 - Thai - this file is also for DOS DOS cp437 - Latin US cp737 - Greek cp775 - BaltRim cp850 - Latin1 cp852 - Latin2 cp855 - Cyrylic cp857 - Turkish cp860 - Portuguese cp861 - Iceland cp862 - Hebrew cp863 - Canada cp864 - Arabic cp865 - Nordic cp866 - Cyrylic Russian (this is the one, used in IE . . .
. . .
IE Cyrillic (DOS) ) cp869 - Greek2 MAC (Apple) x-mac-cyrillic x-mac-greek x-mac-icelandic x-mac-ce x-mac-roman ISO iso-8859-1 iso-8859-2 iso-8859-3 iso-8859-4 iso-8859-5 iso-8859-6 iso-8859-7 iso-8859-8 iso-8859-9 iso-8859-10 iso-8859-11 iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 . . .
. . .
iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 cp500 cp856 cp875 cp1006 cp1026 koi8-r (Cyrillic) koi8-u (Cyrillic Ukrainian) nextstep us-ascii us-ascii-quotes DSP implementation for NeXT stdenc symbol zdingbat And specially for old Polish programs: mazovia This list is to be read . . .
. . .
Access denied; you need the RELOAD privilege. . . Top 10. Enable real-time output of logging data Up to version 1.5 of Sphider-plus, during index / re-index there was no printout available because: - Several servers, especially on Win32, buffer the output from the script until it terminates before transmitting the results to the browser. - . . .
. . .
is seen. - Some versions of Microsoft Internet Explorer only start to display the page after they have received 256 bytes of output. As progress was not presented during index / re- index procedure, waiting for results became a pain in the neck. Selectable in Admin setting together with the update interval (1 - 10 seconds), AJAX technology was . . .
. . .
characters in front of words Warning: This option should be used with special care and not be activated for non ISO-8859 charsets. Some special characters as part of the word ending might be erased by accidental. Top 13. Media search for images, audio streams and videos 13.1 Media indexing Index of media files is enabled by separated Admin . . .
. . .
and 'Found at' - Total clicks - Last clicked - Query input 'Indexed Image Thumbnails' presenting: - Thumbnail 150 x 100 pixel - Image details like title, filename size of original image, link- and thumb-id - Option to delete the thumbnail In order to open the media files all tables contain active links. Media results are also stored in . . .
. . .
Settings' menu of the admin backend: First option: For results of XML product feeds, present the text extract of 350 characters as part of product attributes. If this option is not activated, all products of the XML product feed will be presented in search results, and the hits of the query string will be highlighted in all involved products. . . .
. . .
content of the database tables (those with the same table prefix) will be destroyed by the restore procedure. 16.5 Copy & Move This section of the Database Management will present only those databases, which are correctly configured, assigned and do have a set of installed tables as described in chapter Definition and configuration. This . . .
. . .
results. Never the less to find these x results, the complete database had to been browsed. Starting with version 25, an additional clean option is offered as part of the Admin backend. Main advantage of this option is a significant reduction of the search time for any query, because the content of the db could be limited to offer only x . . .
. . .
<link>http://www.abc.de/images/warp.gif</link> <title>warp.gif</title> <x_size>635</x_size> <y_size>98</y_size> </media_result> <media_result> <num>2</num> <type>audio</type> <url>http://www.abc.de/index.php</url> . . .
. . .
the following content: {"query":"colour","ip":"::1","host_name":"guard_007-hoster","query_time":"2016-03-18 10:56:23 AM","consumed":0.016,"total_results":2,"num_of_results":2,"from":1,"to":2,"text_results":[{"num":1,"weight":"100.0","url":"http:\/\/www.english.le-piaggie.info\/Info_eng.pdf","title":" Info_eng","fulltxt":" . . . . . .
. . .
placed on the ground floor, today's big kitchen with the heating fireplace for open and close mode of . . . ","page_size":"5,661.6kb","domain_name":"www.english.le-piaggie.info"},{"num":2,"weight":"50.0","url":"http:\/\/www.english.le-piaggie.info\/html\/description.html","title":" Description","fulltxt":" . . . cable. Additional active . . .
. . .
D.O.C.G. as far as to the Pratomagno mountains 160 olive-trees Detached house, about 800 m to reach the . . . ","page_size":"26.4 kb","domain_name":"www.english.le-piaggie.info"}]} Top . . .


5a2 All required information. Introduction Release and Legal Info Installation Documentation Change Log [ Documentation Summary ] Preamble: The info presented here is valid only for the latest release of Sphider-plus. At present version 4.2024a published February 12, 2024 is the actual release. 1. Settings, customizing and statistics 2. Indexing . . .
. . .
2. Indexing 2.1 Various options 2.2 Allow other hosts in same domain 2.3 Word stemming 2.4 Periodical Re-indexing 2.5 Preferred indexing 2.6 Multithreaded indexing 2.7 Create thumbnails during index procedure 2.8 Prevent indexing of known malware and pishing pages 2.9 Follow and create sitemap files 2.10 Use private sitemap instead of global . . .
. . .
Sitemap file 3. Using the indexer from command line 3.1 All options 3.2 Multithreaded indexing 4. Keeping pages, words and files from being indexed 4.1 robots.txt 4.2 Must include / must not include string list 4.3 Ignoring links 4.4 Ignoring parts of a page by <! sphider_noindex > 45 Ignoring parts of a page by <div id='abc'> . . .
. . .
charset' 6. Search modes 6.1 Search with wildcards * 6.2 Strict search ! 6.3 Tolerant search 6.4 Link search 6.5 Media search 6.6 Search only in one domain 6.7 Search in categories 6.8 Greek language support 6.9 Block queries 7. Chronological order for result listing 7.1 Text result listing 7.2 Media result listing 8. PDF converter 9. Clean . . .
. . .
of logging data 11. Error messages and Debug mode 12. Delete secondary characters 13. Media search for images, audio streams and videos 13.1 Media indexing 13.2 Not supported media content 13.3 Search for media content 13.4 Statistics for media content 14. Feed support 14.1 XML product feeds 14.2 RDF, RSD, RSS and Atom feeds 15. Result . . .
. . .
Overview 16.2 Definition and configuration 16.3 Activate / disable database 16.4 Backup & Restore of databases 16.5 Copy & and Move 16.6 Enhancing functionality of multiple database support 17. Search in categories 17.1 Hierachical structure 17.2 Parallel structure 18. User suggested sites 19. Vulnerability protection 19.1 Prevent queries . . .
. . .
templates 22.2 Embed the search engine into existing HTML code 22.3 The different style sheet files 23. JSON, XML and RSS result output [ Documentation ] 1. Settings, customizing and statistics If you want to change settings, behavior and design of Sphider-plus, you can do so by means of the Admin interface. There is a wide range of . . .
. . .
interface. There is a wide range of settings foreseen for Sphider-plus. Separated into different submenus like: Sites: - Add Site - Index only the new - Re-index all - Re-index only preferred URLs - Erase Re-index (available also for individual URLs) - Import/export URL list - Approve sites - Banned domains Categories: - Add, edit, delete - . . .
. . .
URL list - Approve sites - Banned domains Categories: - Add, edit, delete - Create new subcategory under Index: - Basic indexing options - Advanced options Clean: - Clean keywords not associated with any link - Clean links not associated with any site - Clean Category table not associated with any site - Clean Media links - Clear Temp table . . .
. . .
- Clear Search log - Clear 'Most Popular Page Links' log - Clear 'Most Popular Media Links' log - Clear Spider log, separate and bulk delete - Clear Thumbnail images, separate and bulk delete - Clear Text cache - Clear Media cache - Clear IDS log file - Clear flood attempts log file - Clear all entries in addurl or banned table - Truncate all . . .
. . .
flood attempts log file - Clear all entries in addurl or banned table - Truncate all tables in database Settings: - General Settings - Index Log Settings - Spider Settings - Search Settings - Order of Result listing - Suggest Options - Page Indexing Weights Database: - Configure up to 5 databases with unlimited number of table sets - Activate . . .
. . .
Database: - Configure up to 5 databases with unlimited number of table sets - Activate separately for 'Admin', 'Search' user and 'Suggest URL user' - Backup / Restore - Copy / Move - Optimize Templates: In order to enable customer's integration of Sphider-plus into existing sites, HTML templates are prepared for Search form Text result . . .
. . .
Search form Text result listing Media result listing Most popular queries etc. Three different designs are offered, which may be selected in submenu 'Settings'. If the layout does not fit the design of your site (which is normal), you may create new designs and modify the appropriate file /templates/My_template/adminstyle.css . . .
. . .
the appropriate file /templates/My_template/adminstyle.css /templates/My_template/userstyle.css Statistics output: - Top keywords (Top 50 with hit counter). - All indexed thumbnails w 3e80 ith ID3 and EXIF info. - Larges pages offering link URL and file size. - Most Popular Searches for text links offering: Link addr., total clicks, last . . .
. . .
Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Searches for media links offering: Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Links (click counter). - Search log offering: Query, Results, Queried at, Time taken, User IP, Country, Host name (Latest100) - Index log offering: . . .
. . .
Country, Host name (Latest100) - Index log offering: File-name, index date and delete option - sitemap log offering: sitemap.xml output sitemap list offering file/page suffixes - IDS log offering: IP, host, query, impact, involved tags, date and time of intrusion. - Flood attempts log offering: IP, query, date and time of flood attempt. - Auto . . .
. . .
attempts log offering: IP, query, date and time of flood attempt. - Auto Re-index log file - Server info offering: Server software, environment, MySQL, PDF-converter, image functions, php.ini file PHP integration, PHP security info. Each item holding lists of details. All text links, media links and thumbnails are active linked. As stated in . . .
. . .
individual Re-indexing, the periodical Re-indexer could be started and aborted in the "Options" menu of each site. 2.5 Preferred Re-indexing Each new URL added to the Admin backend, could be supplied with a priority level. This level will be used by the option 'Re-index only preferred sites'. Level 1 will be interpreted as most important, while . . .
. . .
less all index results will be stored in log files in sub folder /admin/log/ The names of the log files look like: db2_100524-21.47.56_1.html (log file of first thread) db2_100524-21.48.12_2.html (log file of second thread) and is build by the following items: db2 - Number of database. 100524 - Date (May 24, 2010) 21.47.56 - Time when this thread . . .
. . .
spider.php -new1 php spider.php -new2 etc. The IDs will be added to the name of the corresponding log files like: db2_100524-21.47.56_ID1.html (log file of first thread) db2_100524-21.48.12_ID2.html (log file of second thread) IDs could be defined by personal requirements, but the limitations for file names with respect to the OS should be taken . . .
. . .
but will not erase the content of all the other tables. So the check whether the content of a page has changed (MD5sum) is still available for a fast re-index procedure. Once prepared, multithreaded re-indexing could be invoked by starting several threads and adding individual IDs to the option parameter like: php spider.php -erased1 php . . .
. . .
4.4 Ignoring parts of a page Sphider-plus includes an option to exclude parts of pages from being indexed. Thi 775e s can for example be used to prevent search result flooding when certain keywords appear on certain part in most pages (like a header, footer or a menu). Any part of a page between <! sphider_noindex > and <! . . .
. . .
<! sphider_noindex > and <! /sphider_noindex > tags is not indexed, however links in it are followed. 45 Ignoring parts of a page by <div id='abc'> Ignoring parts of a page by the <! sphider_noindex > tags requires direct access to the page, because the tags need to be added (edited) to the page. A more flexible method, . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ menu[0-5] / 4.6 Indexing only parts of a page by <div id='abc'> If enabled in Admin settings, the values as defined in the list-file /include/common/divs_use.txt will be used to index only the content between <div . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ table[0-5] / 4.7 Ignore HTML elements defined by <tagname> . . </tagname> This option is foreseen to cooperate with the new HTML5 elements like section, nav, aside, hgroup, article, header, footer, etc. HTML elements . . .
. . .
contain a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */nav[0-5]/ Please keep in mind that element names placed in /include/common/elements_not.txt will be processed case-sensitive. 4.8 Index only HTML elements defined by <tagname> . . </tagname> This is the vice versa . . .
. . .
of all the duplicate content URLs: http://www.example.com/product.php?item=swedish-fish&category=gummy-candy http://www.example.com/product.php?item=swedish-fish&trackingid=1234&sessionid=5678 and Sphider-plus will understand that the duplicates all refer to the canonical URL: http://www.example.com/product.php?item=swedish-fish. The . . .
. . .
charset are supported by the ConvertCharset function and will be used to convert text into UTF-8 Unicode: WINDOWS windows-1250 - Central Europe windows-1251 - Cyrillic windows-1252 - Latin I windows-1253 - Greek windows-1254 - Turkish windows-1255 - Hebrew windows-1256 - Arabic windows-1257 - Baltic windows-1258 - Viet Nam cp874 - Thai - this . . .
. . .
- Baltic windows-1258 - Viet Nam cp874 - Thai - this file is also for DOS DOS cp437 - Latin US cp737 - Greek cp775 - BaltRim cp850 - Latin1 cp852 - Latin2 cp855 - Cyrylic cp857 - Turkish cp860 - Portuguese cp861 - Iceland cp862 - Hebrew cp863 - Canada cp864 - Arabic cp865 - Nordic cp866 - Cyrylic Russian (this is the one, used in IE . . .
. . .
IE Cyrillic (DOS) ) cp869 - Greek2 MAC (Apple) x-mac-cyrillic x-mac-greek x-mac-icelandic x-mac-ce x-mac-roman ISO iso-8859-1 iso-8859-2 iso-8859-3 iso-8859-4 iso-8859-5 iso-8859-6 iso-8859-7 iso-8859-8 iso-8859-9 iso-8859-10 iso-8859-11 iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 . . .
. . .
iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 cp500 cp856 cp875 cp1006 cp1026 koi8-r (Cyrillic) koi8-u (Cyrillic Ukrainian) nextstep us-ascii us-ascii-quotes DSP implementation for NeXT stdenc symbol zdingbat And specially for old Polish programs: mazovia This list is to be read . . .
. . .
Access denied; you need the RELOAD privilege. . . Top 10. Enable real-time output of logging data Up to version 1.5 of Sphider-plus, during index / re-index there was no printout available because: - Several servers, especially on Win32, buffer the output from the script until it terminates before transmitting the results to the browser. - . . .
. . .
is seen. - Some versions of Microsoft Internet Explorer only start to display the page after they have received 256 bytes of output. As progress was not presented during index / re- index procedure, waiting for results became a pain in the neck. Selectable in Admin setting together with the update interval (1 - 10 seconds), AJAX technology was . . .
. . .
characters in front of words Warning: This option should be used with special care and not be activated for non ISO-8859 charsets. Some special characters as part of the word ending might be erased by accidental. Top 13. Media search for images, audio streams and videos 13.1 Media indexing Index of media files is enabled by separated Admin . . .
. . .
and 'Found at' - Total clicks - Last clicked - Query input 'Indexed Image Thumbnails' presenting: - Thumbnail 150 x 100 pixel - Image details like title, filename size of original image, link- and thumb-id - Option to delete the thumbnail In order to open the media files all tables contain active links. Media results are also stored in . . .
. . .
Settings' menu of the admin backend: First option: For results of XML product feeds, present the text extract of 350 characters as part of product attributes. If this option is not activated, all products of the XML product feed will be presented in search results, and the hits of the query string will be highlighted in all involved products. . . .
. . .
content of the database tables (those with the same table prefix) will be destroyed by the restore procedure. 16.5 Copy & Move This section of the Database Management will present only those databases, which are correctly configured, assigned and do have a set of installed tables as described in chapter Definition and configuration. This . . .
. . .
results. Never the less to find these x results, the complete database had to been browsed. Starting with version 2.5, an additional clean option is offered as part of the Admin backend. Main advantage of this option is a significant reduction of the search time for any query, because the content of the db could be limited to offer only x . . .
. . .
<link>http://www.abc.de/images/warp.gif</link> <title>warp.gif</title> <x_size>635</x_size> <y_size>98</y_size> </media_result> <media_result> <num>2</num> <type>audio</type> <url>http://www.abc.de/index.php</url> . . .
. . .
the following content: {"query":"colour","ip":"::1","host_name":"guard_007-hoster","query_time":"2016-03-18 10:56:23 AM","consumed":0.016,"total_results":2,"num_of_results":2,"from":1,"to":2,"text_results":[{"num":1,"weight":"100.0","url":"http:\/\/www.english.le-piaggie.info\/Info_eng.pdf","title":" Info_eng","fulltxt":" . . . . . .
. . .
placed on the ground floor, today's big kitchen with the heating fireplace for open and close mode of . . . ","page_size":"5,661.6kb","domain_name":"www.english.le-piaggie.info"},{"num":2,"weight":"50.0","url":"http:\/\/www.english.le-piaggie.info\/html\/description.html","title":" Description","fulltxt":" . . . cable. Additional active . . .
. . .
D.O.C.G. as far as to the Pratomagno mountains 160 olive-trees Detached house, about 800 m to reach the . . . ","page_size":"26.4 kb","domain_name":"www.english.le-piaggie.info"}]} Top . . .


5a2 All required information. Introduction Release and Legal Info Installation Documentation Change Log [ Documentation Summary ] Preamble: The info presented here is valid only for the latest release of Sphider-plus. At present version 4.2024a published February 12, 2024 is the actual release. 1. Settings, customizing and statistics 2. Indexing . . .
. . .
2. Indexing 2.1 Various options 2.2 Allow other hosts in same domain 2.3 Word stemming 2.4 Periodical Re-indexing 2.5 Preferred indexing 2.6 Multithreaded indexing 2.7 Create thumbnails during index procedure 2.8 Prevent indexing of known malware and pishing pages 2.9 Follow and create sitemap files 2.10 Use private sitemap instead of global . . .
. . .
Sitemap file 3. Using the indexer from command line 3.1 All options 3.2 Multithreaded indexing 4. Keeping pages, words and files from being indexed 4.1 robots.txt 4.2 Must include / must not include string list 4.3 Ignoring links 4.4 Ignoring parts of a page by <! sphider_noindex > 4.5 Ignoring parts of a page by <div id='abc'> . . .
. . .
charset' 6. Search modes 6.1 Search with wildcards * 6.2 Strict search ! 6.3 Tolerant search 6.4 Link search 65 Media search 6.6 Search only in one domain 6.7 Search in categories 6.8 Greek language support 6.9 Block queries 7. Chronological order for result listing 7.1 Text result listing 7.2 Media result listing 8. PDF converter 9. Clean . . .
. . .
of logging data 11. Error messages and Debug mode 12. Delete secondary characters 13. Media search for images, audio streams and videos 13.1 Media indexing 13.2 Not supported media content 13.3 Search for media content 13.4 Statistics for media content 14. Feed support 14.1 XML product feeds 14.2 RDF, RSD, RSS and Atom feeds 15. Result . . .
. . .
Overview 16.2 Definition and configuration 16.3 Activate / disable database 16.4 Backup & Restore of databases 165 Copy & and Move 16.6 Enhancing functionality of multiple database support 17. Search in categories 17.1 Hierachical structure 17.2 Parallel structure 18. User suggested sites 19. Vulnerability protection 19.1 Prevent queries . . .
. . .
templates 22.2 Embed the search engine into existing HTML code 22.3 The different style sheet files 23. JSON, XML and RSS result output [ Documentation ] 1. Settings, customizing and statistics If you want to change settings, behavior and design of Sphider-plus, you can do so by means of the Admin interface. There is a wide range of . . .
. . .
interface. There is a wide range of settings foreseen for Sphider-plus. Separated into different submenus like: Sites: - Add Site - Index only the new - Re-index all - Re-index only preferred URLs - Erase Re-index (available also for individual URLs) - Import/export URL list - Approve sites - Banned domains Categories: - Add, edit, delete - . . .
. . .
URL list - Approve sites - Banned domains Categories: - Add, edit, delete - Create new subcategory under Index: - Basic indexing options - Advanced options Clean: - Clean keywords not associated with any link - Clean links not associated with any site - Clean Category table not associated with any site - Clean Media links - Clear Temp table . . .
. . .
- Clear Search log - Clear 'Most Popular Page Links' log - Clear 'Most Popular Media Links' log - Clear Spider log, separate and bulk delete - Clear Thumbnail images, separate and bulk delete - Clear Text cache - Clear Media cache - Clear IDS log file - Clear flood attempts log file - Clear all entries in addurl or banned table - Truncate all . . .
. . .
flood attempts log file - Clear all entries in addurl or banned table - Truncate all tables in database Settings: - General Settings - Index Log Settings - Spider Settings - Search Settings - Order of Result listing - Suggest Options - Page Indexing Weights Database: - Configure up to 5 databases with unlimited number of table sets - Activate . . .
. . .
Database: - Configure up to 5 databases with unlimited number of table sets - Activate separately for 'Admin', 'Search' user and 'Suggest URL user' - Backup / Restore - Copy / Move - Optimize Templates: In order to enable customer's integration of Sphider-plus into existing sites, HTML templates are prepared for Search form Text result . . .
. . .
Search form Text result listing Media result listing Most popular queries etc. Three different designs are offered, which may be selected in submenu 'Settings'. If the layout does not fit the design of your site (which is normal), you may create new designs and modify the appropriate file /templates/My_template/adminstyle.css . . .
. . .
the appropriate file /templates/My_template/adminstyle.css /templates/My_template/userstyle.css Statistics output: - Top keywords (Top 50 with hit counter). - All indexed thumbnails w 3e80 ith ID3 and EXIF info. - Larges pages offering link URL and file size. - Most Popular Searches for text links offering: Link addr., total clicks, last . . .
. . .
Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Searches for media links offering: Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Links (click counter). - Search log offering: Query, Results, Queried at, Time taken, User IP, Country, Host name (Latest100) - Index log offering: . . .
. . .
Country, Host name (Latest100) - Index log offering: File-name, index date and delete option - sitemap log offering: sitemap.xml output sitemap list offering file/page suffixes - IDS log offering: IP, host, query, impact, involved tags, date and time of intrusion. - Flood attempts log offering: IP, query, date and time of flood attempt. - Auto . . .
. . .
attempts log offering: IP, query, date and time of flood attempt. - Auto Re-index log file - Server info offering: Server software, environment, MySQL, PDF-converter, image functions, php.ini file PHP integration, PHP security info. Each item holding lists of details. All text links, media links and thumbnails are active linked. As stated in . . .
. . .
individual Re-indexing, the periodical Re-indexer could be started and aborted in the "Options" menu of each site. 2.5 Preferred Re-indexing Each new URL added to the Admin backend, could be supplied with a priority level. This level will be used by the option 'Re-index only preferred sites'. Level 1 will be interpreted as most important, while . . .
. . .
less all index results will be stored in log files in sub folder /admin/log/ The names of the log files look like: db2_100524-21.47.56_1.html (log file of first thread) db2_100524-21.48.12_2.html (log file of second thread) and is build by the following items: db2 - Number of database. 100524 - Date (May 24, 2010) 21.47.56 - Time when this thread . . .
. . .
spider.php -new1 php spider.php -new2 etc. The IDs will be added to the name of the corresponding log files like: db2_100524-21.47.56_ID1.html (log file of first thread) db2_100524-21.48.12_ID2.html (log file of second thread) IDs could be defined by personal requirements, but the limitations for file names with respect to the OS should be taken . . .
. . .
but will not erase the content of all the other tables. So the check whether the content of a page has changed (MD5sum) is still available for a fast re-index procedure. Once prepared, multithreaded re-indexing could be invoked by starting several threads and adding individual IDs to the option parameter like: php spider.php -erased1 php . . .
. . .
4.4 Ignoring parts of a page Sphider-plus includes an option to exclude parts of pages from being indexed. Thi 775e s can for example be used to prevent search result flooding when certain keywords appear on certain part in most pages (like a header, footer or a menu). Any part of a page between <! sphider_noindex > and <! . . .
. . .
<! sphider_noindex > and <! /sphider_noindex > tags is not indexed, however links in it are followed. 4.5 Ignoring parts of a page by <div id='abc'> Ignoring parts of a page by the <! sphider_noindex > tags requires direct access to the page, because the tags need to be added (edited) to the page. A more flexible method, . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ menu[0-5] / 4.6 Indexing only parts of a page by <div id='abc'> If enabled in Admin settings, the values as defined in the list-file /include/common/divs_use.txt will be used to index only the content between <div . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ table[0-5] / 4.7 Ignore HTML elements defined by <tagname> . . </tagname> This option is foreseen to cooperate with the new HTML5 elements like section, nav, aside, hgroup, article, header, footer, etc. HTML elements . . .
. . .
contain a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */nav[0-5]/ Please keep in mind that element names placed in /include/common/elements_not.txt will be processed case-sensitive. 4.8 Index only HTML elements defined by <tagname> . . </tagname> This is the vice versa . . .
. . .
of all the duplicate content URLs: http://www.example.com/product.php?item=swedish-fish&category=gummy-candy http://www.example.com/product.php?item=swedish-fish&trackingid=1234&sessionid=5678 and Sphider-plus will understand that the duplicates all refer to the canonical URL: http://www.example.com/product.php?item=swedish-fish. The . . .
. . .
charset are supported by the ConvertCharset function and will be used to convert text into UTF-8 Unicode: WINDOWS windows-1250 - Central Europe windows-1251 - Cyrillic windows-1252 - Latin I windows-1253 - Greek windows-1254 - Turkish windows-1255 - Hebrew windows-1256 - Arabic windows-1257 - Baltic windows-1258 - Viet Nam cp874 - Thai - this . . .
. . .
- Baltic windows-1258 - Viet Nam cp874 - Thai - this file is also for DOS DOS cp437 - Latin US cp737 - Greek cp775 - BaltRim cp850 - Latin1 cp852 - Latin2 cp855 - Cyrylic cp857 - Turkish cp860 - Portuguese cp861 - Iceland cp862 - Hebrew cp863 - Canada cp864 - Arabic cp865 - Nordic cp866 - Cyrylic Russian (this is the one, used in IE . . .
. . .
IE Cyrillic (DOS) ) cp869 - Greek2 MAC (Apple) x-mac-cyrillic x-mac-greek x-mac-icelandic x-mac-ce x-mac-roman ISO iso-8859-1 iso-8859-2 iso-8859-3 iso-8859-4 iso-8859-5 iso-8859-6 iso-8859-7 iso-8859-8 iso-8859-9 iso-8859-10 iso-8859-11 iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 . . .
. . .
iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 cp500 cp856 cp875 cp1006 cp1026 koi8-r (Cyrillic) koi8-u (Cyrillic Ukrainian) nextstep us-ascii us-ascii-quotes DSP implementation for NeXT stdenc symbol zdingbat And specially for old Polish programs: mazovia This list is to be read . . .
. . .
Access denied; you need the RELOAD privilege. . . Top 10. Enable real-time output of logging data Up to version 1.5 of Sphider-plus, during index / re-index there was no printout available because: - Several servers, especially on Win32, buffer the output from the script until it terminates before transmitting the results to the browser. - . . .
. . .
is seen. - Some versions of Microsoft Internet Explorer only start to display the page after they have received 256 bytes of output. As progress was not presented during index / re- index procedure, waiting for results became a pain in the neck. Selectable in Admin setting together with the update interval (1 - 10 seconds), AJAX technology was . . .
. . .
characters in front of words Warning: This option should be used with special care and not be activated for non ISO-8859 charsets. Some special characters as part of the word ending might be erased by accidental. Top 13. Media search for images, audio streams and videos 13.1 Media indexing Index of media files is enabled by separated Admin . . .
. . .
and 'Found at' - Total clicks - Last clicked - Query input 'Indexed Image Thumbnails' presenting: - Thumbnail 150 x 100 pixel - Image details like title, filename size of original image, link- and thumb-id - Option to delete the thumbnail In order to open the media files all tables contain active links. Media results are also stored in . . .
. . .
Settings' menu of the admin backend: First option: For results of XML product feeds, present the text extract of 350 characters as part of product attributes. If this option is not activated, all products of the XML product feed will be presented in search results, and the hits of the query string will be highlighted in all involved products. . . .
. . .
content of the database tables (those with the same table prefix) will be destroyed by the restore procedure. 165 Copy & Move This section of the Database Management will present only those databases, which are correctly configured, assigned and do have a set of installed tables as described in chapter Definition and configuration. This . . .
. . .
results. Never the less to find these x results, the complete database had to been browsed. Starting with version 2.5, an additional clean option is offered as part of the Admin backend. Main advantage of this option is a significant reduction of the search time for any query, because the content of the db could be limited to offer only x . . .
. . .
<link>http://www.abc.de/images/warp.gif</link> <title>warp.gif</title> <x_size>635</x_size> <y_size>98</y_size> </media_result> <media_result> <num>2</num> <type>audio</type> <url>http://www.abc.de/index.php</url> . . .
. . .
the following content: {"query":"colour","ip":"::1","host_name":"guard_007-hoster","query_time":"2016-03-18 10:56:23 AM","consumed":0.016,"total_results":2,"num_of_results":2,"from":1,"to":2,"text_results":[{"num":1,"weight":"100.0","url":"http:\/\/www.english.le-piaggie.info\/Info_eng.pdf","title":" Info_eng","fulltxt":" . . . . . .
. . .
placed on the ground floor, today's big kitchen with the heating fireplace for open and close mode of . . . ","page_size":"5,661.6kb","domain_name":"www.english.le-piaggie.info"},{"num":2,"weight":"50.0","url":"http:\/\/www.english.le-piaggie.info\/html\/description.html","title":" Description","fulltxt":" . . . cable. Additional active . . .
. . .
D.O.C.G. as far as to the Pratomagno mountains 160 olive-trees Detached house, about 800 m to reach the . . . ","page_size":"26.4 kb","domain_name":"www.english.le-piaggie.info"}]} Top . . .


5a2 All required information. Introduction Release and Legal Info Installation Documentation Change Log [ Documentation Summary ] Preamble: The info presented here is valid only for the latest release of Sphider-plus. At present version 4.2024a published February 12, 2024 is the actual release. 1. Settings, customizing and statistics 2. Indexing . . .
. . .
2. Indexing 2.1 Various options 2.2 Allow other hosts in same domain 2.3 Word stemming 2.4 Periodical Re-indexing 2.5 Preferred indexing 2.6 Multithreaded indexing 2.7 Create thumbnails during index procedure 2.8 Prevent indexing of known malware and pishing pages 2.9 Follow and create sitemap files 2.10 Use private sitemap instead of global . . .
. . .
Sitemap file 3. Using the indexer from command line 3.1 All options 3.2 Multithreaded indexing 4. Keeping pages, words and files from being indexed 4.1 robots.txt 4.2 Must include / must not include string list 4.3 Ignoring links 4.4 Ignoring parts of a page by <! sphider_noindex > 4.5 Ignoring parts of a page by <div id='abc'> . . .
. . .
charset' 6. Search modes 6.1 Search with wildcards * 6.2 Strict search ! 6.3 Tolerant search 6.4 Link search 6.5 Media search 6.6 Search only in one domain 6.7 Search in categories 6.8 Greek language support 6.9 Block queries 7. Chronological order for result listing 7.1 Text result listing 7.2 Media result listing 8. PDF converter 9. Clean . . .
. . .
of logging data 11. Error messages and Debug mode 12. Delete secondary characters 13. Media search for images, audio streams and videos 13.1 Media indexing 13.2 Not supported media content 13.3 Search for media content 13.4 Statistics for media content 14. Feed support 14.1 XML product feeds 14.2 RDF, RSD, RSS and Atom feeds 15. Result . . .
. . .
Overview 16.2 Definition and configuration 16.3 Activate / disable database 16.4 Backup & Restore of databases 165 Copy & and Move 16.6 Enhancing functionality of multiple database support 17. Search in categories 17.1 Hierachical structure 17.2 Parallel structure 18. User suggested sites 19. Vulnerability protection 19.1 Prevent queries . . .
. . .
templates 22.2 Embed the search engine into existing HTML code 22.3 The different style sheet files 23. JSON, XML and RSS result output [ Documentation ] 1. Settings, customizing and statistics If you want to change settings, behavior and design of Sphider-plus, you can do so by means of the Admin interface. There is a wide range of . . .
. . .
interface. There is a wide range of settings foreseen for Sphider-plus. Separated into different submenus like: Sites: - Add Site - Index only the new - Re-index all - Re-index only preferred URLs - Erase Re-index (available also for individual URLs) - Import/export URL list - Approve sites - Banned domains Categories: - Add, edit, delete - . . .
. . .
URL list - Approve sites - Banned domains Categories: - Add, edit, delete - Create new subcategory under Index: - Basic indexing options - Advanced options Clean: - Clean keywords not associated with any link - Clean links not associated with any site - Clean Category table not associated with any site - Clean Media links - Clear Temp table . . .
. . .
- Clear Search log - Clear 'Most Popular Page Links' log - Clear 'Most Popular Media Links' log - Clear Spider log, separate and bulk delete - Clear Thumbnail images, separate and bulk delete - Clear Text cache - Clear Media cache - Clear IDS log file - Clear flood attempts log file - Clear all entries in addurl or banned table - Truncate all . . .
. . .
flood attempts log file - Clear all entries in addurl or banned table - Truncate all tables in database Settings: - General Settings - Index Log Settings - Spider Settings - Search Settings - Order of Result listing - Suggest Options - Page Indexing Weights Database: - Configure up to 5 databases with unlimited number of table sets - Activate . . .
. . .
Database: - Configure up to 5 databases with unlimited number of table sets - Activate separately for 'Admin', 'Search' user and 'Suggest URL user' - Backup / Restore - Copy / Move - Optimize Templates: In order to enable customer's integration of Sphider-plus into existing sites, HTML templates are prepared for Search form Text result . . .
. . .
Search form Text result listing Media result listing Most popular queries etc. Three different designs are offered, which may be selected in submenu 'Settings'. If the layout does not fit the design of your site (which is normal), you may create new designs and modify the appropriate file /templates/My_template/adminstyle.css . . .
. . .
the appropriate file /templates/My_template/adminstyle.css /templates/My_template/userstyle.css Statistics output: - Top keywords (Top 50 with hit counter). - All indexed thumbnails w 3e80 ith ID3 and EXIF info. - Larges pages offering link URL and file size. - Most Popular Searches for text links offering: Link addr., total clicks, last . . .
. . .
Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Searches for media links offering: Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Links (click counter). - Search log offering: Query, Results, Queried at, Time taken, User IP, Country, Host name (Latest100) - Index log offering: . . .
. . .
Country, Host name (Latest100) - Index log offering: File-name, index date and delete option - sitemap log offering: sitemap.xml output sitemap list offering file/page suffixes - IDS log offering: IP, host, query, impact, involved tags, date and time of intrusion. - Flood attempts log offering: IP, query, date and time of flood attempt. - Auto . . .
. . .
attempts log offering: IP, query, date and time of flood attempt. - Auto Re-index log file - Server info offering: Server software, environment, MySQL, PDF-converter, image functions, php.ini file PHP integration, PHP security info. Each item holding lists of details. All text links, media links and thumbnails are active linked. As stated in . . .
. . .
individual Re-indexing, the periodical Re-indexer could be started and aborted in the "Options" menu of each site. 2.5 Preferred Re-indexing Each new URL added to the Admin backend, could be supplied with a priority level. This level will be used by the option 'Re-index only preferred sites'. Level 1 will be interpreted as most important, while . . .
. . .
less all index results will be stored in log files in sub folder /admin/log/ The names of the log files look like: db2_100524-21.47.56_1.html (log file of first thread) db2_100524-21.48.12_2.html (log file of second thread) and is build by the following items: db2 - Number of database. 100524 - Date (May 24, 2010) 21.47.56 - Time when this thread . . .
. . .
spider.php -new1 php spider.php -new2 etc. The IDs will be added to the name of the corresponding log files like: db2_100524-21.47.56_ID1.html (log file of first thread) db2_100524-21.48.12_ID2.html (log file of second thread) IDs could be defined by personal requirements, but the limitations for file names with respect to the OS should be taken . . .
. . .
but will not erase the content of all the other tables. So the check whether the content of a page has changed (MD5sum) is still available for a fast re-index procedure. Once prepared, multithreaded re-indexing could be invoked by starting several threads and adding individual IDs to the option parameter like: php spider.php -erased1 php . . .
. . .
4.4 Ignoring parts of a page Sphider-plus includes an option to exclude parts of pages from being indexed. Thi 775e s can for example be used to prevent search result flooding when certain keywords appear on certain part in most pages (like a header, footer or a menu). Any part of a page between <! sphider_noindex > and <! . . .
. . .
<! sphider_noindex > and <! /sphider_noindex > tags is not indexed, however links in it are followed. 4.5 Ignoring parts of a page by <div id='abc'> Ignoring parts of a page by the <! sphider_noindex > tags requires direct access to the page, because the tags need to be added (edited) to the page. A more flexible method, . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ menu[0-5] / 4.6 Indexing only parts of a page by <div id='abc'> If enabled in Admin settings, the values as defined in the list-file /include/common/divs_use.txt will be used to index only the content between <div . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ table[0-5] / 4.7 Ignore HTML elements defined by <tagname> . . </tagname> This option is foreseen to cooperate with the new HTML5 elements like section, nav, aside, hgroup, article, header, footer, etc. HTML elements . . .
. . .
contain a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */nav[0-5]/ Please keep in mind that element names placed in /include/common/elements_not.txt will be processed case-sensitive. 4.8 Index only HTML elements defined by <tagname> . . </tagname> This is the vice versa . . .
. . .
of all the duplicate content URLs: http://www.example.com/product.php?item=swedish-fish&category=gummy-candy http://www.example.com/product.php?item=swedish-fish&trackingid=1234&sessionid=5678 and Sphider-plus will understand that the duplicates all refer to the canonical URL: http://www.example.com/product.php?item=swedish-fish. The . . .
. . .
charset are supported by the ConvertCharset function and will be used to convert text into UTF-8 Unicode: WINDOWS windows-1250 - Central Europe windows-1251 - Cyrillic windows-1252 - Latin I windows-1253 - Greek windows-1254 - Turkish windows-1255 - Hebrew windows-1256 - Arabic windows-1257 - Baltic windows-1258 - Viet Nam cp874 - Thai - this . . .
. . .
- Baltic windows-1258 - Viet Nam cp874 - Thai - this file is also for DOS DOS cp437 - Latin US cp737 - Greek cp775 - BaltRim cp850 - Latin1 cp852 - Latin2 cp855 - Cyrylic cp857 - Turkish cp860 - Portuguese cp861 - Iceland cp862 - Hebrew cp863 - Canada cp864 - Arabic cp865 - Nordic cp866 - Cyrylic Russian (this is the one, used in IE . . .
. . .
IE Cyrillic (DOS) ) cp869 - Greek2 MAC (Apple) x-mac-cyrillic x-mac-greek x-mac-icelandic x-mac-ce x-mac-roman ISO iso-8859-1 iso-8859-2 iso-8859-3 iso-8859-4 iso-8859-5 iso-8859-6 iso-8859-7 iso-8859-8 iso-8859-9 iso-8859-10 iso-8859-11 iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 . . .
. . .
iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 cp500 cp856 cp875 cp1006 cp1026 koi8-r (Cyrillic) koi8-u (Cyrillic Ukrainian) nextstep us-ascii us-ascii-quotes DSP implementation for NeXT stdenc symbol zdingbat And specially for old Polish programs: mazovia This list is to be read . . .
. . .
Access denied; you need the RELOAD privilege. . . Top 10. Enable real-time output of logging data Up to version 1.5 of Sphider-plus, during index / re-index there was no printout available because: - Several servers, especially on Win32, buffer the output from the script until it terminates before transmitting the results to the browser. - . . .
. . .
is seen. - Some versions of Microsoft Internet Explorer only start to display the page after they have received 256 bytes of output. As progress was not presented during index / re- index procedure, waiting for results became a pain in the neck. Selectable in Admin setting together with the update interval (1 - 10 seconds), AJAX technology was . . .
. . .
characters in front of words Warning: This option should be used with special care and not be activated for non ISO-8859 charsets. Some special characters as part of the word ending might be erased by accidental. Top 13. Media search for images, audio streams and videos 13.1 Media indexing Index of media files is enabled by separated Admin . . .
. . .
and 'Found at' - Total clicks - Last clicked - Query input 'Indexed Image Thumbnails' presenting: - Thumbnail 150 x 100 pixel - Image details like title, filename size of original image, link- and thumb-id - Option to delete the thumbnail In order to open the media files all tables contain active links. Media results are also stored in . . .
. . .
Settings' menu of the admin backend: First option: For results of XML product feeds, present the text extract of 350 characters as part of product attributes. If this option is not activated, all products of the XML product feed will be presented in search results, and the hits of the query string will be highlighted in all involved products. . . .
. . .
content of the database tables (those with the same table prefix) will be destroyed by the restore procedure. 165 Copy & Move This section of the Database Management will present only those databases, which are correctly configured, assigned and do have a set of installed tables as described in chapter Definition and configuration. This . . .
. . .
results. Never the less to find these x results, the complete database had to been browsed. Starting with version 2.5, an additional clean option is offered as part of the Admin backend. Main advantage of this option is a significant reduction of the search time for any query, because the content of the db could be limited to offer only x . . .
. . .
<link>http://www.abc.de/images/warp.gif</link> <title>warp.gif</title> <x_size>635</x_size> <y_size>98</y_size> </media_result> <media_result> <num>2</num> <type>audio</type> <url>http://www.abc.de/index.php</url> . . .
. . .
the following content: {"query":"colour","ip":"::1","host_name":"guard_007-hoster","query_time":"2016-03-18 10:56:23 AM","consumed":0.016,"total_results":2,"num_of_results":2,"from":1,"to":2,"text_results":[{"num":1,"weight":"100.0","url":"http:\/\/www.english.le-piaggie.info\/Info_eng.pdf","title":" Info_eng","fulltxt":" . . . . . .
. . .
placed on the ground floor, today's big kitchen with the heating fireplace for open and close mode of . . . ","page_size":"5,661.6kb","domain_name":"www.english.le-piaggie.info"},{"num":2,"weight":"50.0","url":"http:\/\/www.english.le-piaggie.info\/html\/description.html","title":" Description","fulltxt":" . . . cable. Additional active . . .
. . .
D.O.C.G. as far as to the Pratomagno mountains 160 olive-trees Detached house, about 800 m to reach the . . . ","page_size":"26.4 kb","domain_name":"www.english.le-piaggie.info"}]} Top . . .


5a2 All required information. Introduction Release and Legal Info Installation Documentation Change Log [ Documentation Summary ] Preamble: The info presented here is valid only for the latest release of Sphider-plus. At present version 4.2024a published February 12, 2024 is the actual release. 1. Settings, customizing and statistics 2. Indexing . . .
. . .
2. Indexing 2.1 Various options 2.2 Allow other hosts in same domain 2.3 Word stemming 2.4 Periodical Re-indexing 2.5 Preferred indexing 2.6 Multithreaded indexing 2.7 Create thumbnails during index procedure 2.8 Prevent indexing of known malware and pishing pages 2.9 Follow and create sitemap files 2.10 Use private sitemap instead of global . . .
. . .
Sitemap file 3. Using the indexer from command line 3.1 All options 3.2 Multithreaded indexing 4. Keeping pages, words and files from being indexed 4.1 robots.txt 4.2 Must include / must not include string list 4.3 Ignoring links 4.4 Ignoring parts of a page by <! sphider_noindex > 4.5 Ignoring parts of a page by <div id='abc'> . . .
. . .
charset' 6. Search modes 6.1 Search with wildcards * 6.2 Strict search ! 6.3 Tolerant search 6.4 Link search 6.5 Media search 6.6 Search only in one domain 6.7 Search in categories 6.8 Greek language support 6.9 Block queries 7. Chronological order for result listing 7.1 Text result listing 7.2 Media result listing 8. PDF converter 9. Clean . . .
. . .
of logging data 11. Error messages and Debug mode 12. Delete secondary characters 13. Media search for images, audio streams and videos 13.1 Media indexing 13.2 Not supported media content 13.3 Search for media content 13.4 Statistics for media content 14. Feed support 14.1 XML product feeds 14.2 RDF, RSD, RSS and Atom feeds 15. Result . . .
. . .
Overview 16.2 Definition and configuration 16.3 Activate / disable database 16.4 Backup & Restore of databases 16.5 Copy & and Move 16.6 Enhancing functionality of multiple database support 17. Search in categories 17.1 Hierachical structure 17.2 Parallel structure 18. User suggested sites 19. Vulnerability protection 19.1 Prevent queries . . .
. . .
templates 22.2 Embed the search engine into existing HTML code 22.3 The different style sheet files 23. JSON, XML and RSS result output [ Documentation ] 1. Settings, customizing and statistics If you want to change settings, behavior and design of Sphider-plus, you can do so by means of the Admin interface. There is a wide range of . . .
. . .
interface. There is a wide range of settings foreseen for Sphider-plus. Separated into different submenus like: Sites: - Add Site - Index only the new - Re-index all - Re-index only preferred URLs - Erase Re-index (available also for individual URLs) - Import/export URL list - Approve sites - Banned domains Categories: - Add, edit, delete - . . .
. . .
URL list - Approve sites - Banned domains Categories: - Add, edit, delete - Create new subcategory under Index: - Basic indexing options - Advanced options Clean: - Clean keywords not associated with any link - Clean links not associated with any site - Clean Category table not associated with any site - Clean Media links - Clear Temp table . . .
. . .
- Clear Search log - Clear 'Most Popular Page Links' log - Clear 'Most Popular Media Links' log - Clear Spider log, separate and bulk delete - Clear Thumbnail images, separate and bulk delete - Clear Text cache - Clear Media cache - Clear IDS log file - Clear flood attempts log file - Clear all entries in addurl or banned table - Truncate all . . .
. . .
flood attempts log file - Clear all entries in addurl or banned table - Truncate all tables in database Settings: - General Settings - Index Log Settings - Spider Settings - Search Settings - Order of Result listing - Suggest Options - Page Indexing Weights Database: - Configure up to 5 databases with unlimited number of table sets - Activate . . .
. . .
Database: - Configure up to 5 databases with unlimited number of table sets - Activate separately for 'Admin', 'Search' user and 'Suggest URL user' - Backup / Restore - Copy / Move - Optimize Templates: In order to enable customer's integration of Sphider-plus into existing sites, HTML templates are prepared for Search form Text result . . .
. . .
Search form Text result listing Media result listing Most popular queries etc. Three different designs are offered, which may be selected in submenu 'Settings'. If the layout does not fit the design of your site (which is normal), you may create new designs and modify the appropriate file /templates/My_template/adminstyle.css . . .
. . .
the appropriate file /templates/My_template/adminstyle.css /templates/My_template/userstyle.css Statistics output: - Top keywords (Top 50 with hit counter). - All indexed thumbnails w 3e80 ith ID3 and EXIF info. - Larges pages offering link URL and file size. - Most Popular Searches for text links offering: Link addr., total clicks, last . . .
. . .
Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Searches for media links offering: Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Links (click counter). - Search log offering: Query, Results, Queried at, Time taken, User IP, Country, Host name (Latest100) - Index log offering: . . .
. . .
Country, Host name (Latest100) - Index log offering: File-name, index date and delete option - sitemap log offering: sitemap.xml output sitemap list offering file/page suffixes - IDS log offering: IP, host, query, impact, involved tags, date and time of intrusion. - Flood attempts log offering: IP, query, date and time of flood attempt. - Auto . . .
. . .
attempts log offering: IP, query, date and time of flood attempt. - Auto Re-index log file - Server info offering: Server software, environment, MySQL, PDF-converter, image functions, php.ini file PHP integration, PHP security info. Each item holding lists of details. All text links, media links and thumbnails are active linked. As stated in . . .
. . .
individual Re-indexing, the periodical Re-indexer could be started and aborted in the "Options" menu of each site. 2.5 Preferred Re-indexing Each new URL added to the Admin backend, could be supplied with a priority level. This level will be used by the option 'Re-index only preferred sites'. Level 1 will be interpreted as most important, while . . .
. . .
less all index results will be stored in log files in sub folder /admin/log/ The names of the log files look like: db2_100524-21.47.56_1.html (log file of first thread) db2_100524-21.48.12_2.html (log file of second thread) and is build by the following items: db2 - Number of database. 100524 - Date (May 24, 2010) 21.47.56 - Time when this thread . . .
. . .
spider.php -new1 php spider.php -new2 etc. The IDs will be added to the name of the corresponding log files like: db2_100524-21.47.56_ID1.html (log file of first thread) db2_100524-21.48.12_ID2.html (log file of second thread) IDs could be defined by personal requirements, but the limitations for file names with respect to the OS should be taken . . .
. . .
but will not erase the content of all the other tables. So the check whether the content of a page has changed (MD5sum) is still available for a fast re-index procedure. Once prepared, multithreaded re-indexing could be invoked by starting several threads and adding individual IDs to the option parameter like: php spider.php -erased1 php . . .
. . .
4.4 Ignoring parts of a page Sphider-plus includes an option to exclude parts of pages from being indexed. Thi 775e s can for example be used to prevent search result flooding when certain keywords appear on certain part in most pages (like a header, footer or a menu). Any part of a page between <! sphider_noindex > and <! . . .
. . .
<! sphider_noindex > and <! /sphider_noindex > tags is not indexed, however links in it are followed. 4.5 Ignoring parts of a page by <div id='abc'> Ignoring parts of a page by the <! sphider_noindex > tags requires direct access to the page, because the tags need to be added (edited) to the page. A more flexible method, . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ menu[0-5] / 4.6 Indexing only parts of a page by <div id='abc'> If enabled in Admin settings, the values as defined in the list-file /include/common/divs_use.txt will be used to index only the content between <div . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ table[0-5] / 4.7 Ignore HTML elements defined by <tagname> . . </tagname> This option is foreseen to cooperate with the new HTML5 elements like section, nav, aside, hgroup, article, header, footer, etc. HTML elements . . .
. . .
contain a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */nav[0-5]/ Please keep in mind that element names placed in /include/common/elements_not.txt will be processed case-sensitive. 4.8 Index only HTML elements defined by <tagname> . . </tagname> This is the vice versa . . .
. . .
of all the duplicate content URLs: http://www.example.com/product.php?item=swedish-fish&category=gummy-candy http://www.example.com/product.php?item=swedish-fish&trackingid=1234&sessionid=5678 and Sphider-plus will understand that the duplicates all refer to the canonical URL: http://www.example.com/product.php?item=swedish-fish. The . . .
. . .
charset are supported by the ConvertCharset function and will be used to convert text into UTF-8 Unicode: WINDOWS windows-1250 - Central Europe windows-1251 - Cyrillic windows-1252 - Latin I windows-1253 - Greek windows-1254 - Turkish windows-1255 - Hebrew windows-1256 - Arabic windows-1257 - Baltic windows-1258 - Viet Nam cp874 - Thai - this . . .
. . .
- Baltic windows-1258 - Viet Nam cp874 - Thai - this file is also for DOS DOS cp437 - Latin US cp737 - Greek cp775 - BaltRim cp850 - Latin1 cp852 - Latin2 cp855 - Cyrylic cp857 - Turkish cp860 - Portuguese cp861 - Iceland cp862 - Hebrew cp863 - Canada cp864 - Arabic cp865 - Nordic cp866 - Cyrylic Russian (this is the one, used in IE . . .
. . .
IE Cyrillic (DOS) ) cp869 - Greek2 MAC (Apple) x-mac-cyrillic x-mac-greek x-mac-icelandic x-mac-ce x-mac-roman ISO iso-8859-1 iso-8859-2 iso-8859-3 iso-8859-4 iso-8859-5 iso-8859-6 iso-8859-7 iso-8859-8 iso-8859-9 iso-8859-10 iso-8859-11 iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 . . .
. . .
iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 cp500 cp856 cp875 cp1006 cp1026 koi8-r (Cyrillic) koi8-u (Cyrillic Ukrainian) nextstep us-ascii us-ascii-quotes DSP implementation for NeXT stdenc symbol zdingbat And specially for old Polish programs: mazovia This list is to be read . . .
. . .
Access denied; you need the RELOAD privilege. . . Top 10. Enable real-time output of logging data Up to version 1.5 of Sphider-plus, during index / re-index there was no printout available because: - Several servers, especially on Win32, buffer the output from the script until it terminates before transmitting the results to the browser. - . . .
. . .
is seen. - Some versions of Microsoft Internet Explorer only start to display the page after they have received 256 bytes of output. As progress was not presented during index / re- index procedure, waiting for results became a pain in the neck. Selectable in Admin setting together with the update interval (1 - 10 seconds), AJAX technology was . . .
. . .
characters in front of words Warning: This option should be used with special care and not be activated for non ISO-8859 charsets. Some special characters as part of the word ending might be erased by accidental. Top 13. Media search for images, audio streams and videos 13.1 Media indexing Index of media files is enabled by separated Admin . . .
. . .
and 'Found at' - Total clicks - Last clicked - Query input 'Indexed Image Thumbnails' presenting: - Thumbnail 150 x 100 pixel - Image details like title, filename size of original image, link- and thumb-id - Option to delete the thumbnail In order to open the media files all tables contain active links. Media results are also stored in . . .
. . .
Settings' menu of the admin backend: First option: For results of XML product feeds, present the text extract of 350 characters as part of product attributes. If this option is not activated, all products of the XML product feed will be presented in search results, and the hits of the query string will be highlighted in all involved products. . . .
. . .
content of the database tables (those with the same table prefix) will be destroyed by the restore procedure. 16.5 Copy & Move This section of the Database Management will present only those databases, which are correctly configured, assigned and do have a set of installed tables as described in chapter Definition and configuration. This . . .
. . .
results. Never the less to find these x results, the complete database had to been browsed. Starting with version 2.5, an additional clean option is offered as part of the Admin backend. Main advantage of this option is a significant reduction of the search time for any query, because the content of the db could be limited to offer only x . . .
. . .
<link>http://www.abc.de/images/warp.gif</link> <title>warp.gif</title> <x_size>635</x_size> <y_size>98</y_size> </media_result> <media_result> <num>2</num> <type>audio</type> <url>http://www.abc.de/index.php</url> . . .
. . .
the following content: {"query":"colour","ip":"::1","host_name":"guard_007-hoster","query_time":"2016-03-18 10:56:23 AM","consumed":0.016,"total_results":2,"num_of_results":2,"from":1,"to":2,"text_results":[{"num":1,"weight":"100.0","url":"http:\/\/www.english.le-piaggie.info\/Info_eng.pdf","title":" Info_eng","fulltxt":" . . . . . .
. . .
placed on the ground floor, today's big kitchen with the heating fireplace for open and close mode of . . . ","page_size":"5,661.6kb","domain_name":"www.english.le-piaggie.info"},{"num":2,"weight":"50.0","url":"http:\/\/www.english.le-piaggie.info\/html\/description.html","title":" Description","fulltxt":" . . . cable. Additional active . . .
. . .
D.O.C.G. as far as to the Pratomagno mountains 160 olive-trees Detached house, about 800 m to reach the . . . ","page_size":"26.4 kb","domain_name":"www.english.le-piaggie.info"}]} Top . . .


5a2 All required information. Introduction Release and Legal Info Installation Documentation Change Log [ Documentation Summary ] Preamble: The info presented here is valid only for the latest release of Sphider-plus. At present version 4.2024a published February 12, 2024 is the actual release. 1. Settings, customizing and statistics 2. Indexing . . .
. . .
2. Indexing 2.1 Various options 2.2 Allow other hosts in same domain 2.3 Word stemming 2.4 Periodical Re-indexing 2.5 Preferred indexing 2.6 Multithreaded indexing 2.7 Create thumbnails during index procedure 2.8 Prevent indexing of known malware and pishing pages 2.9 Follow and create sitemap files 2.10 Use private sitemap instead of global . . .
. . .
Sitemap file 3. Using the indexer from command line 3.1 All options 3.2 Multithreaded indexing 4. Keeping pages, words and files from being indexed 4.1 robots.txt 4.2 Must include / must not include string list 4.3 Ignoring links 4.4 Ignoring parts of a page by <! sphider_noindex > 4.5 Ignoring parts of a page by <div id='abc'> . . .
. . .
charset' 6. Search modes 6.1 Search with wildcards * 6.2 Strict search ! 6.3 Tolerant search 6.4 Link search 6.5 Media search 6.6 Search only in one domain 6.7 Search in categories 6.8 Greek language support 6.9 Block queries 7. Chronological order for result listing 7.1 Text result listing 7.2 Media result listing 8. PDF converter 9. Clean . . .
. . .
of logging data 11. Error messages and Debug mode 12. Delete secondary characters 13. Media search for images, audio streams and videos 13.1 Media indexing 13.2 Not supported media content 13.3 Search for media content 13.4 Statistics for media content 14. Feed support 14.1 XML product feeds 14.2 RDF, RSD, RSS and Atom feeds 15. Result . . .
. . .
Overview 16.2 Definition and configuration 16.3 Activate / disable database 16.4 Backup & Restore of databases 16.5 Copy & and Move 16.6 Enhancing functionality of multiple database support 17. Search in categories 17.1 Hierachical structure 17.2 Parallel structure 18. User suggested sites 19. Vulnerability protection 19.1 Prevent queries . . .
. . .
templates 22.2 Embed the search engine into existing HTML code 22.3 The different style sheet files 23. JSON, XML and RSS result output [ Documentation ] 1. Settings, customizing and statistics If you want to change settings, behavior and design of Sphider-plus, you can do so by means of the Admin interface. There is a wide range of . . .
. . .
interface. There is a wide range of settings foreseen for Sphider-plus. Separated into different submenus like: Sites: - Add Site - Index only the new - Re-index all - Re-index only preferred URLs - Erase Re-index (available also for individual URLs) - Import/export URL list - Approve sites - Banned domains Categories: - Add, edit, delete - . . .
. . .
URL list - Approve sites - Banned domains Categories: - Add, edit, delete - Create new subcategory under Index: - Basic indexing options - Advanced options Clean: - Clean keywords not associated with any link - Clean links not associated with any site - Clean Category table not associated with any site - Clean Media links - Clear Temp table . . .
. . .
- Clear Search log - Clear 'Most Popular Page Links' log - Clear 'Most Popular Media Links' log - Clear Spider log, separate and bulk delete - Clear Thumbnail images, separate and bulk delete - Clear Text cache - Clear Media cache - Clear IDS log file - Clear flood attempts log file - Clear all entries in addurl or banned table - Truncate all . . .
. . .
flood attempts log file - Clear all entries in addurl or banned table - Truncate all tables in database Settings: - General Settings - Index Log Settings - Spider Settings - Search Settings - Order of Result listing - Suggest Options - Page Indexing Weights Database: - Configure up to 5 databases with unlimited number of table sets - Activate . . .
. . .
Database: - Configure up to 5 databases with unlimited number of table sets - Activate separately for 'Admin', 'Search' user and 'Suggest URL user' - Backup / Restore - Copy / Move - Optimize Templates: In order to enable customer's integration of Sphider-plus into existing sites, HTML templates are prepared for Search form Text result . . .
. . .
Search form Text result listing Media result listing Most popular queries etc. Three different designs are offered, which may be selected in submenu 'Settings'. If the layout does not fit the design of your site (which is normal), you may create new designs and modify the appropriate file /templates/My_template/adminstyle.css . . .
. . .
the appropriate file /templates/My_template/adminstyle.css /templates/My_template/userstyle.css Statistics output: - Top keywords (Top 50 with hit counter). - All indexed thumbnails w 3e80 ith ID3 and EXIF info. - Larges pages offering link URL and file size. - Most Popular Searches for text links offering: Link addr., total clicks, last . . .
. . .
Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Searches for media links offering: Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Links (click counter). - Search log offering: Query, Results, Queried at, Time taken, User IP, Country, Host name (Latest100) - Index log offering: . . .
. . .
Country, Host name (Latest100) - Index log offering: File-name, index date and delete option - sitemap log offering: sitemap.xml output sitemap list offering file/page suffixes - IDS log offering: IP, host, query, impact, involved tags, date and time of intrusion. - Flood attempts log offering: IP, query, date and time of flood attempt. - Auto . . .
. . .
attempts log offering: IP, query, date and time of flood attempt. - Auto Re-index log file - Server info offering: Server software, environment, MySQL, PDF-converter, image functions, php.ini file PHP integration, PHP security info. Each item holding lists of details. All text links, media links and thumbnails are active linked. As stated in . . .
. . .
individual Re-indexing, the periodical Re-indexer could be started and aborted in the "Options" menu of each site. 2.5 Preferred Re-indexing Each new URL added to the Admin backend, could be supplied with a priority level. This level will be used by the option 'Re-index only preferred sites'. Level 1 will be interpreted as most important, while . . .
. . .
less all index results will be stored in log files in sub folder /admin/log/ The names of the log files look like: db2_100524-21.47.56_1.html (log file of first thread) db2_100524-21.48.12_2.html (log file of second thread) and is build by the following items: db2 - Number of database. 100524 - Date (May 24, 2010) 21.47.56 - Time when this thread . . .
. . .
spider.php -new1 php spider.php -new2 etc. The IDs will be added to the name of the corresponding log files like: db2_100524-21.47.56_ID1.html (log file of first thread) db2_100524-21.48.12_ID2.html (log file of second thread) IDs could be defined by personal requirements, but the limitations for file names with respect to the OS should be taken . . .
. . .
but will not erase the content of all the other tables. So the check whether the content of a page has changed (MD5sum) is still available for a fast re-index procedure. Once prepared, multithreaded re-indexing could be invoked by starting several threads and adding individual IDs to the option parameter like: php spider.php -erased1 php . . .
. . .
4.4 Ignoring parts of a page Sphider-plus includes an option to exclude parts of pages from being indexed. Thi 775e s can for example be used to prevent search result flooding when certain keywords appear on certain part in most pages (like a header, footer or a menu). Any part of a page between <! sphider_noindex > and <! . . .
. . .
<! sphider_noindex > and <! /sphider_noindex > tags is not indexed, however links in it are followed. 4.5 Ignoring parts of a page by <div id='abc'> Ignoring parts of a page by the <! sphider_noindex > tags requires direct access to the page, because the tags need to be added (edited) to the page. A more flexible method, . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ menu[0-5] / 4.6 Indexing only parts of a page by <div id='abc'> If enabled in Admin settings, the values as defined in the list-file /include/common/divs_use.txt will be used to index only the content between <div . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ table[0-5] / 4.7 Ignore HTML elements defined by <tagname> . . </tagname> This option is foreseen to cooperate with the new HTML5 elements like section, nav, aside, hgroup, article, header, footer, etc. HTML elements . . .
. . .
contain a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */nav[0-5]/ Please keep in mind that element names placed in /include/common/elements_not.txt will be processed case-sensitive. 4.8 Index only HTML elements defined by <tagname> . . </tagname> This is the vice versa . . .
. . .
of all the duplicate content URLs: http://www.example.com/product.php?item=swedish-fish&category=gummy-candy http://www.example.com/product.php?item=swedish-fish&trackingid=1234&sessionid=5678 and Sphider-plus will understand that the duplicates all refer to the canonical URL: http://www.example.com/product.php?item=swedish-fish. The . . .
. . .
charset are supported by the ConvertCharset function and will be used to convert text into UTF-8 Unicode: WINDOWS windows-1250 - Central Europe windows-1251 - Cyrillic windows-1252 - Latin I windows-1253 - Greek windows-1254 - Turkish windows-1255 - Hebrew windows-1256 - Arabic windows-1257 - Baltic windows-1258 - Viet Nam cp874 - Thai - this . . .
. . .
- Baltic windows-1258 - Viet Nam cp874 - Thai - this file is also for DOS DOS cp437 - Latin US cp737 - Greek cp775 - BaltRim cp850 - Latin1 cp852 - Latin2 cp855 - Cyrylic cp857 - Turkish cp860 - Portuguese cp861 - Iceland cp862 - Hebrew cp863 - Canada cp864 - Arabic cp865 - Nordic cp866 - Cyrylic Russian (this is the one, used in IE . . .
. . .
IE Cyrillic (DOS) ) cp869 - Greek2 MAC (Apple) x-mac-cyrillic x-mac-greek x-mac-icelandic x-mac-ce x-mac-roman ISO iso-8859-1 iso-8859-2 iso-8859-3 iso-8859-4 iso-8859-5 iso-8859-6 iso-8859-7 iso-8859-8 iso-8859-9 iso-8859-10 iso-8859-11 iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 . . .
. . .
iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 cp500 cp856 cp875 cp1006 cp1026 koi8-r (Cyrillic) koi8-u (Cyrillic Ukrainian) nextstep us-ascii us-ascii-quotes DSP implementation for NeXT stdenc symbol zdingbat And specially for old Polish programs: mazovia This list is to be read . . .
. . .
Access denied; you need the RELOAD privilege. . . Top 10. Enable real-time output of logging data Up to version 1.5 of Sphider-plus, during index / re-index there was no printout available because: - Several servers, especially on Win32, buffer the output from the script until it terminates before transmitting the results to the browser. - . . .
. . .
is seen. - Some versions of Microsoft Internet Explorer only start to display the page after they have received 256 bytes of output. As progress was not presented during index / re- index procedure, waiting for results became a pain in the neck. Selectable in Admin setting together with the update interval (1 - 10 seconds), AJAX technology was . . .
. . .
characters in front of words Warning: This option should be used with special care and not be activated for non ISO-8859 charsets. Some special characters as part of the word ending might be erased by accidental. Top 13. Media search for images, audio streams and videos 13.1 Media indexing Index of media files is enabled by separated Admin . . .
. . .
and 'Found at' - Total clicks - Last clicked - Query input 'Indexed Image Thumbnails' presenting: - Thumbnail 150 x 100 pixel - Image details like title, filename size of original image, link- and thumb-id - Option to delete the thumbnail In order to open the media files all tables contain active links. Media results are also stored in . . .
. . .
Settings' menu of the admin backend: First option: For results of XML product feeds, present the text extract of 350 characters as part of product attributes. If this option is not activated, all products of the XML product feed will be presented in search results, and the hits of the query string will be highlighted in all involved products. . . .
. . .
content of the database tables (those with the same table prefix) will be destroyed by the restore procedure. 16.5 Copy & Move This section of the Database Management will present only those databases, which are correctly configured, assigned and do have a set of installed tables as described in chapter Definition and configuration. This . . .
. . .
results. Never the less to find these x results, the complete database had to been browsed. Starting with version 2.5, an additional clean option is offered as part of the Admin backend. Main advantage of this option is a significant reduction of the search time for any query, because the content of the db could be limited to offer only x . . .
. . .
<link>http://www.abc.de/images/warp.gif</link> <title>warp.gif</title> <x_size>635</x_size> <y_size>98</y_size> </media_result> <media_result> <num>2</num> <type>audio</type> <url>http://www.abc.de/index.php</url> . . .
. . .
the following content: {"query":"colour","ip":"::1","host_name":"guard_007-hoster","query_time":"2016-03-18 10:56:23 AM","consumed":0.016,"total_results":2,"num_of_results":2,"from":1,"to":2,"text_results":[{"num":1,"weight":"100.0","url":"http:\/\/www.english.le-piaggie.info\/Info_eng.pdf","title":" Info_eng","fulltxt":" . . . . . .
. . .
placed on the ground floor, today's big kitchen with the heating fireplace for open and close mode of . . . ","page_size":"5,661.6kb","domain_name":"www.english.le-piaggie.info"},{"num":2,"weight":"50.0","url":"http:\/\/www.english.le-piaggie.info\/html\/description.html","title":" Description","fulltxt":" . . . cable. Additional active . . .
. . .
D.O.C.G. as far as to the Pratomagno mountains 160 olive-trees Detached house, about 800 m to reach the . . . ","page_size":"26.4 kb","domain_name":"www.english.le-piaggie.info"}]} Top . . .


5a2 All required information. Introduction Release and Legal Info Installation Documentation Change Log [ Documentation Summary ] Preamble: The info presented here is valid only for the latest release of Sphider-plus. At present version 4.2024a published February 12, 2024 is the actual release. 1. Settings, customizing and statistics 2. Indexing . . .
. . .
2. Indexing 2.1 Various options 2.2 Allow other hosts in same domain 2.3 Word stemming 2.4 Periodical Re-indexing 2.5 Preferred indexing 2.6 Multithreaded indexing 2.7 Create thumbnails during index procedure 2.8 Prevent indexing of known malware and pishing pages 2.9 Follow and create sitemap files 2.10 Use private sitemap instead of global . . .
. . .
Sitemap file 3. Using the indexer from command line 3.1 All options 3.2 Multithreaded indexing 4. Keeping pages, words and files from being indexed 4.1 robots.txt 4.2 Must include / must not include string list 4.3 Ignoring links 4.4 Ignoring parts of a page by <! sphider_noindex > 4.5 Ignoring parts of a page by <div id='abc'> . . .
. . .
charset' 6. Search modes 6.1 Search with wildcards * 6.2 Strict search ! 6.3 Tolerant search 6.4 Link search 6.5 Media search 6.6 Search only in one domain 6.7 Search in categories 6.8 Greek language support 6.9 Block queries 7. Chronological order for result listing 7.1 Text result listing 7.2 Media result listing 8. PDF converter 9. Clean . . .
. . .
of logging data 11. Error messages and Debug mode 12. Delete secondary characters 13. Media search for images, audio streams and videos 13.1 Media indexing 13.2 Not supported media content 13.3 Search for media content 13.4 Statistics for media content 14. Feed support 14.1 XML product feeds 14.2 RDF, RSD, RSS and Atom feeds 15. Result . . .
. . .
Overview 16.2 Definition and configuration 16.3 Activate / disable database 16.4 Backup & Restore of databases 16.5 Copy & and Move 16.6 Enhancing functionality of multiple database support 17. Search in categories 17.1 Hierachical structure 17.2 Parallel structure 18. User suggested sites 19. Vulnerability protection 19.1 Prevent queries . . .
. . .
templates 22.2 Embed the search engine into existing HTML code 22.3 The different style sheet files 23. JSON, XML and RSS result output [ Documentation ] 1. Settings, customizing and statistics If you want to change settings, behavior and design of Sphider-plus, you can do so by means of the Admin interface. There is a wide range of . . .
. . .
interface. There is a wide range of settings foreseen for Sphider-plus. Separated into different submenus like: Sites: - Add Site - Index only the new - Re-index all - Re-index only preferred URLs - Erase Re-index (available also for individual URLs) - Import/export URL list - Approve sites - Banned domains Categories: - Add, edit, delete - . . .
. . .
URL list - Approve sites - Banned domains Categories: - Add, edit, delete - Create new subcategory under Index: - Basic indexing options - Advanced options Clean: - Clean keywords not associated with any link - Clean links not associated with any site - Clean Category table not associated with any site - Clean Media links - Clear Temp table . . .
. . .
- Clear Search log - Clear 'Most Popular Page Links' log - Clear 'Most Popular Media Links' log - Clear Spider log, separate and bulk delete - Clear Thumbnail images, separate and bulk delete - Clear Text cache - Clear Media cache - Clear IDS log file - Clear flood attempts log file - Clear all entries in addurl or banned table - Truncate all . . .
. . .
flood attempts log file - Clear all entries in addurl or banned table - Truncate all tables in database Settings: - General Settings - Index Log Settings - Spider Settings - Search Settings - Order of Result listing - Suggest Options - Page Indexing Weights Database: - Configure up to 5 databases with unlimited number of table sets - Activate . . .
. . .
Database: - Configure up to 5 databases with unlimited number of table sets - Activate separately for 'Admin', 'Search' user and 'Suggest URL user' - Backup / Restore - Copy / Move - Optimize Templates: In order to enable customer's integration of Sphider-plus into existing sites, HTML templates are prepared for Search form Text result . . .
. . .
Search form Text result listing Media result listing Most popular queries etc. Three different designs are offered, which may be selected in submenu 'Settings'. If the layout does not fit the design of your site (which is normal), you may create new designs and modify the appropriate file /templates/My_template/adminstyle.css . . .
. . .
the appropriate file /templates/My_template/adminstyle.css /templates/My_template/userstyle.css Statistics output: - Top keywords (Top 50 with hit counter). - All indexed thumbnails w 3e80 ith ID3 and EXIF info. - Larges pages offering link URL and file size. - Most Popular Searches for text links offering: Link addr., total clicks, last . . .
. . .
Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Searches for media links offering: Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Links (click counter). - Search log offering: Query, Results, Queried at, Time taken, User IP, Country, Host name (Latest100) - Index log offering: . . .
. . .
Country, Host name (Latest100) - Index log offering: File-name, index date and delete option - sitemap log offering: sitemap.xml output sitemap list offering file/page suffixes - IDS log offering: IP, host, query, impact, involved tags, date and time of intrusion. - Flood attempts log offering: IP, query, date and time of flood attempt. - Auto . . .
. . .
attempts log offering: IP, query, date and time of flood attempt. - Auto Re-index log file - Server info offering: Server software, environment, MySQL, PDF-converter, image functions, php.ini file PHP integration, PHP security info. Each item holding lists of details. All text links, media links and thumbnails are active linked. As stated in . . .
. . .
individual Re-indexing, the periodical Re-indexer could be started and aborted in the "Options" menu of each site. 2.5 Preferred Re-indexing Each new URL added to the Admin backend, could be supplied with a priority level. This level will be used by the option 'Re-index only preferred sites'. Level 1 will be interpreted as most important, while . . .
. . .
less all index results will be stored in log files in sub folder /admin/log/ The names of the log files look like: db2_100524-21.47.56_1.html (log file of first thread) db2_100524-21.48.12_2.html (log file of second thread) and is build by the following items: db2 - Number of database. 100524 - Date (May 24, 2010) 21.47.56 - Time when this thread . . .
. . .
spider.php -new1 php spider.php -new2 etc. The IDs will be added to the name of the corresponding log files like: db2_100524-21.47.56_ID1.html (log file of first thread) db2_100524-21.48.12_ID2.html (log file of second thread) IDs could be defined by personal requirements, but the limitations for file names with respect to the OS should be taken . . .
. . .
but will not erase the content of all the other tables. So the check whether the content of a page has changed (MD5sum) is still available for a fast re-index procedure. Once prepared, multithreaded re-indexing could be invoked by starting several threads and adding individual IDs to the option parameter like: php spider.php -erased1 php . . .
. . .
4.4 Ignoring parts of a page Sphider-plus includes an option to exclude parts of pages from being indexed. Thi 775e s can for example be used to prevent search result flooding when certain keywords appear on certain part in most pages (like a header, footer or a menu). Any part of a page between <! sphider_noindex > and <! . . .
. . .
<! sphider_noindex > and <! /sphider_noindex > tags is not indexed, however links in it are followed. 4.5 Ignoring parts of a page by <div id='abc'> Ignoring parts of a page by the <! sphider_noindex > tags requires direct access to the page, because the tags need to be added (edited) to the page. A more flexible method, . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ menu[0-5] / 4.6 Indexing only parts of a page by <div id='abc'> If enabled in Admin settings, the values as defined in the list-file /include/common/divs_use.txt will be used to index only the content between <div . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ table[0-5] / 4.7 Ignore HTML elements defined by <tagname> . . </tagname> This option is foreseen to cooperate with the new HTML5 elements like section, nav, aside, hgroup, article, header, footer, etc. HTML elements . . .
. . .
contain a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */nav[0-5]/ Please keep in mind that element names placed in /include/common/elements_not.txt will be processed case-sensitive. 4.8 Index only HTML elements defined by <tagname> . . </tagname> This is the vice versa . . .
. . .
of all the duplicate content URLs: http://www.example.com/product.php?item=swedish-fish&category=gummy-candy http://www.example.com/product.php?item=swedish-fish&trackingid=1234&sessionid=5678 and Sphider-plus will understand that the duplicates all refer to the canonical URL: http://www.example.com/product.php?item=swedish-fish. The . . .
. . .
charset are supported by the ConvertCharset function and will be used to convert text into UTF-8 Unicode: WINDOWS windows-1250 - Central Europe windows-1251 - Cyrillic windows-1252 - Latin I windows-1253 - Greek windows-1254 - Turkish windows-1255 - Hebrew windows-1256 - Arabic windows-1257 - Baltic windows-1258 - Viet Nam cp874 - Thai - this . . .
. . .
- Baltic windows-1258 - Viet Nam cp874 - Thai - this file is also for DOS DOS cp437 - Latin US cp737 - Greek cp775 - BaltRim cp850 - Latin1 cp852 - Latin2 cp855 - Cyrylic cp857 - Turkish cp860 - Portuguese cp861 - Iceland cp862 - Hebrew cp863 - Canada cp864 - Arabic cp865 - Nordic cp866 - Cyrylic Russian (this is the one, used in IE . . .
. . .
IE Cyrillic (DOS) ) cp869 - Greek2 MAC (Apple) x-mac-cyrillic x-mac-greek x-mac-icelandic x-mac-ce x-mac-roman ISO iso-8859-1 iso-8859-2 iso-8859-3 iso-8859-4 iso-8859-5 iso-8859-6 iso-8859-7 iso-8859-8 iso-8859-9 iso-8859-10 iso-8859-11 iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 . . .
. . .
iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 cp500 cp856 cp875 cp1006 cp1026 koi8-r (Cyrillic) koi8-u (Cyrillic Ukrainian) nextstep us-ascii us-ascii-quotes DSP implementation for NeXT stdenc symbol zdingbat And specially for old Polish programs: mazovia This list is to be read . . .
. . .
Access denied; you need the RELOAD privilege. . . Top 10. Enable real-time output of logging data Up to version 1.5 of Sphider-plus, during index / re-index there was no printout available because: - Several servers, especially on Win32, buffer the output from the script until it terminates before transmitting the results to the browser. - . . .
. . .
is seen. - Some versions of Microsoft Internet Explorer only start to display the page after they have received 256 bytes of output. As progress was not presented during index / re- index procedure, waiting for results became a pain in the neck. Selectable in Admin setting together with the update interval (1 - 10 seconds), AJAX technology was . . .
. . .
characters in front of words Warning: This option should be used with special care and not be activated for non ISO-8859 charsets. Some special characters as part of the word ending might be erased by accidental. Top 13. Media search for images, audio streams and videos 13.1 Media indexing Index of media files is enabled by separated Admin . . .
. . .
and 'Found at' - Total clicks - Last clicked - Query input 'Indexed Image Thumbnails' presenting: - Thumbnail 150 x 100 pixel - Image details like title, filename size of original image, link- and thumb-id - Option to delete the thumbnail In order to open the media files all tables contain active links. Media results are also stored in . . .
. . .
Settings' menu of the admin backend: First option: For results of XML product feeds, present the text extract of 350 characters as part of product attributes. If this option is not activated, all products of the XML product feed will be presented in search results, and the hits of the query string will be highlighted in all involved products. . . .
. . .
content of the database tables (those with the same table prefix) will be destroyed by the restore procedure. 16.5 Copy & Move This section of the Database Management will present only those databases, which are correctly configured, assigned and do have a set of installed tables as described in chapter Definition and configuration. This . . .
. . .
results. Never the less to find these x results, the complete database had to been browsed. Starting with version 2.5, an additional clean option is offered as part of the Admin backend. Main advantage of this option is a significant reduction of the search time for any query, because the content of the db could be limited to offer only x . . .
. . .
<link>http://www.abc.de/images/warp.gif</link> <title>warp.gif</title> <x_size>635</x_size> <y_size>98</y_size> </media_result> <media_result> <num>2</num> <type>audio</type> <url>http://www.abc.de/index.php</url> . . .
. . .
the following content: {"query":"colour","ip":"::1","host_name":"guard_007-hoster","query_time":"2016-03-18 10:56:23 AM","consumed":0.016,"total_results":2,"num_of_results":2,"from":1,"to":2,"text_results":[{"num":1,"weight":"100.0","url":"http:\/\/www.english.le-piaggie.info\/Info_eng.pdf","title":" Info_eng","fulltxt":" . . . . . .
. . .
placed on the ground floor, today's big kitchen with the heating fireplace for open and close mode of . . . ","page_size":"5,661.6kb","domain_name":"www.english.le-piaggie.info"},{"num":2,"weight":"50.0","url":"http:\/\/www.english.le-piaggie.info\/html\/description.html","title":" Description","fulltxt":" . . . cable. Additional active . . .
. . .
D.O.C.G. as far as to the Pratomagno mountains 160 olive-trees Detached house, about 800 m to reach the . . . ","page_size":"26.4 kb","domain_name":"www.english.le-piaggie.info"}]} Top . . .


5a2 All required information. Introduction Release and Legal Info Installation Documentation Change Log [ Documentation Summary ] Preamble: The info presented here is valid only for the latest release of Sphider-plus. At present version 4.2024a published February 12, 2024 is the actual release. 1. Settings, customizing and statistics 2. Indexing . . .
. . .
2. Indexing 2.1 Various options 2.2 Allow other hosts in same domain 2.3 Word stemming 2.4 Periodical Re-indexing 2.5 Preferred indexing 2.6 Multithreaded indexing 2.7 Create thumbnails during index procedure 2.8 Prevent indexing of known malware and pishing pages 2.9 Follow and create sitemap files 2.10 Use private sitemap instead of global . . .
. . .
Sitemap file 3. Using the indexer from command line 3.1 All options 3.2 Multithreaded indexing 4. Keeping pages, words and files from being indexed 4.1 robots.txt 4.2 Must include / must not include string list 4.3 Ignoring links 4.4 Ignoring parts of a page by <! sphider_noindex > 4.5 Ignoring parts of a page by <div id='abc'> . . .
. . .
charset' 6. Search modes 6.1 Search with wildcards * 6.2 Strict search ! 6.3 Tolerant search 6.4 Link search 6.5 Media search 6.6 Search only in one domain 6.7 Search in categories 6.8 Greek language support 6.9 Block queries 7. Chronological order for result listing 7.1 Text result listing 7.2 Media result listing 8. PDF converter 9. Clean . . .
. . .
of logging data 11. Error messages and Debug mode 12. Delete secondary characters 13. Media search for images, audio streams and videos 13.1 Media indexing 13.2 Not supported media content 13.3 Search for media content 13.4 Statistics for media content 14. Feed support 14.1 XML product feeds 14.2 RDF, RSD, RSS and Atom feeds 15. Result . . .
. . .
Overview 16.2 Definition and configuration 16.3 Activate / disable database 16.4 Backup & Restore of databases 16.5 Copy & and Move 16.6 Enhancing functionality of multiple database support 17. Search in categories 17.1 Hierachical structure 17.2 Parallel structure 18. User suggested sites 19. Vulnerability protection 19.1 Prevent queries . . .
. . .
templates 22.2 Embed the search engine into existing HTML code 22.3 The different style sheet files 23. JSON, XML and RSS result output [ Documentation ] 1. Settings, customizing and statistics If you want to change settings, behavior and design of Sphider-plus, you can do so by means of the Admin interface. There is a wide range of . . .
. . .
interface. There is a wide range of settings foreseen for Sphider-plus. Separated into different submenus like: Sites: - Add Site - Index only the new - Re-index all - Re-index only preferred URLs - Erase Re-index (available also for individual URLs) - Import/export URL list - Approve sites - Banned domains Categories: - Add, edit, delete - . . .
. . .
URL list - Approve sites - Banned domains Categories: - Add, edit, delete - Create new subcategory under Index: - Basic indexing options - Advanced options Clean: - Clean keywords not associated with any link - Clean links not associated with any site - Clean Category table not associated with any site - Clean Media links - Clear Temp table . . .
. . .
- Clear Search log - Clear 'Most Popular Page Links' log - Clear 'Most Popular Media Links' log - Clear Spider log, separate and bulk delete - Clear Thumbnail images, separate and bulk delete - Clear Text cache - Clear Media cache - Clear IDS log file - Clear flood attempts log file - Clear all entries in addurl or banned table - Truncate all . . .
. . .
flood attempts log file - Clear all entries in addurl or banned table - Truncate all tables in database Settings: - General Settings - Index Log Settings - Spider Settings - Search Settings - Order of Result listing - Suggest Options - Page Indexing Weights Database: - Configure up to 5 databases with unlimited number of table sets - Activate . . .
. . .
Database: - Configure up to 5 databases with unlimited number of table sets - Activate separately for 'Admin', 'Search' user and 'Suggest URL user' - Backup / Restore - Copy / Move - Optimize Templates: In order to enable customer's integration of Sphider-plus into existing sites, HTML templates are prepared for Search form Text result . . .
. . .
Search form Text result listing Media result listing Most popular queries etc. Three different designs are offered, which may be selected in submenu 'Settings'. If the layout does not fit the design of your site (which is normal), you may create new designs and modify the appropriate file /templates/My_template/adminstyle.css . . .
. . .
the appropriate file /templates/My_template/adminstyle.css /templates/My_template/userstyle.css Statistics output: - Top keywords (Top 50 with hit counter). - All indexed thumbnails w 3e80 ith ID3 and EXIF info. - Larges pages offering link URL and file size. - Most Popular Searches for text links offering: Link addr., total clicks, last . . .
. . .
Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Searches for media links offering: Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Links (click counter). - Search log offering: Query, Results, Queried at, Time taken, User IP, Country, Host name (Latest100) - Index log offering: . . .
. . .
Country, Host name (Latest100) - Index log offering: File-name, index date and delete option - sitemap log offering: sitemap.xml output sitemap list offering file/page suffixes - IDS log offering: IP, host, query, impact, involved tags, date and time of intrusion. - Flood attempts log offering: IP, query, date and time of flood attempt. - Auto . . .
. . .
attempts log offering: IP, query, date and time of flood attempt. - Auto Re-index log file - Server info offering: Server software, environment, MySQL, PDF-converter, image functions, php.ini file PHP integration, PHP security info. Each item holding lists of details. All text links, media links and thumbnails are active linked. As stated in . . .
. . .
individual Re-indexing, the periodical Re-indexer could be started and aborted in the "Options" menu of each site. 2.5 Preferred Re-indexing Each new URL added to the Admin backend, could be supplied with a priority level. This level will be used by the option 'Re-index only preferred sites'. Level 1 will be interpreted as most important, while . . .
. . .
less all index results will be stored in log files in sub folder /admin/log/ The names of the log files look like: db2_100524-21.47.56_1.html (log file of first thread) db2_100524-21.48.12_2.html (log file of second thread) and is build by the following items: db2 - Number of database. 100524 - Date (May 24, 2010) 21.47.56 - Time when this thread . . .
. . .
spider.php -new1 php spider.php -new2 etc. The IDs will be added to the name of the corresponding log files like: db2_100524-21.47.56_ID1.html (log file of first thread) db2_100524-21.48.12_ID2.html (log file of second thread) IDs could be defined by personal requirements, but the limitations for file names with respect to the OS should be taken . . .
. . .
but will not erase the content of all the other tables. So the check whether the content of a page has changed (MD5sum) is still available for a fast re-index procedure. Once prepared, multithreaded re-indexing could be invoked by starting several threads and adding individual IDs to the option parameter like: php spider.php -erased1 php . . .
. . .
4.4 Ignoring parts of a page Sphider-plus includes an option to exclude parts of pages from being indexed. Thi 775e s can for example be used to prevent search result flooding when certain keywords appear on certain part in most pages (like a header, footer or a menu). Any part of a page between <! sphider_noindex > and <! . . .
. . .
<! sphider_noindex > and <! /sphider_noindex > tags is not indexed, however links in it are followed. 4.5 Ignoring parts of a page by <div id='abc'> Ignoring parts of a page by the <! sphider_noindex > tags requires direct access to the page, because the tags need to be added (edited) to the page. A more flexible method, . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ menu[0-5] / 4.6 Indexing only parts of a page by <div id='abc'> If enabled in Admin settings, the values as defined in the list-file /include/common/divs_use.txt will be used to index only the content between <div . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ table[0-5] / 4.7 Ignore HTML elements defined by <tagname> . . </tagname> This option is foreseen to cooperate with the new HTML5 elements like section, nav, aside, hgroup, article, header, footer, etc. HTML elements . . .
. . .
contain a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */nav[0-5]/ Please keep in mind that element names placed in /include/common/elements_not.txt will be processed case-sensitive. 4.8 Index only HTML elements defined by <tagname> . . </tagname> This is the vice versa . . .
. . .
of all the duplicate content URLs: http://www.example.com/product.php?item=swedish-fish&category=gummy-candy http://www.example.com/product.php?item=swedish-fish&trackingid=1234&sessionid=5678 and Sphider-plus will understand that the duplicates all refer to the canonical URL: http://www.example.com/product.php?item=swedish-fish. The . . .
. . .
charset are supported by the ConvertCharset function and will be used to convert text into UTF-8 Unicode: WINDOWS windows-1250 - Central Europe windows-1251 - Cyrillic windows-1252 - Latin I windows-1253 - Greek windows-1254 - Turkish windows-1255 - Hebrew windows-1256 - Arabic windows-1257 - Baltic windows-1258 - Viet Nam cp874 - Thai - this . . .
. . .
- Baltic windows-1258 - Viet Nam cp874 - Thai - this file is also for DOS DOS cp437 - Latin US cp737 - Greek cp775 - BaltRim cp850 - Latin1 cp852 - Latin2 cp855 - Cyrylic cp857 - Turkish cp860 - Portuguese cp861 - Iceland cp862 - Hebrew cp863 - Canada cp864 - Arabic cp865 - Nordic cp866 - Cyrylic Russian (this is the one, used in IE . . .
. . .
IE Cyrillic (DOS) ) cp869 - Greek2 MAC (Apple) x-mac-cyrillic x-mac-greek x-mac-icelandic x-mac-ce x-mac-roman ISO iso-8859-1 iso-8859-2 iso-8859-3 iso-8859-4 iso-8859-5 iso-8859-6 iso-8859-7 iso-8859-8 iso-8859-9 iso-8859-10 iso-8859-11 iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 . . .
. . .
iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 cp500 cp856 cp875 cp1006 cp1026 koi8-r (Cyrillic) koi8-u (Cyrillic Ukrainian) nextstep us-ascii us-ascii-quotes DSP implementation for NeXT stdenc symbol zdingbat And specially for old Polish programs: mazovia This list is to be read . . .
. . .
Access denied; you need the RELOAD privilege. . . Top 10. Enable real-time output of logging data Up to version 1.5 of Sphider-plus, during index / re-index there was no printout available because: - Several servers, especially on Win32, buffer the output from the script until it terminates before transmitting the results to the browser. - . . .
. . .
is seen. - Some versions of Microsoft Internet Explorer only start to display the page after they have received 256 bytes of output. As progress was not presented during index / re- index procedure, waiting for results became a pain in the neck. Selectable in Admin setting together with the update interval (1 - 10 seconds), AJAX technology was . . .
. . .
characters in front of words Warning: This option should be used with special care and not be activated for non ISO-8859 charsets. Some special characters as part of the word ending might be erased by accidental. Top 13. Media search for images, audio streams and videos 13.1 Media indexing Index of media files is enabled by separated Admin . . .
. . .
and 'Found at' - Total clicks - Last clicked - Query input 'Indexed Image Thumbnails' presenting: - Thumbnail 150 x 100 pixel - Image details like title, filename size of original image, link- and thumb-id - Option to delete the thumbnail In order to open the media files all tables contain active links. Media results are also stored in . . .
. . .
Settings' menu of the admin backend: First option: For results of XML product feeds, present the text extract of 350 characters as part of product attributes. If this option is not activated, all products of the XML product feed will be presented in search results, and the hits of the query string will be highlighted in all involved products. . . .
. . .
content of the database tables (those with the same table prefix) will be destroyed by the restore procedure. 16.5 Copy & Move This section of the Database Management will present only those databases, which are correctly configured, assigned and do have a set of installed tables as described in chapter Definition and configuration. This . . .
. . .
results. Never the less to find these x results, the complete database had to been browsed. Starting with version 2.5, an additional clean option is offered as part of the Admin backend. Main advantage of this option is a significant reduction of the search time for any query, because the content of the db could be limited to offer only x . . .
. . .
<link>http://www.abc.de/images/warp.gif</link> <title>warp.gif</title> <x_size>635</x_size> <y_size>98</y_size> </media_result> <media_result> <num>2</num> <type>audio</type> <url>http://www.abc.de/index.php</url> . . .
. . .
the following content: {"query":"colour","ip":"::1","host_name":"guard_007-hoster","query_time":"2016-03-18 10:56:23 AM","consumed":0.016,"total_results":2,"num_of_results":2,"from":1,"to":2,"text_results":[{"num":1,"weight":"100.0","url":"http:\/\/www.english.le-piaggie.info\/Info_eng.pdf","title":" Info_eng","fulltxt":" . . . . . .
. . .
placed on the ground floor, today's big kitchen with the heating fireplace for open and close mode of . . . ","page_size":"5,661.6kb","domain_name":"www.english.le-piaggie.info"},{"num":2,"weight":"50.0","url":"http:\/\/www.english.le-piaggie.info\/html\/description.html","title":" Description","fulltxt":" . . . cable. Additional active . . .
. . .
D.O.C.G. as far as to the Pratomagno mountains 160 olive-trees Detached house, about 800 m to reach the . . . ","page_size":"26.4 kb","domain_name":"www.english.le-piaggie.info"}]} Top . . .


5a2 All required information. Introduction Release and Legal Info Installation Documentation Change Log [ Documentation Summary ] Preamble: The info presented here is valid only for the latest release of Sphider-plus. At present version 4.2024a published February 12, 2024 is the actual release. 1. Settings, customizing and statistics 2. Indexing . . .
. . .
2. Indexing 2.1 Various options 2.2 Allow other hosts in same domain 2.3 Word stemming 2.4 Periodical Re-indexing 2.5 Preferred indexing 2.6 Multithreaded indexing 2.7 Create thumbnails during index procedure 2.8 Prevent indexing of known malware and pishing pages 2.9 Follow and create sitemap files 2.10 Use private sitemap instead of global . . .
. . .
Sitemap file 3. Using the indexer from command line 3.1 All options 3.2 Multithreaded indexing 4. Keeping pages, words and files from being indexed 4.1 robots.txt 4.2 Must include / must not include string list 4.3 Ignoring links 4.4 Ignoring parts of a page by <! sphider_noindex > 4.5 Ignoring parts of a page by <div id='abc'> . . .
. . .
charset' 6. Search modes 6.1 Search with wildcards * 6.2 Strict search ! 6.3 Tolerant search 6.4 Link search 6.5 Media search 6.6 Search only in one domain 6.7 Search in categories 6.8 Greek language support 6.9 Block queries 7. Chronological order for result listing 7.1 Text result listing 7.2 Media result listing 8. PDF converter 9. Clean . . .
. . .
of logging data 11. Error messages and Debug mode 12. Delete secondary characters 13. Media search for images, audio streams and videos 13.1 Media indexing 13.2 Not supported media content 13.3 Search for media content 13.4 Statistics for media content 14. Feed support 14.1 XML product feeds 14.2 RDF, RSD, RSS and Atom feeds 15. Result . . .
. . .
Overview 16.2 Definition and configuration 16.3 Activate / disable database 16.4 Backup & Restore of databases 16.5 Copy & and Move 16.6 Enhancing functionality of multiple database support 17. Search in categories 17.1 Hierachical structure 17.2 Parallel structure 18. User suggested sites 19. Vulnerability protection 19.1 Prevent queries . . .
. . .
templates 22.2 Embed the search engine into existing HTML code 22.3 The different style sheet files 23. JSON, XML and RSS result output [ Documentation ] 1. Settings, customizing and statistics If you want to change settings, behavior and design of Sphider-plus, you can do so by means of the Admin interface. There is a wide range of . . .
. . .
interface. There is a wide range of settings foreseen for Sphider-plus. Separated into different submenus like: Sites: - Add Site - Index only the new - Re-index all - Re-index only preferred URLs - Erase Re-index (available also for individual URLs) - Import/export URL list - Approve sites - Banned domains Categories: - Add, edit, delete - . . .
. . .
URL list - Approve sites - Banned domains Categories: - Add, edit, delete - Create new subcategory under Index: - Basic indexing options - Advanced options Clean: - Clean keywords not associated with any link - Clean links not associated with any site - Clean Category table not associated with any site - Clean Media links - Clear Temp table . . .
. . .
- Clear Search log - Clear 'Most Popular Page Links' log - Clear 'Most Popular Media Links' log - Clear Spider log, separate and bulk delete - Clear Thumbnail images, separate and bulk delete - Clear Text cache - Clear Media cache - Clear IDS log file - Clear flood attempts log file - Clear all entries in addurl or banned table - Truncate all . . .
. . .
flood attempts log file - Clear all entries in addurl or banned table - Truncate all tables in database Settings: - General Settings - Index Log Settings - Spider Settings - Search Settings - Order of Result listing - Suggest Options - Page Indexing Weights Database: - Configure up to 5 databases with unlimited number of table sets - Activate . . .
. . .
Database: - Configure up to 5 databases with unlimited number of table sets - Activate separately for 'Admin', 'Search' user and 'Suggest URL user' - Backup / Restore - Copy / Move - Optimize Templates: In order to enable customer's integration of Sphider-plus into existing sites, HTML templates are prepared for Search form Text result . . .
. . .
Search form Text result listing Media result listing Most popular queries etc. Three different designs are offered, which may be selected in submenu 'Settings'. If the layout does not fit the design of your site (which is normal), you may create new designs and modify the appropriate file /templates/My_template/adminstyle.css . . .
. . .
the appropriate file /templates/My_template/adminstyle.css /templates/My_template/userstyle.css Statistics output: - Top keywords (Top 50 with hit counter). - All indexed thumbnails w 3e80 ith ID3 and EXIF info. - Larges pages offering link URL and file size. - Most Popular Searches for text links offering: Link addr., total clicks, last . . .
. . .
Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Searches for media links offering: Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Links (click counter). - Search log offering: Query, Results, Queried at, Time taken, User IP, Country, Host name (Latest100) - Index log offering: . . .
. . .
Country, Host name (Latest100) - Index log offering: File-name, index date and delete option - sitemap log offering: sitemap.xml output sitemap list offering file/page suffixes - IDS log offering: IP, host, query, impact, involved tags, date and time of intrusion. - Flood attempts log offering: IP, query, date and time of flood attempt. - Auto . . .
. . .
attempts log offering: IP, query, date and time of flood attempt. - Auto Re-index log file - Server info offering: Server software, environment, MySQL, PDF-converter, image functions, php.ini file PHP integration, PHP security info. Each item holding lists of details. All text links, media links and thumbnails are active linked. As stated in . . .
. . .
individual Re-indexing, the periodical Re-indexer could be started and aborted in the "Options" menu of each site. 2.5 Preferred Re-indexing Each new URL added to the Admin backend, could be supplied with a priority level. This level will be used by the option 'Re-index only preferred sites'. Level 1 will be interpreted as most important, while . . .
. . .
less all index results will be stored in log files in sub folder /admin/log/ The names of the log files look like: db2_100524-21.47.56_1.html (log file of first thread) db2_100524-21.48.12_2.html (log file of second thread) and is build by the following items: db2 - Number of database. 100524 - Date (May 24, 2010) 21.47.56 - Time when this thread . . .
. . .
spider.php -new1 php spider.php -new2 etc. The IDs will be added to the name of the corresponding log files like: db2_100524-21.47.56_ID1.html (log file of first thread) db2_100524-21.48.12_ID2.html (log file of second thread) IDs could be defined by personal requirements, but the limitations for file names with respect to the OS should be taken . . .
. . .
but will not erase the content of all the other tables. So the check whether the content of a page has changed (MD5sum) is still available for a fast re-index procedure. Once prepared, multithreaded re-indexing could be invoked by starting several threads and adding individual IDs to the option parameter like: php spider.php -erased1 php . . .
. . .
4.4 Ignoring parts of a page Sphider-plus includes an option to exclude parts of pages from being indexed. Thi 775e s can for example be used to prevent search result flooding when certain keywords appear on certain part in most pages (like a header, footer or a menu). Any part of a page between <! sphider_noindex > and <! . . .
. . .
<! sphider_noindex > and <! /sphider_noindex > tags is not indexed, however links in it are followed. 4.5 Ignoring parts of a page by <div id='abc'> Ignoring parts of a page by the <! sphider_noindex > tags requires direct access to the page, because the tags need to be added (edited) to the page. A more flexible method, . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ menu[0-5] / 4.6 Indexing only parts of a page by <div id='abc'> If enabled in Admin settings, the values as defined in the list-file /include/common/divs_use.txt will be used to index only the content between <div . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ table[0-5] / 4.7 Ignore HTML elements defined by <tagname> . . </tagname> This option is foreseen to cooperate with the new HTML5 elements like section, nav, aside, hgroup, article, header, footer, etc. HTML elements . . .
. . .
contain a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */nav[0-5]/ Please keep in mind that element names placed in /include/common/elements_not.txt will be processed case-sensitive. 4.8 Index only HTML elements defined by <tagname> . . </tagname> This is the vice versa . . .
. . .
of all the duplicate content URLs: http://www.example.com/product.php?item=swedish-fish&category=gummy-candy http://www.example.com/product.php?item=swedish-fish&trackingid=1234&sessionid=5678 and Sphider-plus will understand that the duplicates all refer to the canonical URL: http://www.example.com/product.php?item=swedish-fish. The . . .
. . .
charset are supported by the ConvertCharset function and will be used to convert text into UTF-8 Unicode: WINDOWS windows-1250 - Central Europe windows-1251 - Cyrillic windows-1252 - Latin I windows-1253 - Greek windows-1254 - Turkish windows-1255 - Hebrew windows-1256 - Arabic windows-1257 - Baltic windows-1258 - Viet Nam cp874 - Thai - this . . .
. . .
- Baltic windows-1258 - Viet Nam cp874 - Thai - this file is also for DOS DOS cp437 - Latin US cp737 - Greek cp775 - BaltRim cp850 - Latin1 cp852 - Latin2 cp855 - Cyrylic cp857 - Turkish cp860 - Portuguese cp861 - Iceland cp862 - Hebrew cp863 - Canada cp864 - Arabic cp865 - Nordic cp866 - Cyrylic Russian (this is the one, used in IE . . .
. . .
IE Cyrillic (DOS) ) cp869 - Greek2 MAC (Apple) x-mac-cyrillic x-mac-greek x-mac-icelandic x-mac-ce x-mac-roman ISO iso-8859-1 iso-8859-2 iso-8859-3 iso-8859-4 iso-8859-5 iso-8859-6 iso-8859-7 iso-8859-8 iso-8859-9 iso-8859-10 iso-8859-11 iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 . . .
. . .
iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 cp500 cp856 cp875 cp1006 cp1026 koi8-r (Cyrillic) koi8-u (Cyrillic Ukrainian) nextstep us-ascii us-ascii-quotes DSP implementation for NeXT stdenc symbol zdingbat And specially for old Polish programs: mazovia This list is to be read . . .
. . .
Access denied; you need the RELOAD privilege. . . Top 10. Enable real-time output of logging data Up to version 1.5 of Sphider-plus, during index / re-index there was no printout available because: - Several servers, especially on Win32, buffer the output from the script until it terminates before transmitting the results to the browser. - . . .
. . .
is seen. - Some versions of Microsoft Internet Explorer only start to display the page after they have received 256 bytes of output. As progress was not presented during index / re- index procedure, waiting for results became a pain in the neck. Selectable in Admin setting together with the update interval (1 - 10 seconds), AJAX technology was . . .
. . .
characters in front of words Warning: This option should be used with special care and not be activated for non ISO-8859 charsets. Some special characters as part of the word ending might be erased by accidental. Top 13. Media search for images, audio streams and videos 13.1 Media indexing Index of media files is enabled by separated Admin . . .
. . .
and 'Found at' - Total clicks - Last clicked - Query input 'Indexed Image Thumbnails' presenting: - Thumbnail 150 x 100 pixel - Image details like title, filename size of original image, link- and thumb-id - Option to delete the thumbnail In order to open the media files all tables contain active links. Media results are also stored in . . .
. . .
Settings' menu of the admin backend: First option: For results of XML product feeds, present the text extract of 350 characters as part of product attributes. If this option is not activated, all products of the XML product feed will be presented in search results, and the hits of the query string will be highlighted in all involved products. . . .
. . .
content of the database tables (those with the same table prefix) will be destroyed by the restore procedure. 16.5 Copy & Move This section of the Database Management will present only those databases, which are correctly configured, assigned and do have a set of installed tables as described in chapter Definition and configuration. This . . .
. . .
results. Never the less to find these x results, the complete database had to been browsed. Starting with version 2.5, an additional clean option is offered as part of the Admin backend. Main advantage of this option is a significant reduction of the search time for any query, because the content of the db could be limited to offer only x . . .
. . .
<link>http://www.abc.de/images/warp.gif</link> <title>warp.gif</title> <x_size>635</x_size> <y_size>98</y_size> </media_result> <media_result> <num>2</num> <type>audio</type> <url>http://www.abc.de/index.php</url> . . .
. . .
the following content: {"query":"colour","ip":"::1","host_name":"guard_007-hoster","query_time":"2016-03-18 10:56:23 AM","consumed":0.016,"total_results":2,"num_of_results":2,"from":1,"to":2,"text_results":[{"num":1,"weight":"100.0","url":"http:\/\/www.english.le-piaggie.info\/Info_eng.pdf","title":" Info_eng","fulltxt":" . . . . . .
. . .
placed on the ground floor, today's big kitchen with the heating fireplace for open and close mode of . . . ","page_size":"5,661.6kb","domain_name":"www.english.le-piaggie.info"},{"num":2,"weight":"50.0","url":"http:\/\/www.english.le-piaggie.info\/html\/description.html","title":" Description","fulltxt":" . . . cable. Additional active . . .
. . .
D.O.C.G. as far as to the Pratomagno mountains 160 olive-trees Detached house, about 800 m to reach the . . . ","page_size":"26.4 kb","domain_name":"www.english.le-piaggie.info"}]} Top . . .


5a2 All required information. Introduction Release and Legal Info Installation Documentation Change Log [ Documentation Summary ] Preamble: The info presented here is valid only for the latest release of Sphider-plus. At present version 4.2024a published February 12, 2024 is the actual release. 1. Settings, customizing and statistics 2. Indexing . . .
. . .
2. Indexing 2.1 Various options 2.2 Allow other hosts in same domain 2.3 Word stemming 2.4 Periodical Re-indexing 2.5 Preferred indexing 2.6 Multithreaded indexing 2.7 Create thumbnails during index procedure 2.8 Prevent indexing of known malware and pishing pages 2.9 Follow and create sitemap files 2.10 Use private sitemap instead of global . . .
. . .
Sitemap file 3. Using the indexer from command line 3.1 All options 3.2 Multithreaded indexing 4. Keeping pages, words and files from being indexed 4.1 robots.txt 4.2 Must include / must not include string list 4.3 Ignoring links 4.4 Ignoring parts of a page by <! sphider_noindex > 4.5 Ignoring parts of a page by <div id='abc'> . . .
. . .
charset' 6. Search modes 6.1 Search with wildcards * 6.2 Strict search ! 6.3 Tolerant search 6.4 Link search 6.5 Media search 6.6 Search only in one domain 6.7 Search in categories 6.8 Greek language support 6.9 Block queries 7. Chronological order for result listing 7.1 Text result listing 7.2 Media result listing 8. PDF converter 9. Clean . . .
. . .
of logging data 11. Error messages and Debug mode 12. Delete secondary characters 13. Media search for images, audio streams and videos 13.1 Media indexing 13.2 Not supported media content 13.3 Search for media content 13.4 Statistics for media content 14. Feed support 14.1 XML product feeds 14.2 RDF, RSD, RSS and Atom feeds 15. Result . . .
. . .
Overview 16.2 Definition and configuration 16.3 Activate / disable database 16.4 Backup & Restore of databases 16.5 Copy & and Move 16.6 Enhancing functionality of multiple database support 17. Search in categories 17.1 Hierachical structure 17.2 Parallel structure 18. User suggested sites 19. Vulnerability protection 19.1 Prevent queries . . .
. . .
templates 22.2 Embed the search engine into existing HTML code 22.3 The different style sheet files 23. JSON, XML and RSS result output [ Documentation ] 1. Settings, customizing and statistics If you want to change settings, behavior and design of Sphider-plus, you can do so by means of the Admin interface. There is a wide range of . . .
. . .
interface. There is a wide range of settings foreseen for Sphider-plus. Separated into different submenus like: Sites: - Add Site - Index only the new - Re-index all - Re-index only preferred URLs - Erase Re-index (available also for individual URLs) - Import/export URL list - Approve sites - Banned domains Categories: - Add, edit, delete - . . .
. . .
URL list - Approve sites - Banned domains Categories: - Add, edit, delete - Create new subcategory under Index: - Basic indexing options - Advanced options Clean: - Clean keywords not associated with any link - Clean links not associated with any site - Clean Category table not associated with any site - Clean Media links - Clear Temp table . . .
. . .
- Clear Search log - Clear 'Most Popular Page Links' log - Clear 'Most Popular Media Links' log - Clear Spider log, separate and bulk delete - Clear Thumbnail images, separate and bulk delete - Clear Text cache - Clear Media cache - Clear IDS log file - Clear flood attempts log file - Clear all entries in addurl or banned table - Truncate all . . .
. . .
flood attempts log file - Clear all entries in addurl or banned table - Truncate all tables in database Settings: - General Settings - Index Log Settings - Spider Settings - Search Settings - Order of Result listing - Suggest Options - Page Indexing Weights Database: - Configure up to 5 databases with unlimited number of table sets - Activate . . .
. . .
Database: - Configure up to 5 databases with unlimited number of table sets - Activate separately for 'Admin', 'Search' user and 'Suggest URL user' - Backup / Restore - Copy / Move - Optimize Templates: In order to enable customer's integration of Sphider-plus into existing sites, HTML templates are prepared for Search form Text result . . .
. . .
Search form Text result listing Media result listing Most popular queries etc. Three different designs are offered, which may be selected in submenu 'Settings'. If the layout does not fit the design of your site (which is normal), you may create new designs and modify the appropriate file /templates/My_template/adminstyle.css . . .
. . .
the appropriate file /templates/My_template/adminstyle.css /templates/My_template/userstyle.css Statistics output: - Top keywords (Top 50 with hit counter). - All indexed thumbnails w 3e80 ith ID3 and EXIF info. - Larges pages offering link URL and file size. - Most Popular Searches for text links offering: Link addr., total clicks, last . . .
. . .
Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Searches for media links offering: Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Links (click counter). - Search log offering: Query, Results, Queried at, Time taken, User IP, Country, Host name (Latest100) - Index log offering: . . .
. . .
Country, Host name (Latest100) - Index log offering: File-name, index date and delete option - sitemap log offering: sitemap.xml output sitemap list offering file/page suffixes - IDS log offering: IP, host, query, impact, involved tags, date and time of intrusion. - Flood attempts log offering: IP, query, date and time of flood attempt. - Auto . . .
. . .
attempts log offering: IP, query, date and time of flood attempt. - Auto Re-index log file - Server info offering: Server software, environment, MySQL, PDF-converter, image functions, php.ini file PHP integration, PHP security info. Each item holding lists of details. All text links, media links and thumbnails are active linked. As stated in . . .
. . .
individual Re-indexing, the periodical Re-indexer could be started and aborted in the "Options" menu of each site. 2.5 Preferred Re-indexing Each new URL added to the Admin backend, could be supplied with a priority level. This level will be used by the option 'Re-index only preferred sites'. Level 1 will be interpreted as most important, while . . .
. . .
less all index results will be stored in log files in sub folder /admin/log/ The names of the log files look like: db2_100524-21.47.56_1.html (log file of first thread) db2_100524-21.48.12_2.html (log file of second thread) and is build by the following items: db2 - Number of database. 100524 - Date (May 24, 2010) 21.47.56 - Time when this thread . . .
. . .
spider.php -new1 php spider.php -new2 etc. The IDs will be added to the name of the corresponding log files like: db2_100524-21.47.56_ID1.html (log file of first thread) db2_100524-21.48.12_ID2.html (log file of second thread) IDs could be defined by personal requirements, but the limitations for file names with respect to the OS should be taken . . .
. . .
but will not erase the content of all the other tables. So the check whether the content of a page has changed (MD5sum) is still available for a fast re-index procedure. Once prepared, multithreaded re-indexing could be invoked by starting several threads and adding individual IDs to the option parameter like: php spider.php -erased1 php . . .
. . .
4.4 Ignoring parts of a page Sphider-plus includes an option to exclude parts of pages from being indexed. Thi 775e s can for example be used to prevent search result flooding when certain keywords appear on certain part in most pages (like a header, footer or a menu). Any part of a page between <! sphider_noindex > and <! . . .
. . .
<! sphider_noindex > and <! /sphider_noindex > tags is not indexed, however links in it are followed. 4.5 Ignoring parts of a page by <div id='abc'> Ignoring parts of a page by the <! sphider_noindex > tags requires direct access to the page, because the tags need to be added (edited) to the page. A more flexible method, . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ menu[0-5] / 4.6 Indexing only parts of a page by <div id='abc'> If enabled in Admin settings, the values as defined in the list-file /include/common/divs_use.txt will be used to index only the content between <div . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ table[0-5] / 4.7 Ignore HTML elements defined by <tagname> . . </tagname> This option is foreseen to cooperate with the new HTML5 elements like section, nav, aside, hgroup, article, header, footer, etc. HTML elements . . .
. . .
contain a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */nav[0-5]/ Please keep in mind that element names placed in /include/common/elements_not.txt will be processed case-sensitive. 4.8 Index only HTML elements defined by <tagname> . . </tagname> This is the vice versa . . .
. . .
of all the duplicate content URLs: http://www.example.com/product.php?item=swedish-fish&category=gummy-candy http://www.example.com/product.php?item=swedish-fish&trackingid=1234&sessionid=5678 and Sphider-plus will understand that the duplicates all refer to the canonical URL: http://www.example.com/product.php?item=swedish-fish. The . . .
. . .
charset are supported by the ConvertCharset function and will be used to convert text into UTF-8 Unicode: WINDOWS windows-1250 - Central Europe windows-1251 - Cyrillic windows-1252 - Latin I windows-1253 - Greek windows-1254 - Turkish windows-1255 - Hebrew windows-1256 - Arabic windows-1257 - Baltic windows-1258 - Viet Nam cp874 - Thai - this . . .
. . .
- Baltic windows-1258 - Viet Nam cp874 - Thai - this file is also for DOS DOS cp437 - Latin US cp737 - Greek cp775 - BaltRim cp850 - Latin1 cp852 - Latin2 cp855 - Cyrylic cp857 - Turkish cp860 - Portuguese cp861 - Iceland cp862 - Hebrew cp863 - Canada cp864 - Arabic cp865 - Nordic cp866 - Cyrylic Russian (this is the one, used in IE . . .
. . .
IE Cyrillic (DOS) ) cp869 - Greek2 MAC (Apple) x-mac-cyrillic x-mac-greek x-mac-icelandic x-mac-ce x-mac-roman ISO iso-8859-1 iso-8859-2 iso-8859-3 iso-8859-4 iso-8859-5 iso-8859-6 iso-8859-7 iso-8859-8 iso-8859-9 iso-8859-10 iso-8859-11 iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 . . .
. . .
iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 cp500 cp856 cp875 cp1006 cp1026 koi8-r (Cyrillic) koi8-u (Cyrillic Ukrainian) nextstep us-ascii us-ascii-quotes DSP implementation for NeXT stdenc symbol zdingbat And specially for old Polish programs: mazovia This list is to be read . . .
. . .
Access denied; you need the RELOAD privilege. . . Top 10. Enable real-time output of logging data Up to version 1.5 of Sphider-plus, during index / re-index there was no printout available because: - Several servers, especially on Win32, buffer the output from the script until it terminates before transmitting the results to the browser. - . . .
. . .
is seen. - Some versions of Microsoft Internet Explorer only start to display the page after they have received 256 bytes of output. As progress was not presented during index / re- index procedure, waiting for results became a pain in the neck. Selectable in Admin setting together with the update interval (1 - 10 seconds), AJAX technology was . . .
. . .
characters in front of words Warning: This option should be used with special care and not be activated for non ISO-8859 charsets. Some special characters as part of the word ending might be erased by accidental. Top 13. Media search for images, audio streams and videos 13.1 Media indexing Index of media files is enabled by separated Admin . . .
. . .
and 'Found at' - Total clicks - Last clicked - Query input 'Indexed Image Thumbnails' presenting: - Thumbnail 150 x 100 pixel - Image details like title, filename size of original image, link- and thumb-id - Option to delete the thumbnail In order to open the media files all tables contain active links. Media results are also stored in . . .
. . .
Settings' menu of the admin backend: First option: For results of XML product feeds, present the text extract of 350 characters as part of product attributes. If this option is not activated, all products of the XML product feed will be presented in search results, and the hits of the query string will be highlighted in all involved products. . . .
. . .
content of the database tables (those with the same table prefix) will be destroyed by the restore procedure. 16.5 Copy & Move This section of the Database Management will present only those databases, which are correctly configured, assigned and do have a set of installed tables as described in chapter Definition and configuration. This . . .
. . .
results. Never the less to find these x results, the complete database had to been browsed. Starting with version 2.5, an additional clean option is offered as part of the Admin backend. Main advantage of this option is a significant reduction of the search time for any query, because the content of the db could be limited to offer only x . . .
. . .
<link>http://www.abc.de/images/warp.gif</link> <title>warp.gif</title> <x_size>635</x_size> <y_size>98</y_size> </media_result> <media_result> <num>2</num> <type>audio</type> <url>http://www.abc.de/index.php</url> . . .
. . .
the following content: {"query":"colour","ip":"::1","host_name":"guard_007-hoster","query_time":"2016-03-18 10:56:23 AM","consumed":0.016,"total_results":2,"num_of_results":2,"from":1,"to":2,"text_results":[{"num":1,"weight":"100.0","url":"http:\/\/www.english.le-piaggie.info\/Info_eng.pdf","title":" Info_eng","fulltxt":" . . . . . .
. . .
placed on the ground floor, today's big kitchen with the heating fireplace for open and close mode of . . . ","page_size":"5,661.6kb","domain_name":"www.english.le-piaggie.info"},{"num":2,"weight":"50.0","url":"http:\/\/www.english.le-piaggie.info\/html\/description.html","title":" Description","fulltxt":" . . . cable. Additional active . . .
. . .
D.O.C.G. as far as to the Pratomagno mountains 160 olive-trees Detached house, about 800 m to reach the . . . ","page_size":"26.4 kb","domain_name":"www.english.le-piaggie.info"}]} Top . . .


5a2 All required information. Introduction Release and Legal Info Installation Documentation Change Log [ Documentation Summary ] Preamble: The info presented here is valid only for the latest release of Sphider-plus. At present version 4.2024a published February 12, 2024 is the actual release. 1. Settings, customizing and statistics 2. Indexing . . .
. . .
2. Indexing 2.1 Various options 2.2 Allow other hosts in same domain 2.3 Word stemming 2.4 Periodical Re-indexing 2.5 Preferred indexing 2.6 Multithreaded indexing 2.7 Create thumbnails during index procedure 2.8 Prevent indexing of known malware and pishing pages 2.9 Follow and create sitemap files 2.10 Use private sitemap instead of global . . .
. . .
Sitemap file 3. Using the indexer from command line 3.1 All options 3.2 Multithreaded indexing 4. Keeping pages, words and files from being indexed 4.1 robots.txt 4.2 Must include / must not include string list 4.3 Ignoring links 4.4 Ignoring parts of a page by <! sphider_noindex > 4.5 Ignoring parts of a page by <div id='abc'> . . .
. . .
charset' 6. Search modes 6.1 Search with wildcards * 6.2 Strict search ! 6.3 Tolerant search 6.4 Link search 6.5 Media search 6.6 Search only in one domain 6.7 Search in categories 6.8 Greek language support 6.9 Block queries 7. Chronological order for result listing 7.1 Text result listing 7.2 Media result listing 8. PDF converter 9. Clean . . .
. . .
of logging data 11. Error messages and Debug mode 12. Delete secondary characters 13. Media search for images, audio streams and videos 13.1 Media indexing 13.2 Not supported media content 13.3 Search for media content 13.4 Statistics for media content 14. Feed support 14.1 XML product feeds 14.2 RDF, RSD, RSS and Atom feeds 15. Result . . .
. . .
Overview 16.2 Definition and configuration 16.3 Activate / disable database 16.4 Backup & Restore of databases 16.5 Copy & and Move 16.6 Enhancing functionality of multiple database support 17. Search in categories 17.1 Hierachical structure 17.2 Parallel structure 18. User suggested sites 19. Vulnerability protection 19.1 Prevent queries . . .
. . .
templates 22.2 Embed the search engine into existing HTML code 22.3 The different style sheet files 23. JSON, XML and RSS result output [ Documentation ] 1. Settings, customizing and statistics If you want to change settings, behavior and design of Sphider-plus, you can do so by means of the Admin interface. There is a wide range of . . .
. . .
interface. There is a wide range of settings foreseen for Sphider-plus. Separated into different submenus like: Sites: - Add Site - Index only the new - Re-index all - Re-index only preferred URLs - Erase Re-index (available also for individual URLs) - Import/export URL list - Approve sites - Banned domains Categories: - Add, edit, delete - . . .
. . .
URL list - Approve sites - Banned domains Categories: - Add, edit, delete - Create new subcategory under Index: - Basic indexing options - Advanced options Clean: - Clean keywords not associated with any link - Clean links not associated with any site - Clean Category table not associated with any site - Clean Media links - Clear Temp table . . .
. . .
- Clear Search log - Clear 'Most Popular Page Links' log - Clear 'Most Popular Media Links' log - Clear Spider log, separate and bulk delete - Clear Thumbnail images, separate and bulk delete - Clear Text cache - Clear Media cache - Clear IDS log file - Clear flood attempts log file - Clear all entries in addurl or banned table - Truncate all . . .
. . .
flood attempts log file - Clear all entries in addurl or banned table - Truncate all tables in database Settings: - General Settings - Index Log Settings - Spider Settings - Search Settings - Order of Result listing - Suggest Options - Page Indexing Weights Database: - Configure up to 5 databases with unlimited number of table sets - Activate . . .
. . .
Database: - Configure up to 5 databases with unlimited number of table sets - Activate separately for 'Admin', 'Search' user and 'Suggest URL user' - Backup / Restore - Copy / Move - Optimize Templates: In order to enable customer's integration of Sphider-plus into existing sites, HTML templates are prepared for Search form Text result . . .
. . .
Search form Text result listing Media result listing Most popular queries etc. Three different designs are offered, which may be selected in submenu 'Settings'. If the layout does not fit the design of your site (which is normal), you may create new designs and modify the appropriate file /templates/My_template/adminstyle.css . . .
. . .
the appropriate file /templates/My_template/adminstyle.css /templates/My_template/userstyle.css Statistics output: - Top keywords (Top 50 with hit counter). - All indexed thumbnails w 3e80 ith ID3 and EXIF info. - Larges pages offering link URL and file size. - Most Popular Searches for text links offering: Link addr., total clicks, last . . .
. . .
Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Searches for media links offering: Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Links (click counter). - Search log offering: Query, Results, Queried at, Time taken, User IP, Country, Host name (Latest100) - Index log offering: . . .
. . .
Country, Host name (Latest100) - Index log offering: File-name, index date and delete option - sitemap log offering: sitemap.xml output sitemap list offering file/page suffixes - IDS log offering: IP, host, query, impact, involved tags, date and time of intrusion. - Flood attempts log offering: IP, query, date and time of flood attempt. - Auto . . .
. . .
attempts log offering: IP, query, date and time of flood attempt. - Auto Re-index log file - Server info offering: Server software, environment, MySQL, PDF-converter, image functions, php.ini file PHP integration, PHP security info. Each item holding lists of details. All text links, media links and thumbnails are active linked. As stated in . . .
. . .
individual Re-indexing, the periodical Re-indexer could be started and aborted in the "Options" menu of each site. 2.5 Preferred Re-indexing Each new URL added to the Admin backend, could be supplied with a priority level. This level will be used by the option 'Re-index only preferred sites'. Level 1 will be interpreted as most important, while . . .
. . .
less all index results will be stored in log files in sub folder /admin/log/ The names of the log files look like: db2_100524-21.47.56_1.html (log file of first thread) db2_100524-21.48.12_2.html (log file of second thread) and is build by the following items: db2 - Number of database. 100524 - Date (May 24, 2010) 21.47.56 - Time when this thread . . .
. . .
spider.php -new1 php spider.php -new2 etc. The IDs will be added to the name of the corresponding log files like: db2_100524-21.47.56_ID1.html (log file of first thread) db2_100524-21.48.12_ID2.html (log file of second thread) IDs could be defined by personal requirements, but the limitations for file names with respect to the OS should be taken . . .
. . .
but will not erase the content of all the other tables. So the check whether the content of a page has changed (MD5sum) is still available for a fast re-index procedure. Once prepared, multithreaded re-indexing could be invoked by starting several threads and adding individual IDs to the option parameter like: php spider.php -erased1 php . . .
. . .
4.4 Ignoring parts of a page Sphider-plus includes an option to exclude parts of pages from being indexed. Thi 775e s can for example be used to prevent search result flooding when certain keywords appear on certain part in most pages (like a header, footer or a menu). Any part of a page between <! sphider_noindex > and <! . . .
. . .
<! sphider_noindex > and <! /sphider_noindex > tags is not indexed, however links in it are followed. 4.5 Ignoring parts of a page by <div id='abc'> Ignoring parts of a page by the <! sphider_noindex > tags requires direct access to the page, because the tags need to be added (edited) to the page. A more flexible method, . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ menu[0-5] / 4.6 Indexing only parts of a page by <div id='abc'> If enabled in Admin settings, the values as defined in the list-file /include/common/divs_use.txt will be used to index only the content between <div . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ table[0-5] / 4.7 Ignore HTML elements defined by <tagname> . . </tagname> This option is foreseen to cooperate with the new HTML5 elements like section, nav, aside, hgroup, article, header, footer, etc. HTML elements . . .
. . .
contain a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */nav[0-5]/ Please keep in mind that element names placed in /include/common/elements_not.txt will be processed case-sensitive. 4.8 Index only HTML elements defined by <tagname> . . </tagname> This is the vice versa . . .
. . .
of all the duplicate content URLs: http://www.example.com/product.php?item=swedish-fish&category=gummy-candy http://www.example.com/product.php?item=swedish-fish&trackingid=1234&sessionid=5678 and Sphider-plus will understand that the duplicates all refer to the canonical URL: http://www.example.com/product.php?item=swedish-fish. The . . .
. . .
charset are supported by the ConvertCharset function and will be used to convert text into UTF-8 Unicode: WINDOWS windows-1250 - Central Europe windows-1251 - Cyrillic windows-1252 - Latin I windows-1253 - Greek windows-1254 - Turkish windows-1255 - Hebrew windows-1256 - Arabic windows-1257 - Baltic windows-1258 - Viet Nam cp874 - Thai - this . . .
. . .
- Baltic windows-1258 - Viet Nam cp874 - Thai - this file is also for DOS DOS cp437 - Latin US cp737 - Greek cp775 - BaltRim cp850 - Latin1 cp852 - Latin2 cp855 - Cyrylic cp857 - Turkish cp860 - Portuguese cp861 - Iceland cp862 - Hebrew cp863 - Canada cp864 - Arabic cp865 - Nordic cp866 - Cyrylic Russian (this is the one, used in IE . . .
. . .
IE Cyrillic (DOS) ) cp869 - Greek2 MAC (Apple) x-mac-cyrillic x-mac-greek x-mac-icelandic x-mac-ce x-mac-roman ISO iso-8859-1 iso-8859-2 iso-8859-3 iso-8859-4 iso-8859-5 iso-8859-6 iso-8859-7 iso-8859-8 iso-8859-9 iso-8859-10 iso-8859-11 iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 . . .
. . .
iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 cp500 cp856 cp875 cp1006 cp1026 koi8-r (Cyrillic) koi8-u (Cyrillic Ukrainian) nextstep us-ascii us-ascii-quotes DSP implementation for NeXT stdenc symbol zdingbat And specially for old Polish programs: mazovia This list is to be read . . .
. . .
Access denied; you need the RELOAD privilege. . . Top 10. Enable real-time output of logging data Up to version 1.5 of Sphider-plus, during index / re-index there was no printout available because: - Several servers, especially on Win32, buffer the output from the script until it terminates before transmitting the results to the browser. - . . .
. . .
is seen. - Some versions of Microsoft Internet Explorer only start to display the page after they have received 256 bytes of output. As progress was not presented during index / re- index procedure, waiting for results became a pain in the neck. Selectable in Admin setting together with the update interval (1 - 10 seconds), AJAX technology was . . .
. . .
characters in front of words Warning: This option should be used with special care and not be activated for non ISO-8859 charsets. Some special characters as part of the word ending might be erased by accidental. Top 13. Media search for images, audio streams and videos 13.1 Media indexing Index of media files is enabled by separated Admin . . .
. . .
and 'Found at' - Total clicks - Last clicked - Query input 'Indexed Image Thumbnails' presenting: - Thumbnail 150 x 100 pixel - Image details like title, filename size of original image, link- and thumb-id - Option to delete the thumbnail In order to open the media files all tables contain active links. Media results are also stored in . . .
. . .
Settings' menu of the admin backend: First option: For results of XML product feeds, present the text extract of 350 characters as part of product attributes. If this option is not activated, all products of the XML product feed will be presented in search results, and the hits of the query string will be highlighted in all involved products. . . .
. . .
content of the database tables (those with the same table prefix) will be destroyed by the restore procedure. 16.5 Copy & Move This section of the Database Management will present only those databases, which are correctly configured, assigned and do have a set of installed tables as described in chapter Definition and configuration. This . . .
. . .
results. Never the less to find these x results, the complete database had to been browsed. Starting with version 2.5, an additional clean option is offered as part of the Admin backend. Main advantage of this option is a significant reduction of the search time for any query, because the content of the db could be limited to offer only x . . .
. . .
<link>http://www.abc.de/images/warp.gif</link> <title>warp.gif</title> <x_size>635</x_size> <y_size>98</y_size> </media_result> <media_result> <num>2</num> <type>audio</type> <url>http://www.abc.de/index.php</url> . . .
. . .
the following content: {"query":"colour","ip":"::1","host_name":"guard_007-hoster","query_time":"2016-03-18 10:56:23 AM","consumed":0.016,"total_results":2,"num_of_results":2,"from":1,"to":2,"text_results":[{"num":1,"weight":"100.0","url":"http:\/\/www.english.le-piaggie.info\/Info_eng.pdf","title":" Info_eng","fulltxt":" . . . . . .
. . .
placed on the ground floor, today's big kitchen with the heating fireplace for open and close mode of . . . ","page_size":"5,661.6kb","domain_name":"www.english.le-piaggie.info"},{"num":2,"weight":"50.0","url":"http:\/\/www.english.le-piaggie.info\/html\/description.html","title":" Description","fulltxt":" . . . cable. Additional active . . .
. . .
D.O.C.G. as far as to the Pratomagno mountains 160 olive-trees Detached house, about 800 m to reach the . . . ","page_size":"26.4 kb","domain_name":"www.english.le-piaggie.info"}]} Top . . .


5a2 All required information. Introduction Release and Legal Info Installation Documentation Change Log [ Documentation Summary ] Preamble: The info presented here is valid only for the latest release of Sphider-plus. At present version 4.2024a published February 12, 2024 is the actual release. 1. Settings, customizing and statistics 2. Indexing . . .
. . .
2. Indexing 2.1 Various options 2.2 Allow other hosts in same domain 2.3 Word stemming 2.4 Periodical Re-indexing 2.5 Preferred indexing 2.6 Multithreaded indexing 2.7 Create thumbnails during index procedure 2.8 Prevent indexing of known malware and pishing pages 2.9 Follow and create sitemap files 2.10 Use private sitemap instead of global . . .
. . .
Sitemap file 3. Using the indexer from command line 3.1 All options 3.2 Multithreaded indexing 4. Keeping pages, words and files from being indexed 4.1 robots.txt 4.2 Must include / must not include string list 4.3 Ignoring links 4.4 Ignoring parts of a page by <! sphider_noindex > 4.5 Ignoring parts of a page by <div id='abc'> . . .
. . .
charset' 6. Search modes 6.1 Search with wildcards * 6.2 Strict search ! 6.3 Tolerant search 6.4 Link search 6.5 Media search 6.6 Search only in one domain 6.7 Search in categories 6.8 Greek language support 6.9 Block queries 7. Chronological order for result listing 7.1 Text result listing 7.2 Media result listing 8. PDF converter 9. Clean . . .
. . .
of logging data 11. Error messages and Debug mode 12. Delete secondary characters 13. Media search for images, audio streams and videos 13.1 Media indexing 13.2 Not supported media content 13.3 Search for media content 13.4 Statistics for media content 14. Feed support 14.1 XML product feeds 14.2 RDF, RSD, RSS and Atom feeds 15. Result . . .
. . .
Overview 16.2 Definition and configuration 16.3 Activate / disable database 16.4 Backup & Restore of databases 16.5 Copy & and Move 16.6 Enhancing functionality of multiple database support 17. Search in categories 17.1 Hierachical structure 17.2 Parallel structure 18. User suggested sites 19. Vulnerability protection 19.1 Prevent queries . . .
. . .
templates 22.2 Embed the search engine into existing HTML code 22.3 The different style sheet files 23. JSON, XML and RSS result output [ Documentation ] 1. Settings, customizing and statistics If you want to change settings, behavior and design of Sphider-plus, you can do so by means of the Admin interface. There is a wide range of . . .
. . .
interface. There is a wide range of settings foreseen for Sphider-plus. Separated into different submenus like: Sites: - Add Site - Index only the new - Re-index all - Re-index only preferred URLs - Erase Re-index (available also for individual URLs) - Import/export URL list - Approve sites - Banned domains Categories: - Add, edit, delete - . . .
. . .
URL list - Approve sites - Banned domains Categories: - Add, edit, delete - Create new subcategory under Index: - Basic indexing options - Advanced options Clean: - Clean keywords not associated with any link - Clean links not associated with any site - Clean Category table not associated with any site - Clean Media links - Clear Temp table . . .
. . .
- Clear Search log - Clear 'Most Popular Page Links' log - Clear 'Most Popular Media Links' log - Clear Spider log, separate and bulk delete - Clear Thumbnail images, separate and bulk delete - Clear Text cache - Clear Media cache - Clear IDS log file - Clear flood attempts log file - Clear all entries in addurl or banned table - Truncate all . . .
. . .
flood attempts log file - Clear all entries in addurl or banned table - Truncate all tables in database Settings: - General Settings - Index Log Settings - Spider Settings - Search Settings - Order of Result listing - Suggest Options - Page Indexing Weights Database: - Configure up to 5 databases with unlimited number of table sets - Activate . . .
. . .
Database: - Configure up to 5 databases with unlimited number of table sets - Activate separately for 'Admin', 'Search' user and 'Suggest URL user' - Backup / Restore - Copy / Move - Optimize Templates: In order to enable customer's integration of Sphider-plus into existing sites, HTML templates are prepared for Search form Text result . . .
. . .
Search form Text result listing Media result listing Most popular queries etc. Three different designs are offered, which may be selected in submenu 'Settings'. If the layout does not fit the design of your site (which is normal), you may create new designs and modify the appropriate file /templates/My_template/adminstyle.css . . .
. . .
the appropriate file /templates/My_template/adminstyle.css /templates/My_template/userstyle.css Statistics output: - Top keywords (Top 50 with hit counter). - All indexed thumbnails w 3e80 ith ID3 and EXIF info. - Larges pages offering link URL and file size. - Most Popular Searches for text links offering: Link addr., total clicks, last . . .
. . .
Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Searches for media links offering: Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Links (click counter). - Search log offering: Query, Results, Queried at, Time taken, User IP, Country, Host name (Latest100) - Index log offering: . . .
. . .
Country, Host name (Latest100) - Index log offering: File-name, index date and delete option - sitemap log offering: sitemap.xml output sitemap list offering file/page suffixes - IDS log offering: IP, host, query, impact, involved tags, date and time of intrusion. - Flood attempts log offering: IP, query, date and time of flood attempt. - Auto . . .
. . .
attempts log offering: IP, query, date and time of flood attempt. - Auto Re-index log file - Server info offering: Server software, environment, MySQL, PDF-converter, image functions, php.ini file PHP integration, PHP security info. Each item holding lists of details. All text links, media links and thumbnails are active linked. As stated in . . .
. . .
individual Re-indexing, the periodical Re-indexer could be started and aborted in the "Options" menu of each site. 2.5 Preferred Re-indexing Each new URL added to the Admin backend, could be supplied with a priority level. This level will be used by the option 'Re-index only preferred sites'. Level 1 will be interpreted as most important, while . . .
. . .
less all index results will be stored in log files in sub folder /admin/log/ The names of the log files look like: db2_100524-21.47.56_1.html (log file of first thread) db2_100524-21.48.12_2.html (log file of second thread) and is build by the following items: db2 - Number of database. 100524 - Date (May 24, 2010) 21.47.56 - Time when this thread . . .
. . .
spider.php -new1 php spider.php -new2 etc. The IDs will be added to the name of the corresponding log files like: db2_100524-21.47.56_ID1.html (log file of first thread) db2_100524-21.48.12_ID2.html (log file of second thread) IDs could be defined by personal requirements, but the limitations for file names with respect to the OS should be taken . . .
. . .
but will not erase the content of all the other tables. So the check whether the content of a page has changed (MD5sum) is still available for a fast re-index procedure. Once prepared, multithreaded re-indexing could be invoked by starting several threads and adding individual IDs to the option parameter like: php spider.php -erased1 php . . .
. . .
4.4 Ignoring parts of a page Sphider-plus includes an option to exclude parts of pages from being indexed. Thi 775e s can for example be used to prevent search result flooding when certain keywords appear on certain part in most pages (like a header, footer or a menu). Any part of a page between <! sphider_noindex > and <! . . .
. . .
<! sphider_noindex > and <! /sphider_noindex > tags is not indexed, however links in it are followed. 4.5 Ignoring parts of a page by <div id='abc'> Ignoring parts of a page by the <! sphider_noindex > tags requires direct access to the page, because the tags need to be added (edited) to the page. A more flexible method, . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ menu[0-5] / 4.6 Indexing only parts of a page by <div id='abc'> If enabled in Admin settings, the values as defined in the list-file /include/common/divs_use.txt will be used to index only the content between <div . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ table[0-5] / 4.7 Ignore HTML elements defined by <tagname> . . </tagname> This option is foreseen to cooperate with the new HTML5 elements like section, nav, aside, hgroup, article, header, footer, etc. HTML elements . . .
. . .
contain a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */nav[0-5]/ Please keep in mind that element names placed in /include/common/elements_not.txt will be processed case-sensitive. 4.8 Index only HTML elements defined by <tagname> . . </tagname> This is the vice versa . . .
. . .
of all the duplicate content URLs: http://www.example.com/product.php?item=swedish-fish&category=gummy-candy http://www.example.com/product.php?item=swedish-fish&trackingid=1234&sessionid=5678 and Sphider-plus will understand that the duplicates all refer to the canonical URL: http://www.example.com/product.php?item=swedish-fish. The . . .
. . .
charset are supported by the ConvertCharset function and will be used to convert text into UTF-8 Unicode: WINDOWS windows-1250 - Central Europe windows-1251 - Cyrillic windows-1252 - Latin I windows-1253 - Greek windows-1254 - Turkish windows-1255 - Hebrew windows-1256 - Arabic windows-1257 - Baltic windows-1258 - Viet Nam cp874 - Thai - this . . .
. . .
- Baltic windows-1258 - Viet Nam cp874 - Thai - this file is also for DOS DOS cp437 - Latin US cp737 - Greek cp775 - BaltRim cp850 - Latin1 cp852 - Latin2 cp855 - Cyrylic cp857 - Turkish cp860 - Portuguese cp861 - Iceland cp862 - Hebrew cp863 - Canada cp864 - Arabic cp865 - Nordic cp866 - Cyrylic Russian (this is the one, used in IE . . .
. . .
IE Cyrillic (DOS) ) cp869 - Greek2 MAC (Apple) x-mac-cyrillic x-mac-greek x-mac-icelandic x-mac-ce x-mac-roman ISO iso-8859-1 iso-8859-2 iso-8859-3 iso-8859-4 iso-8859-5 iso-8859-6 iso-8859-7 iso-8859-8 iso-8859-9 iso-8859-10 iso-8859-11 iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 . . .
. . .
iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 cp500 cp856 cp875 cp1006 cp1026 koi8-r (Cyrillic) koi8-u (Cyrillic Ukrainian) nextstep us-ascii us-ascii-quotes DSP implementation for NeXT stdenc symbol zdingbat And specially for old Polish programs: mazovia This list is to be read . . .
. . .
Access denied; you need the RELOAD privilege. . . Top 10. Enable real-time output of logging data Up to version 1.5 of Sphider-plus, during index / re-index there was no printout available because: - Several servers, especially on Win32, buffer the output from the script until it terminates before transmitting the results to the browser. - . . .
. . .
is seen. - Some versions of Microsoft Internet Explorer only start to display the page after they have received 256 bytes of output. As progress was not presented during index / re- index procedure, waiting for results became a pain in the neck. Selectable in Admin setting together with the update interval (1 - 10 seconds), AJAX technology was . . .
. . .
characters in front of words Warning: This option should be used with special care and not be activated for non ISO-8859 charsets. Some special characters as part of the word ending might be erased by accidental. Top 13. Media search for images, audio streams and videos 13.1 Media indexing Index of media files is enabled by separated Admin . . .
. . .
and 'Found at' - Total clicks - Last clicked - Query input 'Indexed Image Thumbnails' presenting: - Thumbnail 150 x 100 pixel - Image details like title, filename size of original image, link- and thumb-id - Option to delete the thumbnail In order to open the media files all tables contain active links. Media results are also stored in . . .
. . .
Settings' menu of the admin backend: First option: For results of XML product feeds, present the text extract of 350 characters as part of product attributes. If this option is not activated, all products of the XML product feed will be presented in search results, and the hits of the query string will be highlighted in all involved products. . . .
. . .
content of the database tables (those with the same table prefix) will be destroyed by the restore procedure. 16.5 Copy & Move This section of the Database Management will present only those databases, which are correctly configured, assigned and do have a set of installed tables as described in chapter Definition and configuration. This . . .
. . .
results. Never the less to find these x results, the complete database had to been browsed. Starting with version 2.5, an additional clean option is offered as part of the Admin backend. Main advantage of this option is a significant reduction of the search time for any query, because the content of the db could be limited to offer only x . . .
. . .
<link>http://www.abc.de/images/warp.gif</link> <title>warp.gif</title> <x_size>635</x_size> <y_size>98</y_size> </media_result> <media_result> <num>2</num> <type>audio</type> <url>http://www.abc.de/index.php</url> . . .
. . .
the following content: {"query":"colour","ip":"::1","host_name":"guard_007-hoster","query_time":"2016-03-18 10:56:23 AM","consumed":0.016,"total_results":2,"num_of_results":2,"from":1,"to":2,"text_results":[{"num":1,"weight":"100.0","url":"http:\/\/www.english.le-piaggie.info\/Info_eng.pdf","title":" Info_eng","fulltxt":" . . . . . .
. . .
placed on the ground floor, today's big kitchen with the heating fireplace for open and close mode of . . . ","page_size":"5,661.6kb","domain_name":"www.english.le-piaggie.info"},{"num":2,"weight":"50.0","url":"http:\/\/www.english.le-piaggie.info\/html\/description.html","title":" Description","fulltxt":" . . . cable. Additional active . . .
. . .
D.O.C.G. as far as to the Pratomagno mountains 160 olive-trees Detached house, about 800 m to reach the . . . ","page_size":"26.4 kb","domain_name":"www.english.le-piaggie.info"}]} Top . . .


5a2 All required information. Introduction Release and Legal Info Installation Documentation Change Log [ Documentation Summary ] Preamble: The info presented here is valid only for the latest release of Sphider-plus. At present version 4.2024a published February 12, 2024 is the actual release. 1. Settings, customizing and statistics 2. Indexing . . .
. . .
2. Indexing 2.1 Various options 2.2 Allow other hosts in same domain 2.3 Word stemming 2.4 Periodical Re-indexing 2.5 Preferred indexing 2.6 Multithreaded indexing 2.7 Create thumbnails during index procedure 2.8 Prevent indexing of known malware and pishing pages 2.9 Follow and create sitemap files 2.10 Use private sitemap instead of global . . .
. . .
Sitemap file 3. Using the indexer from command line 3.1 All options 3.2 Multithreaded indexing 4. Keeping pages, words and files from being indexed 4.1 robots.txt 4.2 Must include / must not include string list 4.3 Ignoring links 4.4 Ignoring parts of a page by <! sphider_noindex > 4.5 Ignoring parts of a page by <div id='abc'> . . .
. . .
charset' 6. Search modes 6.1 Search with wildcards * 6.2 Strict search ! 6.3 Tolerant search 6.4 Link search 6.5 Media search 6.6 Search only in one domain 6.7 Search in categories 6.8 Greek language support 6.9 Block queries 7. Chronological order for result listing 7.1 Text result listing 7.2 Media result listing 8. PDF converter 9. Clean . . .
. . .
of logging data 11. Error messages and Debug mode 12. Delete secondary characters 13. Media search for images, audio streams and videos 13.1 Media indexing 13.2 Not supported media content 13.3 Search for media content 13.4 Statistics for media content 14. Feed support 14.1 XML product feeds 14.2 RDF, RSD, RSS and Atom feeds 15. Result . . .
. . .
Overview 16.2 Definition and configuration 16.3 Activate / disable database 16.4 Backup & Restore of databases 16.5 Copy & and Move 16.6 Enhancing functionality of multiple database support 17. Search in categories 17.1 Hierachical structure 17.2 Parallel structure 18. User suggested sites 19. Vulnerability protection 19.1 Prevent queries . . .
. . .
templates 22.2 Embed the search engine into existing HTML code 22.3 The different style sheet files 23. JSON, XML and RSS result output [ Documentation ] 1. Settings, customizing and statistics If you want to change settings, behavior and design of Sphider-plus, you can do so by means of the Admin interface. There is a wide range of . . .
. . .
interface. There is a wide range of settings foreseen for Sphider-plus. Separated into different submenus like: Sites: - Add Site - Index only the new - Re-index all - Re-index only preferred URLs - Erase Re-index (available also for individual URLs) - Import/export URL list - Approve sites - Banned domains Categories: - Add, edit, delete - . . .
. . .
URL list - Approve sites - Banned domains Categories: - Add, edit, delete - Create new subcategory under Index: - Basic indexing options - Advanced options Clean: - Clean keywords not associated with any link - Clean links not associated with any site - Clean Category table not associated with any site - Clean Media links - Clear Temp table . . .
. . .
- Clear Search log - Clear 'Most Popular Page Links' log - Clear 'Most Popular Media Links' log - Clear Spider log, separate and bulk delete - Clear Thumbnail images, separate and bulk delete - Clear Text cache - Clear Media cache - Clear IDS log file - Clear flood attempts log file - Clear all entries in addurl or banned table - Truncate all . . .
. . .
flood attempts log file - Clear all entries in addurl or banned table - Truncate all tables in database Settings: - General Settings - Index Log Settings - Spider Settings - Search Settings - Order of Result listing - Suggest Options - Page Indexing Weights Database: - Configure up to 5 databases with unlimited number of table sets - Activate . . .
. . .
Database: - Configure up to 5 databases with unlimited number of table sets - Activate separately for 'Admin', 'Search' user and 'Suggest URL user' - Backup / Restore - Copy / Move - Optimize Templates: In order to enable customer's integration of Sphider-plus into existing sites, HTML templates are prepared for Search form Text result . . .
. . .
Search form Text result listing Media result listing Most popular queries etc. Three different designs are offered, which may be selected in submenu 'Settings'. If the layout does not fit the design of your site (which is normal), you may create new designs and modify the appropriate file /templates/My_template/adminstyle.css . . .
. . .
the appropriate file /templates/My_template/adminstyle.css /templates/My_template/userstyle.css Statistics output: - Top keywords (Top 50 with hit counter). - All indexed thumbnails w 3e80 ith ID3 and EXIF info. - Larges pages offering link URL and file size. - Most Popular Searches for text links offering: Link addr., total clicks, last . . .
. . .
Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Searches for media links offering: Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Links (click counter). - Search log offering: Query, Results, Queried at, Time taken, User IP, Country, Host name (Latest100) - Index log offering: . . .
. . .
Country, Host name (Latest100) - Index log offering: File-name, index date and delete option - sitemap log offering: sitemap.xml output sitemap list offering file/page suffixes - IDS log offering: IP, host, query, impact, involved tags, date and time of intrusion. - Flood attempts log offering: IP, query, date and time of flood attempt. - Auto . . .
. . .
attempts log offering: IP, query, date and time of flood attempt. - Auto Re-index log file - Server info offering: Server software, environment, MySQL, PDF-converter, image functions, php.ini file PHP integration, PHP security info. Each item holding lists of details. All text links, media links and thumbnails are active linked. As stated in . . .
. . .
individual Re-indexing, the periodical Re-indexer could be started and aborted in the "Options" menu of each site. 2.5 Preferred Re-indexing Each new URL added to the Admin backend, could be supplied with a priority level. This level will be used by the option 'Re-index only preferred sites'. Level 1 will be interpreted as most important, while . . .
. . .
less all index results will be stored in log files in sub folder /admin/log/ The names of the log files look like: db2_100524-21.47.56_1.html (log file of first thread) db2_100524-21.48.12_2.html (log file of second thread) and is build by the following items: db2 - Number of database. 100524 - Date (May 24, 2010) 21.47.56 - Time when this thread . . .
. . .
spider.php -new1 php spider.php -new2 etc. The IDs will be added to the name of the corresponding log files like: db2_100524-21.47.56_ID1.html (log file of first thread) db2_100524-21.48.12_ID2.html (log file of second thread) IDs could be defined by personal requirements, but the limitations for file names with respect to the OS should be taken . . .
. . .
but will not erase the content of all the other tables. So the check whether the content of a page has changed (MD5sum) is still available for a fast re-index procedure. Once prepared, multithreaded re-indexing could be invoked by starting several threads and adding individual IDs to the option parameter like: php spider.php -erased1 php . . .
. . .
4.4 Ignoring parts of a page Sphider-plus includes an option to exclude parts of pages from being indexed. Thi 775e s can for example be used to prevent search result flooding when certain keywords appear on certain part in most pages (like a header, footer or a menu). Any part of a page between <! sphider_noindex > and <! . . .
. . .
<! sphider_noindex > and <! /sphider_noindex > tags is not indexed, however links in it are followed. 4.5 Ignoring parts of a page by <div id='abc'> Ignoring parts of a page by the <! sphider_noindex > tags requires direct access to the page, because the tags need to be added (edited) to the page. A more flexible method, . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ menu[0-5] / 4.6 Indexing only parts of a page by <div id='abc'> If enabled in Admin settings, the values as defined in the list-file /include/common/divs_use.txt will be used to index only the content between <div . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ table[0-5] / 4.7 Ignore HTML elements defined by <tagname> . . </tagname> This option is foreseen to cooperate with the new HTML5 elements like section, nav, aside, hgroup, article, header, footer, etc. HTML elements . . .
. . .
contain a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */nav[0-5]/ Please keep in mind that element names placed in /include/common/elements_not.txt will be processed case-sensitive. 4.8 Index only HTML elements defined by <tagname> . . </tagname> This is the vice versa . . .
. . .
of all the duplicate content URLs: http://www.example.com/product.php?item=swedish-fish&category=gummy-candy http://www.example.com/product.php?item=swedish-fish&trackingid=1234&sessionid=5678 and Sphider-plus will understand that the duplicates all refer to the canonical URL: http://www.example.com/product.php?item=swedish-fish. The . . .
. . .
charset are supported by the ConvertCharset function and will be used to convert text into UTF-8 Unicode: WINDOWS windows-1250 - Central Europe windows-1251 - Cyrillic windows-1252 - Latin I windows-1253 - Greek windows-1254 - Turkish windows-1255 - Hebrew windows-1256 - Arabic windows-1257 - Baltic windows-1258 - Viet Nam cp874 - Thai - this . . .
. . .
- Baltic windows-1258 - Viet Nam cp874 - Thai - this file is also for DOS DOS cp437 - Latin US cp737 - Greek cp775 - BaltRim cp850 - Latin1 cp852 - Latin2 cp855 - Cyrylic cp857 - Turkish cp860 - Portuguese cp861 - Iceland cp862 - Hebrew cp863 - Canada cp864 - Arabic cp865 - Nordic cp866 - Cyrylic Russian (this is the one, used in IE . . .
. . .
IE Cyrillic (DOS) ) cp869 - Greek2 MAC (Apple) x-mac-cyrillic x-mac-greek x-mac-icelandic x-mac-ce x-mac-roman ISO iso-8859-1 iso-8859-2 iso-8859-3 iso-8859-4 iso-8859-5 iso-8859-6 iso-8859-7 iso-8859-8 iso-8859-9 iso-8859-10 iso-8859-11 iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 . . .
. . .
iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 cp500 cp856 cp875 cp1006 cp1026 koi8-r (Cyrillic) koi8-u (Cyrillic Ukrainian) nextstep us-ascii us-ascii-quotes DSP implementation for NeXT stdenc symbol zdingbat And specially for old Polish programs: mazovia This list is to be read . . .
. . .
Access denied; you need the RELOAD privilege. . . Top 10. Enable real-time output of logging data Up to version 1.5 of Sphider-plus, during index / re-index there was no printout available because: - Several servers, especially on Win32, buffer the output from the script until it terminates before transmitting the results to the browser. - . . .
. . .
is seen. - Some versions of Microsoft Internet Explorer only start to display the page after they have received 256 bytes of output. As progress was not presented during index / re- index procedure, waiting for results became a pain in the neck. Selectable in Admin setting together with the update interval (1 - 10 seconds), AJAX technology was . . .
. . .
characters in front of words Warning: This option should be used with special care and not be activated for non ISO-8859 charsets. Some special characters as part of the word ending might be erased by accidental. Top 13. Media search for images, audio streams and videos 13.1 Media indexing Index of media files is enabled by separated Admin . . .
. . .
and 'Found at' - Total clicks - Last clicked - Query input 'Indexed Image Thumbnails' presenting: - Thumbnail 150 x 100 pixel - Image details like title, filename size of original image, link- and thumb-id - Option to delete the thumbnail In order to open the media files all tables contain active links. Media results are also stored in . . .
. . .
Settings' menu of the admin backend: First option: For results of XML product feeds, present the text extract of 350 characters as part of product attributes. If this option is not activated, all products of the XML product feed will be presented in search results, and the hits of the query string will be highlighted in all involved products. . . .
. . .
content of the database tables (those with the same table prefix) will be destroyed by the restore procedure. 16.5 Copy & Move This section of the Database Management will present only those databases, which are correctly configured, assigned and do have a set of installed tables as described in chapter Definition and configuration. This . . .
. . .
results. Never the less to find these x results, the complete database had to been browsed. Starting with version 2.5, an additional clean option is offered as part of the Admin backend. Main advantage of this option is a significant reduction of the search time for any query, because the content of the db could be limited to offer only x . . .
. . .
<link>http://www.abc.de/images/warp.gif</link> <title>warp.gif</title> <x_size>635</x_size> <y_size>98</y_size> </media_result> <media_result> <num>2</num> <type>audio</type> <url>http://www.abc.de/index.php</url> . . .
. . .
the following content: {"query":"colour","ip":"::1","host_name":"guard_007-hoster","query_time":"2016-03-18 10:56:23 AM","consumed":0.016,"total_results":2,"num_of_results":2,"from":1,"to":2,"text_results":[{"num":1,"weight":"100.0","url":"http:\/\/www.english.le-piaggie.info\/Info_eng.pdf","title":" Info_eng","fulltxt":" . . . . . .
. . .
placed on the ground floor, today's big kitchen with the heating fireplace for open and close mode of . . . ","page_size":"5,661.6kb","domain_name":"www.english.le-piaggie.info"},{"num":2,"weight":"50.0","url":"http:\/\/www.english.le-piaggie.info\/html\/description.html","title":" Description","fulltxt":" . . . cable. Additional active . . .
. . .
D.O.C.G. as far as to the Pratomagno mountains 160 olive-trees Detached house, about 800 m to reach the . . . ","page_size":"26.4 kb","domain_name":"www.english.le-piaggie.info"}]} Top . . .


5a2 All required information. Introduction Release and Legal Info Installation Documentation Change Log [ Documentation Summary ] Preamble: The info presented here is valid only for the latest release of Sphider-plus. At present version 4.2024a published February 12, 2024 is the actual release. 1. Settings, customizing and statistics 2. Indexing . . .
. . .
2. Indexing 2.1 Various options 2.2 Allow other hosts in same domain 2.3 Word stemming 2.4 Periodical Re-indexing 2.5 Preferred indexing 2.6 Multithreaded indexing 2.7 Create thumbnails during index procedure 2.8 Prevent indexing of known malware and pishing pages 2.9 Follow and create sitemap files 2.10 Use private sitemap instead of global . . .
. . .
Sitemap file 3. Using the indexer from command line 3.1 All options 3.2 Multithreaded indexing 4. Keeping pages, words and files from being indexed 4.1 robots.txt 4.2 Must include / must not include string list 4.3 Ignoring links 4.4 Ignoring parts of a page by <! sphider_noindex > 4.5 Ignoring parts of a page by <div id='abc'> . . .
. . .
charset' 6. Search modes 6.1 Search with wildcards * 6.2 Strict search ! 6.3 Tolerant search 6.4 Link search 6.5 Media search 6.6 Search only in one domain 6.7 Search in categories 6.8 Greek language support 6.9 Block queries 7. Chronological order for result listing 7.1 Text result listing 7.2 Media result listing 8. PDF converter 9. Clean . . .
. . .
of logging data 11. Error messages and Debug mode 12. Delete secondary characters 13. Media search for images, audio streams and videos 13.1 Media indexing 13.2 Not supported media content 13.3 Search for media content 13.4 Statistics for media content 14. Feed support 14.1 XML product feeds 14.2 RDF, RSD, RSS and Atom feeds 15. Result . . .
. . .
Overview 16.2 Definition and configuration 16.3 Activate / disable database 16.4 Backup & Restore of databases 16.5 Copy & and Move 16.6 Enhancing functionality of multiple database support 17. Search in categories 17.1 Hierachical structure 17.2 Parallel structure 18. User suggested sites 19. Vulnerability protection 19.1 Prevent queries . . .
. . .
templates 22.2 Embed the search engine into existing HTML code 22.3 The different style sheet files 23. JSON, XML and RSS result output [ Documentation ] 1. Settings, customizing and statistics If you want to change settings, behavior and design of Sphider-plus, you can do so by means of the Admin interface. There is a wide range of . . .
. . .
interface. There is a wide range of settings foreseen for Sphider-plus. Separated into different submenus like: Sites: - Add Site - Index only the new - Re-index all - Re-index only preferred URLs - Erase Re-index (available also for individual URLs) - Import/export URL list - Approve sites - Banned domains Categories: - Add, edit, delete - . . .
. . .
URL list - Approve sites - Banned domains Categories: - Add, edit, delete - Create new subcategory under Index: - Basic indexing options - Advanced options Clean: - Clean keywords not associated with any link - Clean links not associated with any site - Clean Category table not associated with any site - Clean Media links - Clear Temp table . . .
. . .
- Clear Search log - Clear 'Most Popular Page Links' log - Clear 'Most Popular Media Links' log - Clear Spider log, separate and bulk delete - Clear Thumbnail images, separate and bulk delete - Clear Text cache - Clear Media cache - Clear IDS log file - Clear flood attempts log file - Clear all entries in addurl or banned table - Truncate all . . .
. . .
flood attempts log file - Clear all entries in addurl or banned table - Truncate all tables in database Settings: - General Settings - Index Log Settings - Spider Settings - Search Settings - Order of Result listing - Suggest Options - Page Indexing Weights Database: - Configure up to 5 databases with unlimited number of table sets - Activate . . .
. . .
Database: - Configure up to 5 databases with unlimited number of table sets - Activate separately for 'Admin', 'Search' user and 'Suggest URL user' - Backup / Restore - Copy / Move - Optimize Templates: In order to enable customer's integration of Sphider-plus into existing sites, HTML templates are prepared for Search form Text result . . .
. . .
Search form Text result listing Media result listing Most popular queries etc. Three different designs are offered, which may be selected in submenu 'Settings'. If the layout does not fit the design of your site (which is normal), you may create new designs and modify the appropriate file /templates/My_template/adminstyle.css . . .
. . .
the appropriate file /templates/My_template/adminstyle.css /templates/My_template/userstyle.css Statistics output: - Top keywords (Top 50 with hit counter). - All indexed thumbnails w 3e80 ith ID3 and EXIF info. - Larges pages offering link URL and file size. - Most Popular Searches for text links offering: Link addr., total clicks, last . . .
. . .
Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Searches for media links offering: Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Links (click counter). - Search log offering: Query, Results, Queried at, Time taken, User IP, Country, Host name (Latest100) - Index log offering: . . .
. . .
Country, Host name (Latest100) - Index log offering: File-name, index date and delete option - sitemap log offering: sitemap.xml output sitemap list offering file/page suffixes - IDS log offering: IP, host, query, impact, involved tags, date and time of intrusion. - Flood attempts log offering: IP, query, date and time of flood attempt. - Auto . . .
. . .
attempts log offering: IP, query, date and time of flood attempt. - Auto Re-index log file - Server info offering: Server software, environment, MySQL, PDF-converter, image functions, php.ini file PHP integration, PHP security info. Each item holding lists of details. All text links, media links and thumbnails are active linked. As stated in . . .
. . .
individual Re-indexing, the periodical Re-indexer could be started and aborted in the "Options" menu of each site. 2.5 Preferred Re-indexing Each new URL added to the Admin backend, could be supplied with a priority level. This level will be used by the option 'Re-index only preferred sites'. Level 1 will be interpreted as most important, while . . .
. . .
less all index results will be stored in log files in sub folder /admin/log/ The names of the log files look like: db2_100524-21.47.56_1.html (log file of first thread) db2_100524-21.48.12_2.html (log file of second thread) and is build by the following items: db2 - Number of database. 100524 - Date (May 24, 2010) 21.47.56 - Time when this thread . . .
. . .
spider.php -new1 php spider.php -new2 etc. The IDs will be added to the name of the corresponding log files like: db2_100524-21.47.56_ID1.html (log file of first thread) db2_100524-21.48.12_ID2.html (log file of second thread) IDs could be defined by personal requirements, but the limitations for file names with respect to the OS should be taken . . .
. . .
but will not erase the content of all the other tables. So the check whether the content of a page has changed (MD5sum) is still available for a fast re-index procedure. Once prepared, multithreaded re-indexing could be invoked by starting several threads and adding individual IDs to the option parameter like: php spider.php -erased1 php . . .
. . .
4.4 Ignoring parts of a page Sphider-plus includes an option to exclude parts of pages from being indexed. Thi 775e s can for example be used to prevent search result flooding when certain keywords appear on certain part in most pages (like a header, footer or a menu). Any part of a page between <! sphider_noindex > and <! . . .
. . .
<! sphider_noindex > and <! /sphider_noindex > tags is not indexed, however links in it are followed. 4.5 Ignoring parts of a page by <div id='abc'> Ignoring parts of a page by the <! sphider_noindex > tags requires direct access to the page, because the tags need to be added (edited) to the page. A more flexible method, . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ menu[0-5] / 4.6 Indexing only parts of a page by <div id='abc'> If enabled in Admin settings, the values as defined in the list-file /include/common/divs_use.txt will be used to index only the content between <div . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ table[0-5] / 4.7 Ignore HTML elements defined by <tagname> . . </tagname> This option is foreseen to cooperate with the new HTML5 elements like section, nav, aside, hgroup, article, header, footer, etc. HTML elements . . .
. . .
contain a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */nav[0-5]/ Please keep in mind that element names placed in /include/common/elements_not.txt will be processed case-sensitive. 4.8 Index only HTML elements defined by <tagname> . . </tagname> This is the vice versa . . .
. . .
of all the duplicate content URLs: http://www.example.com/product.php?item=swedish-fish&category=gummy-candy http://www.example.com/product.php?item=swedish-fish&trackingid=1234&sessionid=5678 and Sphider-plus will understand that the duplicates all refer to the canonical URL: http://www.example.com/product.php?item=swedish-fish. The . . .
. . .
charset are supported by the ConvertCharset function and will be used to convert text into UTF-8 Unicode: WINDOWS windows-1250 - Central Europe windows-1251 - Cyrillic windows-1252 - Latin I windows-1253 - Greek windows-1254 - Turkish windows-1255 - Hebrew windows-1256 - Arabic windows-1257 - Baltic windows-1258 - Viet Nam cp874 - Thai - this . . .
. . .
- Baltic windows-1258 - Viet Nam cp874 - Thai - this file is also for DOS DOS cp437 - Latin US cp737 - Greek cp775 - BaltRim cp850 - Latin1 cp852 - Latin2 cp855 - Cyrylic cp857 - Turkish cp860 - Portuguese cp861 - Iceland cp862 - Hebrew cp863 - Canada cp864 - Arabic cp865 - Nordic cp866 - Cyrylic Russian (this is the one, used in IE . . .
. . .
IE Cyrillic (DOS) ) cp869 - Greek2 MAC (Apple) x-mac-cyrillic x-mac-greek x-mac-icelandic x-mac-ce x-mac-roman ISO iso-8859-1 iso-8859-2 iso-8859-3 iso-8859-4 iso-8859-5 iso-8859-6 iso-8859-7 iso-8859-8 iso-8859-9 iso-8859-10 iso-8859-11 iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 . . .
. . .
iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 cp500 cp856 cp875 cp1006 cp1026 koi8-r (Cyrillic) koi8-u (Cyrillic Ukrainian) nextstep us-ascii us-ascii-quotes DSP implementation for NeXT stdenc symbol zdingbat And specially for old Polish programs: mazovia This list is to be read . . .
. . .
Access denied; you need the RELOAD privilege. . . Top 10. Enable real-time output of logging data Up to version 1.5 of Sphider-plus, during index / re-index there was no printout available because: - Several servers, especially on Win32, buffer the output from the script until it terminates before transmitting the results to the browser. - . . .
. . .
is seen. - Some versions of Microsoft Internet Explorer only start to display the page after they have received 256 bytes of output. As progress was not presented during index / re- index procedure, waiting for results became a pain in the neck. Selectable in Admin setting together with the update interval (1 - 10 seconds), AJAX technology was . . .
. . .
characters in front of words Warning: This option should be used with special care and not be activated for non ISO-8859 charsets. Some special characters as part of the word ending might be erased by accidental. Top 13. Media search for images, audio streams and videos 13.1 Media indexing Index of media files is enabled by separated Admin . . .
. . .
and 'Found at' - Total clicks - Last clicked - Query input 'Indexed Image Thumbnails' presenting: - Thumbnail 150 x 100 pixel - Image details like title, filename size of original image, link- and thumb-id - Option to delete the thumbnail In order to open the media files all tables contain active links. Media results are also stored in . . .
. . .
Settings' menu of the admin backend: First option: For results of XML product feeds, present the text extract of 350 characters as part of product attributes. If this option is not activated, all products of the XML product feed will be presented in search results, and the hits of the query string will be highlighted in all involved products. . . .
. . .
content of the database tables (those with the same table prefix) will be destroyed by the restore procedure. 16.5 Copy & Move This section of the Database Management will present only those databases, which are correctly configured, assigned and do have a set of installed tables as described in chapter Definition and configuration. This . . .
. . .
results. Never the less to find these x results, the complete database had to been browsed. Starting with version 2.5, an additional clean option is offered as part of the Admin backend. Main advantage of this option is a significant reduction of the search time for any query, because the content of the db could be limited to offer only x . . .
. . .
<link>http://www.abc.de/images/warp.gif</link> <title>warp.gif</title> <x_size>635</x_size> <y_size>98</y_size> </media_result> <media_result> <num>2</num> <type>audio</type> <url>http://www.abc.de/index.php</url> . . .
. . .
the following content: {"query":"colour","ip":"::1","host_name":"guard_007-hoster","query_time":"2016-03-18 10:56:23 AM","consumed":0.016,"total_results":2,"num_of_results":2,"from":1,"to":2,"text_results":[{"num":1,"weight":"100.0","url":"http:\/\/www.english.le-piaggie.info\/Info_eng.pdf","title":" Info_eng","fulltxt":" . . . . . .
. . .
placed on the ground floor, today's big kitchen with the heating fireplace for open and close mode of . . . ","page_size":"5,661.6kb","domain_name":"www.english.le-piaggie.info"},{"num":2,"weight":"50.0","url":"http:\/\/www.english.le-piaggie.info\/html\/description.html","title":" Description","fulltxt":" . . . cable. Additional active . . .
. . .
D.O.C.G. as far as to the Pratomagno mountains 160 olive-trees Detached house, about 800 m to reach the . . . ","page_size":"26.4 kb","domain_name":"www.english.le-piaggie.info"}]} Top . . .


5a2 All required information. Introduction Release and Legal Info Installation Documentation Change Log [ Documentation Summary ] Preamble: The info presented here is valid only for the latest release of Sphider-plus. At present version 4.2024a published February 12, 2024 is the actual release. 1. Settings, customizing and statistics 2. Indexing . . .
. . .
2. Indexing 2.1 Various options 2.2 Allow other hosts in same domain 2.3 Word stemming 2.4 Periodical Re-indexing 2.5 Preferred indexing 2.6 Multithreaded indexing 2.7 Create thumbnails during index procedure 2.8 Prevent indexing of known malware and pishing pages 2.9 Follow and create sitemap files 2.10 Use private sitemap instead of global . . .
. . .
Sitemap file 3. Using the indexer from command line 3.1 All options 3.2 Multithreaded indexing 4. Keeping pages, words and files from being indexed 4.1 robots.txt 4.2 Must include / must not include string list 4.3 Ignoring links 4.4 Ignoring parts of a page by <! sphider_noindex > 4.5 Ignoring parts of a page by <div id='abc'> . . .
. . .
charset' 6. Search modes 6.1 Search with wildcards * 6.2 Strict search ! 6.3 Tolerant search 6.4 Link search 6.5 Media search 6.6 Search only in one domain 6.7 Search in categories 6.8 Greek language support 6.9 Block queries 7. Chronological order for result listing 7.1 Text result listing 7.2 Media result listing 8. PDF converter 9. Clean . . .
. . .
of logging data 11. Error messages and Debug mode 12. Delete secondary characters 13. Media search for images, audio streams and videos 13.1 Media indexing 13.2 Not supported media content 13.3 Search for media content 13.4 Statistics for media content 14. Feed support 14.1 XML product feeds 14.2 RDF, RSD, RSS and Atom feeds 15. Result . . .
. . .
Overview 16.2 Definition and configuration 16.3 Activate / disable database 16.4 Backup & Restore of databases 16.5 Copy & and Move 16.6 Enhancing functionality of multiple database support 17. Search in categories 17.1 Hierachical structure 17.2 Parallel structure 18. User suggested sites 19. Vulnerability protection 19.1 Prevent queries . . .
. . .
templates 22.2 Embed the search engine into existing HTML code 22.3 The different style sheet files 23. JSON, XML and RSS result output [ Documentation ] 1. Settings, customizing and statistics If you want to change settings, behavior and design of Sphider-plus, you can do so by means of the Admin interface. There is a wide range of . . .
. . .
interface. There is a wide range of settings foreseen for Sphider-plus. Separated into different submenus like: Sites: - Add Site - Index only the new - Re-index all - Re-index only preferred URLs - Erase Re-index (available also for individual URLs) - Import/export URL list - Approve sites - Banned domains Categories: - Add, edit, delete - . . .
. . .
URL list - Approve sites - Banned domains Categories: - Add, edit, delete - Create new subcategory under Index: - Basic indexing options - Advanced options Clean: - Clean keywords not associated with any link - Clean links not associated with any site - Clean Category table not associated with any site - Clean Media links - Clear Temp table . . .
. . .
- Clear Search log - Clear 'Most Popular Page Links' log - Clear 'Most Popular Media Links' log - Clear Spider log, separate and bulk delete - Clear Thumbnail images, separate and bulk delete - Clear Text cache - Clear Media cache - Clear IDS log file - Clear flood attempts log file - Clear all entries in addurl or banned table - Truncate all . . .
. . .
flood attempts log file - Clear all entries in addurl or banned table - Truncate all tables in database Settings: - General Settings - Index Log Settings - Spider Settings - Search Settings - Order of Result listing - Suggest Options - Page Indexing Weights Database: - Configure up to 5 databases with unlimited number of table sets - Activate . . .
. . .
Database: - Configure up to 5 databases with unlimited number of table sets - Activate separately for 'Admin', 'Search' user and 'Suggest URL user' - Backup / Restore - Copy / Move - Optimize Templates: In order to enable customer's integration of Sphider-plus into existing sites, HTML templates are prepared for Search form Text result . . .
. . .
Search form Text result listing Media result listing Most popular queries etc. Three different designs are offered, which may be selected in submenu 'Settings'. If the layout does not fit the design of your site (which is normal), you may create new designs and modify the appropriate file /templates/My_template/adminstyle.css . . .
. . .
the appropriate file /templates/My_template/adminstyle.css /templates/My_template/userstyle.css Statistics output: - Top keywords (Top 50 with hit counter). - All indexed thumbnails w 3e80 ith ID3 and EXIF info. - Larges pages offering link URL and file size. - Most Popular Searches for text links offering: Link addr., total clicks, last . . .
. . .
Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Searches for media links offering: Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Links (click counter). - Search log offering: Query, Results, Queried at, Time taken, User IP, Country, Host name (Latest100) - Index log offering: . . .
. . .
Country, Host name (Latest100) - Index log offering: File-name, index date and delete option - sitemap log offering: sitemap.xml output sitemap list offering file/page suffixes - IDS log offering: IP, host, query, impact, involved tags, date and time of intrusion. - Flood attempts log offering: IP, query, date and time of flood attempt. - Auto . . .
. . .
attempts log offering: IP, query, date and time of flood attempt. - Auto Re-index log file - Server info offering: Server software, environment, MySQL, PDF-converter, image functions, php.ini file PHP integration, PHP security info. Each item holding lists of details. All text links, media links and thumbnails are active linked. As stated in . . .
. . .
individual Re-indexing, the periodical Re-indexer could be started and aborted in the "Options" menu of each site. 2.5 Preferred Re-indexing Each new URL added to the Admin backend, could be supplied with a priority level. This level will be used by the option 'Re-index only preferred sites'. Level 1 will be interpreted as most important, while . . .
. . .
less all index results will be stored in log files in sub folder /admin/log/ The names of the log files look like: db2_100524-21.47.56_1.html (log file of first thread) db2_100524-21.48.12_2.html (log file of second thread) and is build by the following items: db2 - Number of database. 100524 - Date (May 24, 2010) 21.47.56 - Time when this thread . . .
. . .
spider.php -new1 php spider.php -new2 etc. The IDs will be added to the name of the corresponding log files like: db2_100524-21.47.56_ID1.html (log file of first thread) db2_100524-21.48.12_ID2.html (log file of second thread) IDs could be defined by personal requirements, but the limitations for file names with respect to the OS should be taken . . .
. . .
but will not erase the content of all the other tables. So the check whether the content of a page has changed (MD5sum) is still available for a fast re-index procedure. Once prepared, multithreaded re-indexing could be invoked by starting several threads and adding individual IDs to the option parameter like: php spider.php -erased1 php . . .
. . .
4.4 Ignoring parts of a page Sphider-plus includes an option to exclude parts of pages from being indexed. Thi 775e s can for example be used to prevent search result flooding when certain keywords appear on certain part in most pages (like a header, footer or a menu). Any part of a page between <! sphider_noindex > and <! . . .
. . .
<! sphider_noindex > and <! /sphider_noindex > tags is not indexed, however links in it are followed. 4.5 Ignoring parts of a page by <div id='abc'> Ignoring parts of a page by the <! sphider_noindex > tags requires direct access to the page, because the tags need to be added (edited) to the page. A more flexible method, . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ menu[0-5] / 4.6 Indexing only parts of a page by <div id='abc'> If enabled in Admin settings, the values as defined in the list-file /include/common/divs_use.txt will be used to index only the content between <div . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ table[0-5] / 4.7 Ignore HTML elements defined by <tagname> . . </tagname> This option is foreseen to cooperate with the new HTML5 elements like section, nav, aside, hgroup, article, header, footer, etc. HTML elements . . .
. . .
contain a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */nav[0-5]/ Please keep in mind that element names placed in /include/common/elements_not.txt will be processed case-sensitive. 4.8 Index only HTML elements defined by <tagname> . . </tagname> This is the vice versa . . .
. . .
of all the duplicate content URLs: http://www.example.com/product.php?item=swedish-fish&category=gummy-candy http://www.example.com/product.php?item=swedish-fish&trackingid=1234&sessionid=5678 and Sphider-plus will understand that the duplicates all refer to the canonical URL: http://www.example.com/product.php?item=swedish-fish. The . . .
. . .
charset are supported by the ConvertCharset function and will be used to convert text into UTF-8 Unicode: WINDOWS windows-1250 - Central Europe windows-1251 - Cyrillic windows-1252 - Latin I windows-1253 - Greek windows-1254 - Turkish windows-1255 - Hebrew windows-1256 - Arabic windows-1257 - Baltic windows-1258 - Viet Nam cp874 - Thai - this . . .
. . .
- Baltic windows-1258 - Viet Nam cp874 - Thai - this file is also for DOS DOS cp437 - Latin US cp737 - Greek cp775 - BaltRim cp850 - Latin1 cp852 - Latin2 cp855 - Cyrylic cp857 - Turkish cp860 - Portuguese cp861 - Iceland cp862 - Hebrew cp863 - Canada cp864 - Arabic cp865 - Nordic cp866 - Cyrylic Russian (this is the one, used in IE . . .
. . .
IE Cyrillic (DOS) ) cp869 - Greek2 MAC (Apple) x-mac-cyrillic x-mac-greek x-mac-icelandic x-mac-ce x-mac-roman ISO iso-8859-1 iso-8859-2 iso-8859-3 iso-8859-4 iso-8859-5 iso-8859-6 iso-8859-7 iso-8859-8 iso-8859-9 iso-8859-10 iso-8859-11 iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 . . .
. . .
iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 cp500 cp856 cp875 cp1006 cp1026 koi8-r (Cyrillic) koi8-u (Cyrillic Ukrainian) nextstep us-ascii us-ascii-quotes DSP implementation for NeXT stdenc symbol zdingbat And specially for old Polish programs: mazovia This list is to be read . . .
. . .
Access denied; you need the RELOAD privilege. . . Top 10. Enable real-time output of logging data Up to version 1.5 of Sphider-plus, during index / re-index there was no printout available because: - Several servers, especially on Win32, buffer the output from the script until it terminates before transmitting the results to the browser. - . . .
. . .
is seen. - Some versions of Microsoft Internet Explorer only start to display the page after they have received 256 bytes of output. As progress was not presented during index / re- index procedure, waiting for results became a pain in the neck. Selectable in Admin setting together with the update interval (1 - 10 seconds), AJAX technology was . . .
. . .
characters in front of words Warning: This option should be used with special care and not be activated for non ISO-8859 charsets. Some special characters as part of the word ending might be erased by accidental. Top 13. Media search for images, audio streams and videos 13.1 Media indexing Index of media files is enabled by separated Admin . . .
. . .
and 'Found at' - Total clicks - Last clicked - Query input 'Indexed Image Thumbnails' presenting: - Thumbnail 150 x 100 pixel - Image details like title, filename size of original image, link- and thumb-id - Option to delete the thumbnail In order to open the media files all tables contain active links. Media results are also stored in . . .
. . .
Settings' menu of the admin backend: First option: For results of XML product feeds, present the text extract of 350 characters as part of product attributes. If this option is not activated, all products of the XML product feed will be presented in search results, and the hits of the query string will be highlighted in all involved products. . . .
. . .
content of the database tables (those with the same table prefix) will be destroyed by the restore procedure. 16.5 Copy & Move This section of the Database Management will present only those databases, which are correctly configured, assigned and do have a set of installed tables as described in chapter Definition and configuration. This . . .
. . .
results. Never the less to find these x results, the complete database had to been browsed. Starting with version 2.5, an additional clean option is offered as part of the Admin backend. Main advantage of this option is a significant reduction of the search time for any query, because the content of the db could be limited to offer only x . . .
. . .
<link>http://www.abc.de/images/warp.gif</link> <title>warp.gif</title> <x_size>635</x_size> <y_size>98</y_size> </media_result> <media_result> <num>2</num> <type>audio</type> <url>http://www.abc.de/index.php</url> . . .
. . .
the following content: {"query":"colour","ip":"::1","host_name":"guard_007-hoster","query_time":"2016-03-18 10:56:23 AM","consumed":0.016,"total_results":2,"num_of_results":2,"from":1,"to":2,"text_results":[{"num":1,"weight":"100.0","url":"http:\/\/www.english.le-piaggie.info\/Info_eng.pdf","title":" Info_eng","fulltxt":" . . . . . .
. . .
placed on the ground floor, today's big kitchen with the heating fireplace for open and close mode of . . . ","page_size":"5,661.6kb","domain_name":"www.english.le-piaggie.info"},{"num":2,"weight":"50.0","url":"http:\/\/www.english.le-piaggie.info\/html\/description.html","title":" Description","fulltxt":" . . . cable. Additional active . . .
. . .
D.O.C.G. as far as to the Pratomagno mountains 160 olive-trees Detached house, about 800 m to reach the . . . ","page_size":"26.4 kb","domain_name":"www.english.le-piaggie.info"}]} Top . . .


5a2 All required information. Introduction Release and Legal Info Installation Documentation Change Log [ Documentation Summary ] Preamble: The info presented here is valid only for the latest release of Sphider-plus. At present version 4.2024a published February 12, 2024 is the actual release. 1. Settings, customizing and statistics 2. Indexing . . .
. . .
2. Indexing 2.1 Various options 2.2 Allow other hosts in same domain 2.3 Word stemming 2.4 Periodical Re-indexing 2.5 Preferred indexing 2.6 Multithreaded indexing 2.7 Create thumbnails during index procedure 2.8 Prevent indexing of known malware and pishing pages 2.9 Follow and create sitemap files 2.10 Use private sitemap instead of global . . .
. . .
Sitemap file 3. Using the indexer from command line 3.1 All options 3.2 Multithreaded indexing 4. Keeping pages, words and files from being indexed 4.1 robots.txt 4.2 Must include / must not include string list 4.3 Ignoring links 4.4 Ignoring parts of a page by <! sphider_noindex > 4.5 Ignoring parts of a page by <div id='abc'> . . .
. . .
charset' 6. Search modes 6.1 Search with wildcards * 6.2 Strict search ! 6.3 Tolerant search 6.4 Link search 6.5 Media search 6.6 Search only in one domain 6.7 Search in categories 6.8 Greek language support 6.9 Block queries 7. Chronological order for result listing 7.1 Text result listing 7.2 Media result listing 8. PDF converter 9. Clean . . .
. . .
of logging data 11. Error messages and Debug mode 12. Delete secondary characters 13. Media search for images, audio streams and videos 13.1 Media indexing 13.2 Not supported media content 13.3 Search for media content 13.4 Statistics for media content 14. Feed support 14.1 XML product feeds 14.2 RDF, RSD, RSS and Atom feeds 15. Result . . .
. . .
Overview 16.2 Definition and configuration 16.3 Activate / disable database 16.4 Backup & Restore of databases 16.5 Copy & and Move 16.6 Enhancing functionality of multiple database support 17. Search in categories 17.1 Hierachical structure 17.2 Parallel structure 18. User suggested sites 19. Vulnerability protection 19.1 Prevent queries . . .
. . .
templates 22.2 Embed the search engine into existing HTML code 22.3 The different style sheet files 23. JSON, XML and RSS result output [ Documentation ] 1. Settings, customizing and statistics If you want to change settings, behavior and design of Sphider-plus, you can do so by means of the Admin interface. There is a wide range of . . .
. . .
interface. There is a wide range of settings foreseen for Sphider-plus. Separated into different submenus like: Sites: - Add Site - Index only the new - Re-index all - Re-index only preferred URLs - Erase Re-index (available also for individual URLs) - Import/export URL list - Approve sites - Banned domains Categories: - Add, edit, delete - . . .
. . .
URL list - Approve sites - Banned domains Categories: - Add, edit, delete - Create new subcategory under Index: - Basic indexing options - Advanced options Clean: - Clean keywords not associated with any link - Clean links not associated with any site - Clean Category table not associated with any site - Clean Media links - Clear Temp table . . .
. . .
- Clear Search log - Clear 'Most Popular Page Links' log - Clear 'Most Popular Media Links' log - Clear Spider log, separate and bulk delete - Clear Thumbnail images, separate and bulk delete - Clear Text cache - Clear Media cache - Clear IDS log file - Clear flood attempts log file - Clear all entries in addurl or banned table - Truncate all . . .
. . .
flood attempts log file - Clear all entries in addurl or banned table - Truncate all tables in database Settings: - General Settings - Index Log Settings - Spider Settings - Search Settings - Order of Result listing - Suggest Options - Page Indexing Weights Database: - Configure up to 5 databases with unlimited number of table sets - Activate . . .
. . .
Database: - Configure up to 5 databases with unlimited number of table sets - Activate separately for 'Admin', 'Search' user and 'Suggest URL user' - Backup / Restore - Copy / Move - Optimize Templates: In order to enable customer's integration of Sphider-plus into existing sites, HTML templates are prepared for Search form Text result . . .
. . .
Search form Text result listing Media result listing Most popular queries etc. Three different designs are offered, which may be selected in submenu 'Settings'. If the layout does not fit the design of your site (which is normal), you may create new designs and modify the appropriate file /templates/My_template/adminstyle.css . . .
. . .
the appropriate file /templates/My_template/adminstyle.css /templates/My_template/userstyle.css Statistics output: - Top keywords (Top 50 with hit counter). - All indexed thumbnails w 3e80 ith ID3 and EXIF info. - Larges pages offering link URL and file size. - Most Popular Searches for text links offering: Link addr., total clicks, last . . .
. . .
Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Searches for media links offering: Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Links (click counter). - Search log offering: Query, Results, Queried at, Time taken, User IP, Country, Host name (Latest100) - Index log offering: . . .
. . .
Country, Host name (Latest100) - Index log offering: File-name, index date and delete option - sitemap log offering: sitemap.xml output sitemap list offering file/page suffixes - IDS log offering: IP, host, query, impact, involved tags, date and time of intrusion. - Flood attempts log offering: IP, query, date and time of flood attempt. - Auto . . .
. . .
attempts log offering: IP, query, date and time of flood attempt. - Auto Re-index log file - Server info offering: Server software, environment, MySQL, PDF-converter, image functions, php.ini file PHP integration, PHP security info. Each item holding lists of details. All text links, media links and thumbnails are active linked. As stated in . . .
. . .
individual Re-indexing, the periodical Re-indexer could be started and aborted in the "Options" menu of each site. 2.5 Preferred Re-indexing Each new URL added to the Admin backend, could be supplied with a priority level. This level will be used by the option 'Re-index only preferred sites'. Level 1 will be interpreted as most important, while . . .
. . .
less all index results will be stored in log files in sub folder /admin/log/ The names of the log files look like: db2_100524-21.47.56_1.html (log file of first thread) db2_100524-21.48.12_2.html (log file of second thread) and is build by the following items: db2 - Number of database. 100524 - Date (May 24, 2010) 21.47.56 - Time when this thread . . .
. . .
spider.php -new1 php spider.php -new2 etc. The IDs will be added to the name of the corresponding log files like: db2_100524-21.47.56_ID1.html (log file of first thread) db2_100524-21.48.12_ID2.html (log file of second thread) IDs could be defined by personal requirements, but the limitations for file names with respect to the OS should be taken . . .
. . .
but will not erase the content of all the other tables. So the check whether the content of a page has changed (MD5sum) is still available for a fast re-index procedure. Once prepared, multithreaded re-indexing could be invoked by starting several threads and adding individual IDs to the option parameter like: php spider.php -erased1 php . . .
. . .
4.4 Ignoring parts of a page Sphider-plus includes an option to exclude parts of pages from being indexed. Thi 775e s can for example be used to prevent search result flooding when certain keywords appear on certain part in most pages (like a header, footer or a menu). Any part of a page between <! sphider_noindex > and <! . . .
. . .
<! sphider_noindex > and <! /sphider_noindex > tags is not indexed, however links in it are followed. 4.5 Ignoring parts of a page by <div id='abc'> Ignoring parts of a page by the <! sphider_noindex > tags requires direct access to the page, because the tags need to be added (edited) to the page. A more flexible method, . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ menu[0-5] / 4.6 Indexing only parts of a page by <div id='abc'> If enabled in Admin settings, the values as defined in the list-file /include/common/divs_use.txt will be used to index only the content between <div . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ table[0-5] / 4.7 Ignore HTML elements defined by <tagname> . . </tagname> This option is foreseen to cooperate with the new HTML5 elements like section, nav, aside, hgroup, article, header, footer, etc. HTML elements . . .
. . .
contain a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */nav[0-5]/ Please keep in mind that element names placed in /include/common/elements_not.txt will be processed case-sensitive. 4.8 Index only HTML elements defined by <tagname> . . </tagname> This is the vice versa . . .
. . .
of all the duplicate content URLs: http://www.example.com/product.php?item=swedish-fish&category=gummy-candy http://www.example.com/product.php?item=swedish-fish&trackingid=1234&sessionid=5678 and Sphider-plus will understand that the duplicates all refer to the canonical URL: http://www.example.com/product.php?item=swedish-fish. The . . .
. . .
charset are supported by the ConvertCharset function and will be used to convert text into UTF-8 Unicode: WINDOWS windows-1250 - Central Europe windows-1251 - Cyrillic windows-1252 - Latin I windows-1253 - Greek windows-1254 - Turkish windows-1255 - Hebrew windows-1256 - Arabic windows-1257 - Baltic windows-1258 - Viet Nam cp874 - Thai - this . . .
. . .
- Baltic windows-1258 - Viet Nam cp874 - Thai - this file is also for DOS DOS cp437 - Latin US cp737 - Greek cp775 - BaltRim cp850 - Latin1 cp852 - Latin2 cp855 - Cyrylic cp857 - Turkish cp860 - Portuguese cp861 - Iceland cp862 - Hebrew cp863 - Canada cp864 - Arabic cp865 - Nordic cp866 - Cyrylic Russian (this is the one, used in IE . . .
. . .
IE Cyrillic (DOS) ) cp869 - Greek2 MAC (Apple) x-mac-cyrillic x-mac-greek x-mac-icelandic x-mac-ce x-mac-roman ISO iso-8859-1 iso-8859-2 iso-8859-3 iso-8859-4 iso-8859-5 iso-8859-6 iso-8859-7 iso-8859-8 iso-8859-9 iso-8859-10 iso-8859-11 iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 . . .
. . .
iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 cp500 cp856 cp875 cp1006 cp1026 koi8-r (Cyrillic) koi8-u (Cyrillic Ukrainian) nextstep us-ascii us-ascii-quotes DSP implementation for NeXT stdenc symbol zdingbat And specially for old Polish programs: mazovia This list is to be read . . .
. . .
Access denied; you need the RELOAD privilege. . . Top 10. Enable real-time output of logging data Up to version 1.5 of Sphider-plus, during index / re-index there was no printout available because: - Several servers, especially on Win32, buffer the output from the script until it terminates before transmitting the results to the browser. - . . .
. . .
is seen. - Some versions of Microsoft Internet Explorer only start to display the page after they have received 256 bytes of output. As progress was not presented during index / re- index procedure, waiting for results became a pain in the neck. Selectable in Admin setting together with the update interval (1 - 10 seconds), AJAX technology was . . .
. . .
characters in front of words Warning: This option should be used with special care and not be activated for non ISO-8859 charsets. Some special characters as part of the word ending might be erased by accidental. Top 13. Media search for images, audio streams and videos 13.1 Media indexing Index of media files is enabled by separated Admin . . .
. . .
and 'Found at' - Total clicks - Last clicked - Query input 'Indexed Image Thumbnails' presenting: - Thumbnail 150 x 100 pixel - Image details like title, filename size of original image, link- and thumb-id - Option to delete the thumbnail In order to open the media files all tables contain active links. Media results are also stored in . . .
. . .
Settings' menu of the admin backend: First option: For results of XML product feeds, present the text extract of 350 characters as part of product attributes. If this option is not activated, all products of the XML product feed will be presented in search results, and the hits of the query string will be highlighted in all involved products. . . .
. . .
content of the database tables (those with the same table prefix) will be destroyed by the restore procedure. 16.5 Copy & Move This section of the Database Management will present only those databases, which are correctly configured, assigned and do have a set of installed tables as described in chapter Definition and configuration. This . . .
. . .
results. Never the less to find these x results, the complete database had to been browsed. Starting with version 2.5, an additional clean option is offered as part of the Admin backend. Main advantage of this option is a significant reduction of the search time for any query, because the content of the db could be limited to offer only x . . .
. . .
<link>http://www.abc.de/images/warp.gif</link> <title>warp.gif</title> <x_size>635</x_size> <y_size>98</y_size> </media_result> <media_result> <num>2</num> <type>audio</type> <url>http://www.abc.de/index.php</url> . . .
. . .
the following content: {"query":"colour","ip":"::1","host_name":"guard_007-hoster","query_time":"2016-03-18 10:56:23 AM","consumed":0.016,"total_results":2,"num_of_results":2,"from":1,"to":2,"text_results":[{"num":1,"weight":"100.0","url":"http:\/\/www.english.le-piaggie.info\/Info_eng.pdf","title":" Info_eng","fulltxt":" . . . . . .
. . .
placed on the ground floor, today's big kitchen with the heating fireplace for open and close mode of . . . ","page_size":"5,661.6kb","domain_name":"www.english.le-piaggie.info"},{"num":2,"weight":"50.0","url":"http:\/\/www.english.le-piaggie.info\/html\/description.html","title":" Description","fulltxt":" . . . cable. Additional active . . .
. . .
D.O.C.G. as far as to the Pratomagno mountains 160 olive-trees Detached house, about 800 m to reach the . . . ","page_size":"26.4 kb","domain_name":"www.english.le-piaggie.info"}]} Top . . .


5a2 All required information. Introduction Release and Legal Info Installation Documentation Change Log [ Documentation Summary ] Preamble: The info presented here is valid only for the latest release of Sphider-plus. At present version 4.2024a published February 12, 2024 is the actual release. 1. Settings, customizing and statistics 2. Indexing . . .
. . .
2. Indexing 2.1 Various options 2.2 Allow other hosts in same domain 2.3 Word stemming 2.4 Periodical Re-indexing 2.5 Preferred indexing 2.6 Multithreaded indexing 2.7 Create thumbnails during index procedure 2.8 Prevent indexing of known malware and pishing pages 2.9 Follow and create sitemap files 2.10 Use private sitemap instead of global . . .
. . .
Sitemap file 3. Using the indexer from command line 3.1 All options 3.2 Multithreaded indexing 4. Keeping pages, words and files from being indexed 4.1 robots.txt 4.2 Must include / must not include string list 4.3 Ignoring links 4.4 Ignoring parts of a page by <! sphider_noindex > 4.5 Ignoring parts of a page by <div id='abc'> . . .
. . .
charset' 6. Search modes 6.1 Search with wildcards * 6.2 Strict search ! 6.3 Tolerant search 6.4 Link search 6.5 Media search 6.6 Search only in one domain 6.7 Search in categories 6.8 Greek language support 6.9 Block queries 7. Chronological order for result listing 7.1 Text result listing 7.2 Media result listing 8. PDF converter 9. Clean . . .
. . .
of logging data 11. Error messages and Debug mode 12. Delete secondary characters 13. Media search for images, audio streams and videos 13.1 Media indexing 13.2 Not supported media content 13.3 Search for media content 13.4 Statistics for media content 14. Feed support 14.1 XML product feeds 14.2 RDF, RSD, RSS and Atom feeds 15. Result . . .
. . .
Overview 16.2 Definition and configuration 16.3 Activate / disable database 16.4 Backup & Restore of databases 16.5 Copy & and Move 16.6 Enhancing functionality of multiple database support 17. Search in categories 17.1 Hierachical structure 17.2 Parallel structure 18. User suggested sites 19. Vulnerability protection 19.1 Prevent queries . . .
. . .
templates 22.2 Embed the search engine into existing HTML code 22.3 The different style sheet files 23. JSON, XML and RSS result output [ Documentation ] 1. Settings, customizing and statistics If you want to change settings, behavior and design of Sphider-plus, you can do so by means of the Admin interface. There is a wide range of . . .
. . .
interface. There is a wide range of settings foreseen for Sphider-plus. Separated into different submenus like: Sites: - Add Site - Index only the new - Re-index all - Re-index only preferred URLs - Erase Re-index (available also for individual URLs) - Import/export URL list - Approve sites - Banned domains Categories: - Add, edit, delete - . . .
. . .
URL list - Approve sites - Banned domains Categories: - Add, edit, delete - Create new subcategory under Index: - Basic indexing options - Advanced options Clean: - Clean keywords not associated with any link - Clean links not associated with any site - Clean Category table not associated with any site - Clean Media links - Clear Temp table . . .
. . .
- Clear Search log - Clear 'Most Popular Page Links' log - Clear 'Most Popular Media Links' log - Clear Spider log, separate and bulk delete - Clear Thumbnail images, separate and bulk delete - Clear Text cache - Clear Media cache - Clear IDS log file - Clear flood attempts log file - Clear all entries in addurl or banned table - Truncate all . . .
. . .
flood attempts log file - Clear all entries in addurl or banned table - Truncate all tables in database Settings: - General Settings - Index Log Settings - Spider Settings - Search Settings - Order of Result listing - Suggest Options - Page Indexing Weights Database: - Configure up to 5 databases with unlimited number of table sets - Activate . . .
. . .
Database: - Configure up to 5 databases with unlimited number of table sets - Activate separately for 'Admin', 'Search' user and 'Suggest URL user' - Backup / Restore - Copy / Move - Optimize Templates: In order to enable customer's integration of Sphider-plus into existing sites, HTML templates are prepared for Search form Text result . . .
. . .
Search form Text result listing Media result listing Most popular queries etc. Three different designs are offered, which may be selected in submenu 'Settings'. If the layout does not fit the design of your site (which is normal), you may create new designs and modify the appropriate file /templates/My_template/adminstyle.css . . .
. . .
the appropriate file /templates/My_template/adminstyle.css /templates/My_template/userstyle.css Statistics output: - Top keywords (Top 50 with hit counter). - All indexed thumbnails w 3e80 ith ID3 and EXIF info. - Larges pages offering link URL and file size. - Most Popular Searches for text links offering: Link addr., total clicks, last . . .
. . .
Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Searches for media links offering: Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Links (click counter). - Search log offering: Query, Results, Queried at, Time taken, User IP, Country, Host name (Latest100) - Index log offering: . . .
. . .
Country, Host name (Latest100) - Index log offering: File-name, index date and delete option - sitemap log offering: sitemap.xml output sitemap list offering file/page suffixes - IDS log offering: IP, host, query, impact, involved tags, date and time of intrusion. - Flood attempts log offering: IP, query, date and time of flood attempt. - Auto . . .
. . .
attempts log offering: IP, query, date and time of flood attempt. - Auto Re-index log file - Server info offering: Server software, environment, MySQL, PDF-converter, image functions, php.ini file PHP integration, PHP security info. Each item holding lists of details. All text links, media links and thumbnails are active linked. As stated in . . .
. . .
individual Re-indexing, the periodical Re-indexer could be started and aborted in the "Options" menu of each site. 2.5 Preferred Re-indexing Each new URL added to the Admin backend, could be supplied with a priority level. This level will be used by the option 'Re-index only preferred sites'. Level 1 will be interpreted as most important, while . . .
. . .
less all index results will be stored in log files in sub folder /admin/log/ The names of the log files look like: db2_100524-21.47.56_1.html (log file of first thread) db2_100524-21.48.12_2.html (log file of second thread) and is build by the following items: db2 - Number of database. 100524 - Date (May 24, 2010) 21.47.56 - Time when this thread . . .
. . .
spider.php -new1 php spider.php -new2 etc. The IDs will be added to the name of the corresponding log files like: db2_100524-21.47.56_ID1.html (log file of first thread) db2_100524-21.48.12_ID2.html (log file of second thread) IDs could be defined by personal requirements, but the limitations for file names with respect to the OS should be taken . . .
. . .
but will not erase the content of all the other tables. So the check whether the content of a page has changed (MD5sum) is still available for a fast re-index procedure. Once prepared, multithreaded re-indexing could be invoked by starting several threads and adding individual IDs to the option parameter like: php spider.php -erased1 php . . .
. . .
4.4 Ignoring parts of a page Sphider-plus includes an option to exclude parts of pages from being indexed. Thi 775e s can for example be used to prevent search result flooding when certain keywords appear on certain part in most pages (like a header, footer or a menu). Any part of a page between <! sphider_noindex > and <! . . .
. . .
<! sphider_noindex > and <! /sphider_noindex > tags is not indexed, however links in it are followed. 4.5 Ignoring parts of a page by <div id='abc'> Ignoring parts of a page by the <! sphider_noindex > tags requires direct access to the page, because the tags need to be added (edited) to the page. A more flexible method, . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ menu[0-5] / 4.6 Indexing only parts of a page by <div id='abc'> If enabled in Admin settings, the values as defined in the list-file /include/common/divs_use.txt will be used to index only the content between <div . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ table[0-5] / 4.7 Ignore HTML elements defined by <tagname> . . </tagname> This option is foreseen to cooperate with the new HTML5 elements like section, nav, aside, hgroup, article, header, footer, etc. HTML elements . . .
. . .
contain a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */nav[0-5]/ Please keep in mind that element names placed in /include/common/elements_not.txt will be processed case-sensitive. 4.8 Index only HTML elements defined by <tagname> . . </tagname> This is the vice versa . . .
. . .
of all the duplicate content URLs: http://www.example.com/product.php?item=swedish-fish&category=gummy-candy http://www.example.com/product.php?item=swedish-fish&trackingid=1234&sessionid=5678 and Sphider-plus will understand that the duplicates all refer to the canonical URL: http://www.example.com/product.php?item=swedish-fish. The . . .
. . .
charset are supported by the ConvertCharset function and will be used to convert text into UTF-8 Unicode: WINDOWS windows-1250 - Central Europe windows-1251 - Cyrillic windows-1252 - Latin I windows-1253 - Greek windows-1254 - Turkish windows-1255 - Hebrew windows-1256 - Arabic windows-1257 - Baltic windows-1258 - Viet Nam cp874 - Thai - this . . .
. . .
- Baltic windows-1258 - Viet Nam cp874 - Thai - this file is also for DOS DOS cp437 - Latin US cp737 - Greek cp775 - BaltRim cp850 - Latin1 cp852 - Latin2 cp855 - Cyrylic cp857 - Turkish cp860 - Portuguese cp861 - Iceland cp862 - Hebrew cp863 - Canada cp864 - Arabic cp865 - Nordic cp866 - Cyrylic Russian (this is the one, used in IE . . .
. . .
IE Cyrillic (DOS) ) cp869 - Greek2 MAC (Apple) x-mac-cyrillic x-mac-greek x-mac-icelandic x-mac-ce x-mac-roman ISO iso-8859-1 iso-8859-2 iso-8859-3 iso-8859-4 iso-8859-5 iso-8859-6 iso-8859-7 iso-8859-8 iso-8859-9 iso-8859-10 iso-8859-11 iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 . . .
. . .
iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 cp500 cp856 cp875 cp1006 cp1026 koi8-r (Cyrillic) koi8-u (Cyrillic Ukrainian) nextstep us-ascii us-ascii-quotes DSP implementation for NeXT stdenc symbol zdingbat And specially for old Polish programs: mazovia This list is to be read . . .
. . .
Access denied; you need the RELOAD privilege. . . Top 10. Enable real-time output of logging data Up to version 1.5 of Sphider-plus, during index / re-index there was no printout available because: - Several servers, especially on Win32, buffer the output from the script until it terminates before transmitting the results to the browser. - . . .
. . .
is seen. - Some versions of Microsoft Internet Explorer only start to display the page after they have received 256 bytes of output. As progress was not presented during index / re- index procedure, waiting for results became a pain in the neck. Selectable in Admin setting together with the update interval (1 - 10 seconds), AJAX technology was . . .
. . .
characters in front of words Warning: This option should be used with special care and not be activated for non ISO-8859 charsets. Some special characters as part of the word ending might be erased by accidental. Top 13. Media search for images, audio streams and videos 13.1 Media indexing Index of media files is enabled by separated Admin . . .
. . .
and 'Found at' - Total clicks - Last clicked - Query input 'Indexed Image Thumbnails' presenting: - Thumbnail 150 x 100 pixel - Image details like title, filename size of original image, link- and thumb-id - Option to delete the thumbnail In order to open the media files all tables contain active links. Media results are also stored in . . .
. . .
Settings' menu of the admin backend: First option: For results of XML product feeds, present the text extract of 350 characters as part of product attributes. If this option is not activated, all products of the XML product feed will be presented in search results, and the hits of the query string will be highlighted in all involved products. . . .
. . .
content of the database tables (those with the same table prefix) will be destroyed by the restore procedure. 16.5 Copy & Move This section of the Database Management will present only those databases, which are correctly configured, assigned and do have a set of installed tables as described in chapter Definition and configuration. This . . .
. . .
results. Never the less to find these x results, the complete database had to been browsed. Starting with version 2.5, an additional clean option is offered as part of the Admin backend. Main advantage of this option is a significant reduction of the search time for any query, because the content of the db could be limited to offer only x . . .
. . .
<link>http://www.abc.de/images/warp.gif</link> <title>warp.gif</title> <x_size>635</x_size> <y_size>98</y_size> </media_result> <media_result> <num>2</num> <type>audio</type> <url>http://www.abc.de/index.php</url> . . .
. . .
the following content: {"query":"colour","ip":"::1","host_name":"guard_007-hoster","query_time":"2016-03-18 10:56:23 AM","consumed":0.016,"total_results":2,"num_of_results":2,"from":1,"to":2,"text_results":[{"num":1,"weight":"100.0","url":"http:\/\/www.english.le-piaggie.info\/Info_eng.pdf","title":" Info_eng","fulltxt":" . . . . . .
. . .
placed on the ground floor, today's big kitchen with the heating fireplace for open and close mode of . . . ","page_size":"5,661.6kb","domain_name":"www.english.le-piaggie.info"},{"num":2,"weight":"50.0","url":"http:\/\/www.english.le-piaggie.info\/html\/description.html","title":" Description","fulltxt":" . . . cable. Additional active . . .
. . .
D.O.C.G. as far as to the Pratomagno mountains 160 olive-trees Detached house, about 800 m to reach the . . . ","page_size":"26.4 kb","domain_name":"www.english.le-piaggie.info"}]} Top . . .


5a2 All required information. Introduction Release and Legal Info Installation Documentation Change Log [ Documentation Summary ] Preamble: The info presented here is valid only for the latest release of Sphider-plus. At present version 4.2024a published February 12, 2024 is the actual release. 1. Settings, customizing and statistics 2. Indexing . . .
. . .
2. Indexing 2.1 Various options 2.2 Allow other hosts in same domain 2.3 Word stemming 2.4 Periodical Re-indexing 2.5 Preferred indexing 2.6 Multithreaded indexing 2.7 Create thumbnails during index procedure 2.8 Prevent indexing of known malware and pishing pages 2.9 Follow and create sitemap files 2.10 Use private sitemap instead of global . . .
. . .
Sitemap file 3. Using the indexer from command line 3.1 All options 3.2 Multithreaded indexing 4. Keeping pages, words and files from being indexed 4.1 robots.txt 4.2 Must include / must not include string list 4.3 Ignoring links 4.4 Ignoring parts of a page by <! sphider_noindex > 4.5 Ignoring parts of a page by <div id='abc'> . . .
. . .
charset' 6. Search modes 6.1 Search with wildcards * 6.2 Strict search ! 6.3 Tolerant search 6.4 Link search 6.5 Media search 6.6 Search only in one domain 6.7 Search in categories 6.8 Greek language support 6.9 Block queries 7. Chronological order for result listing 7.1 Text result listing 7.2 Media result listing 8. PDF converter 9. Clean . . .
. . .
of logging data 11. Error messages and Debug mode 12. Delete secondary characters 13. Media search for images, audio streams and videos 13.1 Media indexing 13.2 Not supported media content 13.3 Search for media content 13.4 Statistics for media content 14. Feed support 14.1 XML product feeds 14.2 RDF, RSD, RSS and Atom feeds 15. Result . . .
. . .
Overview 16.2 Definition and configuration 16.3 Activate / disable database 16.4 Backup & Restore of databases 16.5 Copy & and Move 16.6 Enhancing functionality of multiple database support 17. Search in categories 17.1 Hierachical structure 17.2 Parallel structure 18. User suggested sites 19. Vulnerability protection 19.1 Prevent queries . . .
. . .
templates 22.2 Embed the search engine into existing HTML code 22.3 The different style sheet files 23. JSON, XML and RSS result output [ Documentation ] 1. Settings, customizing and statistics If you want to change settings, behavior and design of Sphider-plus, you can do so by means of the Admin interface. There is a wide range of . . .
. . .
interface. There is a wide range of settings foreseen for Sphider-plus. Separated into different submenus like: Sites: - Add Site - Index only the new - Re-index all - Re-index only preferred URLs - Erase Re-index (available also for individual URLs) - Import/export URL list - Approve sites - Banned domains Categories: - Add, edit, delete - . . .
. . .
URL list - Approve sites - Banned domains Categories: - Add, edit, delete - Create new subcategory under Index: - Basic indexing options - Advanced options Clean: - Clean keywords not associated with any link - Clean links not associated with any site - Clean Category table not associated with any site - Clean Media links - Clear Temp table . . .
. . .
- Clear Search log - Clear 'Most Popular Page Links' log - Clear 'Most Popular Media Links' log - Clear Spider log, separate and bulk delete - Clear Thumbnail images, separate and bulk delete - Clear Text cache - Clear Media cache - Clear IDS log file - Clear flood attempts log file - Clear all entries in addurl or banned table - Truncate all . . .
. . .
flood attempts log file - Clear all entries in addurl or banned table - Truncate all tables in database Settings: - General Settings - Index Log Settings - Spider Settings - Search Settings - Order of Result listing - Suggest Options - Page Indexing Weights Database: - Configure up to 5 databases with unlimited number of table sets - Activate . . .
. . .
Database: - Configure up to 5 databases with unlimited number of table sets - Activate separately for 'Admin', 'Search' user and 'Suggest URL user' - Backup / Restore - Copy / Move - Optimize Templates: In order to enable customer's integration of Sphider-plus into existing sites, HTML templates are prepared for Search form Text result . . .
. . .
Search form Text result listing Media result listing Most popular queries etc. Three different designs are offered, which may be selected in submenu 'Settings'. If the layout does not fit the design of your site (which is normal), you may create new designs and modify the appropriate file /templates/My_template/adminstyle.css . . .
. . .
the appropriate file /templates/My_template/adminstyle.css /templates/My_template/userstyle.css Statistics output: - Top keywords (Top 50 with hit counter). - All indexed thumbnails w 3e80 ith ID3 and EXIF info. - Larges pages offering link URL and file size. - Most Popular Searches for text links offering: Link addr., total clicks, last . . .
. . .
Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Searches for media links offering: Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Links (click counter). - Search log offering: Query, Results, Queried at, Time taken, User IP, Country, Host name (Latest100) - Index log offering: . . .
. . .
Country, Host name (Latest100) - Index log offering: File-name, index date and delete option - sitemap log offering: sitemap.xml output sitemap list offering file/page suffixes - IDS log offering: IP, host, query, impact, involved tags, date and time of intrusion. - Flood attempts log offering: IP, query, date and time of flood attempt. - Auto . . .
. . .
attempts log offering: IP, query, date and time of flood attempt. - Auto Re-index log file - Server info offering: Server software, environment, MySQL, PDF-converter, image functions, php.ini file PHP integration, PHP security info. Each item holding lists of details. All text links, media links and thumbnails are active linked. As stated in . . .
. . .
individual Re-indexing, the periodical Re-indexer could be started and aborted in the "Options" menu of each site. 2.5 Preferred Re-indexing Each new URL added to the Admin backend, could be supplied with a priority level. This level will be used by the option 'Re-index only preferred sites'. Level 1 will be interpreted as most important, while . . .
. . .
less all index results will be stored in log files in sub folder /admin/log/ The names of the log files look like: db2_100524-21.47.56_1.html (log file of first thread) db2_100524-21.48.12_2.html (log file of second thread) and is build by the following items: db2 - Number of database. 100524 - Date (May 24, 2010) 21.47.56 - Time when this thread . . .
. . .
spider.php -new1 php spider.php -new2 etc. The IDs will be added to the name of the corresponding log files like: db2_100524-21.47.56_ID1.html (log file of first thread) db2_100524-21.48.12_ID2.html (log file of second thread) IDs could be defined by personal requirements, but the limitations for file names with respect to the OS should be taken . . .
. . .
but will not erase the content of all the other tables. So the check whether the content of a page has changed (MD5sum) is still available for a fast re-index procedure. Once prepared, multithreaded re-indexing could be invoked by starting several threads and adding individual IDs to the option parameter like: php spider.php -erased1 php . . .
. . .
4.4 Ignoring parts of a page Sphider-plus includes an option to exclude parts of pages from being indexed. Thi 775e s can for example be used to prevent search result flooding when certain keywords appear on certain part in most pages (like a header, footer or a menu). Any part of a page between <! sphider_noindex > and <! . . .
. . .
<! sphider_noindex > and <! /sphider_noindex > tags is not indexed, however links in it are followed. 4.5 Ignoring parts of a page by <div id='abc'> Ignoring parts of a page by the <! sphider_noindex > tags requires direct access to the page, because the tags need to be added (edited) to the page. A more flexible method, . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ menu[0-5] / 4.6 Indexing only parts of a page by <div id='abc'> If enabled in Admin settings, the values as defined in the list-file /include/common/divs_use.txt will be used to index only the content between <div . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ table[0-5] / 4.7 Ignore HTML elements defined by <tagname> . . </tagname> This option is foreseen to cooperate with the new HTML5 elements like section, nav, aside, hgroup, article, header, footer, etc. HTML elements . . .
. . .
contain a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */nav[0-5]/ Please keep in mind that element names placed in /include/common/elements_not.txt will be processed case-sensitive. 4.8 Index only HTML elements defined by <tagname> . . </tagname> This is the vice versa . . .
. . .
of all the duplicate content URLs: http://www.example.com/product.php?item=swedish-fish&category=gummy-candy http://www.example.com/product.php?item=swedish-fish&trackingid=1234&sessionid=5678 and Sphider-plus will understand that the duplicates all refer to the canonical URL: http://www.example.com/product.php?item=swedish-fish. The . . .
. . .
charset are supported by the ConvertCharset function and will be used to convert text into UTF-8 Unicode: WINDOWS windows-1250 - Central Europe windows-1251 - Cyrillic windows-1252 - Latin I windows-1253 - Greek windows-1254 - Turkish windows-1255 - Hebrew windows-1256 - Arabic windows-1257 - Baltic windows-1258 - Viet Nam cp874 - Thai - this . . .
. . .
- Baltic windows-1258 - Viet Nam cp874 - Thai - this file is also for DOS DOS cp437 - Latin US cp737 - Greek cp775 - BaltRim cp850 - Latin1 cp852 - Latin2 cp855 - Cyrylic cp857 - Turkish cp860 - Portuguese cp861 - Iceland cp862 - Hebrew cp863 - Canada cp864 - Arabic cp865 - Nordic cp866 - Cyrylic Russian (this is the one, used in IE . . .
. . .
IE Cyrillic (DOS) ) cp869 - Greek2 MAC (Apple) x-mac-cyrillic x-mac-greek x-mac-icelandic x-mac-ce x-mac-roman ISO iso-8859-1 iso-8859-2 iso-8859-3 iso-8859-4 iso-8859-5 iso-8859-6 iso-8859-7 iso-8859-8 iso-8859-9 iso-8859-10 iso-8859-11 iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 . . .
. . .
iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 cp500 cp856 cp875 cp1006 cp1026 koi8-r (Cyrillic) koi8-u (Cyrillic Ukrainian) nextstep us-ascii us-ascii-quotes DSP implementation for NeXT stdenc symbol zdingbat And specially for old Polish programs: mazovia This list is to be read . . .
. . .
Access denied; you need the RELOAD privilege. . . Top 10. Enable real-time output of logging data Up to version 1.5 of Sphider-plus, during index / re-index there was no printout available because: - Several servers, especially on Win32, buffer the output from the script until it terminates before transmitting the results to the browser. - . . .
. . .
is seen. - Some versions of Microsoft Internet Explorer only start to display the page after they have received 256 bytes of output. As progress was not presented during index / re- index procedure, waiting for results became a pain in the neck. Selectable in Admin setting together with the update interval (1 - 10 seconds), AJAX technology was . . .
. . .
characters in front of words Warning: This option should be used with special care and not be activated for non ISO-8859 charsets. Some special characters as part of the word ending might be erased by accidental. Top 13. Media search for images, audio streams and videos 13.1 Media indexing Index of media files is enabled by separated Admin . . .
. . .
and 'Found at' - Total clicks - Last clicked - Query input 'Indexed Image Thumbnails' presenting: - Thumbnail 150 x 100 pixel - Image details like title, filename size of original image, link- and thumb-id - Option to delete the thumbnail In order to open the media files all tables contain active links. Media results are also stored in . . .
. . .
Settings' menu of the admin backend: First option: For results of XML product feeds, present the text extract of 350 characters as part of product attributes. If this option is not activated, all products of the XML product feed will be presented in search results, and the hits of the query string will be highlighted in all involved products. . . .
. . .
content of the database tables (those with the same table prefix) will be destroyed by the restore procedure. 16.5 Copy & Move This section of the Database Management will present only those databases, which are correctly configured, assigned and do have a set of installed tables as described in chapter Definition and configuration. This . . .
. . .
results. Never the less to find these x results, the complete database had to been browsed. Starting with version 2.5, an additional clean option is offered as part of the Admin backend. Main advantage of this option is a significant reduction of the search time for any query, because the content of the db could be limited to offer only x . . .
. . .
<link>http://www.abc.de/images/warp.gif</link> <title>warp.gif</title> <x_size>635</x_size> <y_size>98</y_size> </media_result> <media_result> <num>2</num> <type>audio</type> <url>http://www.abc.de/index.php</url> . . .
. . .
the following content: {"query":"colour","ip":"::1","host_name":"guard_007-hoster","query_time":"2016-03-18 10:56:23 AM","consumed":0.016,"total_results":2,"num_of_results":2,"from":1,"to":2,"text_results":[{"num":1,"weight":"100.0","url":"http:\/\/www.english.le-piaggie.info\/Info_eng.pdf","title":" Info_eng","fulltxt":" . . . . . .
. . .
placed on the ground floor, today's big kitchen with the heating fireplace for open and close mode of . . . ","page_size":"5,661.6kb","domain_name":"www.english.le-piaggie.info"},{"num":2,"weight":"50.0","url":"http:\/\/www.english.le-piaggie.info\/html\/description.html","title":" Description","fulltxt":" . . . cable. Additional active . . .
. . .
D.O.C.G. as far as to the Pratomagno mountains 160 olive-trees Detached house, about 800 m to reach the . . . ","page_size":"26.4 kb","domain_name":"www.english.le-piaggie.info"}]} Top . . .


5a2 All required information. Introduction Release and Legal Info Installation Documentation Change Log [ Documentation Summary ] Preamble: The info presented here is valid only for the latest release of Sphider-plus. At present version 4.2024a published February 12, 2024 is the actual release. 1. Settings, customizing and statistics 2. Indexing . . .
. . .
2. Indexing 2.1 Various options 2.2 Allow other hosts in same domain 2.3 Word stemming 2.4 Periodical Re-indexing 2.5 Preferred indexing 2.6 Multithreaded indexing 2.7 Create thumbnails during index procedure 2.8 Prevent indexing of known malware and pishing pages 2.9 Follow and create sitemap files 2.10 Use private sitemap instead of global . . .
. . .
Sitemap file 3. Using the indexer from command line 3.1 All options 3.2 Multithreaded indexing 4. Keeping pages, words and files from being indexed 4.1 robots.txt 4.2 Must include / must not include string list 4.3 Ignoring links 4.4 Ignoring parts of a page by <! sphider_noindex > 4.5 Ignoring parts of a page by <div id='abc'> . . .
. . .
charset' 6. Search modes 6.1 Search with wildcards * 6.2 Strict search ! 6.3 Tolerant search 6.4 Link search 6.5 Media search 6.6 Search only in one domain 6.7 Search in categories 6.8 Greek language support 6.9 Block queries 7. Chronological order for result listing 7.1 Text result listing 7.2 Media result listing 8. PDF converter 9. Clean . . .
. . .
of logging data 11. Error messages and Debug mode 12. Delete secondary characters 13. Media search for images, audio streams and videos 13.1 Media indexing 13.2 Not supported media content 13.3 Search for media content 13.4 Statistics for media content 14. Feed support 14.1 XML product feeds 14.2 RDF, RSD, RSS and Atom feeds 15. Result . . .
. . .
Overview 16.2 Definition and configuration 16.3 Activate / disable database 16.4 Backup & Restore of databases 16.5 Copy & and Move 16.6 Enhancing functionality of multiple database support 17. Search in categories 17.1 Hierachical structure 17.2 Parallel structure 18. User suggested sites 19. Vulnerability protection 19.1 Prevent queries . . .
. . .
templates 22.2 Embed the search engine into existing HTML code 22.3 The different style sheet files 23. JSON, XML and RSS result output [ Documentation ] 1. Settings, customizing and statistics If you want to change settings, behavior and design of Sphider-plus, you can do so by means of the Admin interface. There is a wide range of . . .
. . .
interface. There is a wide range of settings foreseen for Sphider-plus. Separated into different submenus like: Sites: - Add Site - Index only the new - Re-index all - Re-index only preferred URLs - Erase Re-index (available also for individual URLs) - Import/export URL list - Approve sites - Banned domains Categories: - Add, edit, delete - . . .
. . .
URL list - Approve sites - Banned domains Categories: - Add, edit, delete - Create new subcategory under Index: - Basic indexing options - Advanced options Clean: - Clean keywords not associated with any link - Clean links not associated with any site - Clean Category table not associated with any site - Clean Media links - Clear Temp table . . .
. . .
- Clear Search log - Clear 'Most Popular Page Links' log - Clear 'Most Popular Media Links' log - Clear Spider log, separate and bulk delete - Clear Thumbnail images, separate and bulk delete - Clear Text cache - Clear Media cache - Clear IDS log file - Clear flood attempts log file - Clear all entries in addurl or banned table - Truncate all . . .
. . .
flood attempts log file - Clear all entries in addurl or banned table - Truncate all tables in database Settings: - General Settings - Index Log Settings - Spider Settings - Search Settings - Order of Result listing - Suggest Options - Page Indexing Weights Database: - Configure up to 5 databases with unlimited number of table sets - Activate . . .
. . .
Database: - Configure up to 5 databases with unlimited number of table sets - Activate separately for 'Admin', 'Search' user and 'Suggest URL user' - Backup / Restore - Copy / Move - Optimize Templates: In order to enable customer's integration of Sphider-plus into existing sites, HTML templates are prepared for Search form Text result . . .
. . .
Search form Text result listing Media result listing Most popular queries etc. Three different designs are offered, which may be selected in submenu 'Settings'. If the layout does not fit the design of your site (which is normal), you may create new designs and modify the appropriate file /templates/My_template/adminstyle.css . . .
. . .
the appropriate file /templates/My_template/adminstyle.css /templates/My_template/userstyle.css Statistics output: - Top keywords (Top 50 with hit counter). - All indexed thumbnails w 3e80 ith ID3 and EXIF info. - Larges pages offering link URL and file size. - Most Popular Searches for text links offering: Link addr., total clicks, last . . .
. . .
Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Searches for media links offering: Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Links (click counter). - Search log offering: Query, Results, Queried at, Time taken, User IP, Country, Host name (Latest100) - Index log offering: . . .
. . .
Country, Host name (Latest100) - Index log offering: File-name, index date and delete option - sitemap log offering: sitemap.xml output sitemap list offering file/page suffixes - IDS log offering: IP, host, query, impact, involved tags, date and time of intrusion. - Flood attempts log offering: IP, query, date and time of flood attempt. - Auto . . .
. . .
attempts log offering: IP, query, date and time of flood attempt. - Auto Re-index log file - Server info offering: Server software, environment, MySQL, PDF-converter, image functions, php.ini file PHP integration, PHP security info. Each item holding lists of details. All text links, media links and thumbnails are active linked. As stated in . . .
. . .
individual Re-indexing, the periodical Re-indexer could be started and aborted in the "Options" menu of each site. 2.5 Preferred Re-indexing Each new URL added to the Admin backend, could be supplied with a priority level. This level will be used by the option 'Re-index only preferred sites'. Level 1 will be interpreted as most important, while . . .
. . .
less all index results will be stored in log files in sub folder /admin/log/ The names of the log files look like: db2_100524-21.47.56_1.html (log file of first thread) db2_100524-21.48.12_2.html (log file of second thread) and is build by the following items: db2 - Number of database. 100524 - Date (May 24, 2010) 21.47.56 - Time when this thread . . .
. . .
spider.php -new1 php spider.php -new2 etc. The IDs will be added to the name of the corresponding log files like: db2_100524-21.47.56_ID1.html (log file of first thread) db2_100524-21.48.12_ID2.html (log file of second thread) IDs could be defined by personal requirements, but the limitations for file names with respect to the OS should be taken . . .
. . .
but will not erase the content of all the other tables. So the check whether the content of a page has changed (MD5sum) is still available for a fast re-index procedure. Once prepared, multithreaded re-indexing could be invoked by starting several threads and adding individual IDs to the option parameter like: php spider.php -erased1 php . . .
. . .
4.4 Ignoring parts of a page Sphider-plus includes an option to exclude parts of pages from being indexed. Thi 775e s can for example be used to prevent search result flooding when certain keywords appear on certain part in most pages (like a header, footer or a menu). Any part of a page between <! sphider_noindex > and <! . . .
. . .
<! sphider_noindex > and <! /sphider_noindex > tags is not indexed, however links in it are followed. 4.5 Ignoring parts of a page by <div id='abc'> Ignoring parts of a page by the <! sphider_noindex > tags requires direct access to the page, because the tags need to be added (edited) to the page. A more flexible method, . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ menu[0-5] / 4.6 Indexing only parts of a page by <div id='abc'> If enabled in Admin settings, the values as defined in the list-file /include/common/divs_use.txt will be used to index only the content between <div . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ table[0-5] / 4.7 Ignore HTML elements defined by <tagname> . . </tagname> This option is foreseen to cooperate with the new HTML5 elements like section, nav, aside, hgroup, article, header, footer, etc. HTML elements . . .
. . .
contain a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */nav[0-5]/ Please keep in mind that element names placed in /include/common/elements_not.txt will be processed case-sensitive. 4.8 Index only HTML elements defined by <tagname> . . </tagname> This is the vice versa . . .
. . .
of all the duplicate content URLs: http://www.example.com/product.php?item=swedish-fish&category=gummy-candy http://www.example.com/product.php?item=swedish-fish&trackingid=1234&sessionid=5678 and Sphider-plus will understand that the duplicates all refer to the canonical URL: http://www.example.com/product.php?item=swedish-fish. The . . .
. . .
charset are supported by the ConvertCharset function and will be used to convert text into UTF-8 Unicode: WINDOWS windows-1250 - Central Europe windows-1251 - Cyrillic windows-1252 - Latin I windows-1253 - Greek windows-1254 - Turkish windows-1255 - Hebrew windows-1256 - Arabic windows-1257 - Baltic windows-1258 - Viet Nam cp874 - Thai - this . . .
. . .
- Baltic windows-1258 - Viet Nam cp874 - Thai - this file is also for DOS DOS cp437 - Latin US cp737 - Greek cp775 - BaltRim cp850 - Latin1 cp852 - Latin2 cp855 - Cyrylic cp857 - Turkish cp860 - Portuguese cp861 - Iceland cp862 - Hebrew cp863 - Canada cp864 - Arabic cp865 - Nordic cp866 - Cyrylic Russian (this is the one, used in IE . . .
. . .
IE Cyrillic (DOS) ) cp869 - Greek2 MAC (Apple) x-mac-cyrillic x-mac-greek x-mac-icelandic x-mac-ce x-mac-roman ISO iso-8859-1 iso-8859-2 iso-8859-3 iso-8859-4 iso-8859-5 iso-8859-6 iso-8859-7 iso-8859-8 iso-8859-9 iso-8859-10 iso-8859-11 iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 . . .
. . .
iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 cp500 cp856 cp875 cp1006 cp1026 koi8-r (Cyrillic) koi8-u (Cyrillic Ukrainian) nextstep us-ascii us-ascii-quotes DSP implementation for NeXT stdenc symbol zdingbat And specially for old Polish programs: mazovia This list is to be read . . .
. . .
Access denied; you need the RELOAD privilege. . . Top 10. Enable real-time output of logging data Up to version 1.5 of Sphider-plus, during index / re-index there was no printout available because: - Several servers, especially on Win32, buffer the output from the script until it terminates before transmitting the results to the browser. - . . .
. . .
is seen. - Some versions of Microsoft Internet Explorer only start to display the page after they have received 256 bytes of output. As progress was not presented during index / re- index procedure, waiting for results became a pain in the neck. Selectable in Admin setting together with the update interval (1 - 10 seconds), AJAX technology was . . .
. . .
characters in front of words Warning: This option should be used with special care and not be activated for non ISO-8859 charsets. Some special characters as part of the word ending might be erased by accidental. Top 13. Media search for images, audio streams and videos 13.1 Media indexing Index of media files is enabled by separated Admin . . .
. . .
and 'Found at' - Total clicks - Last clicked - Query input 'Indexed Image Thumbnails' presenting: - Thumbnail 150 x 100 pixel - Image details like title, filename size of original image, link- and thumb-id - Option to delete the thumbnail In order to open the media files all tables contain active links. Media results are also stored in . . .
. . .
Settings' menu of the admin backend: First option: For results of XML product feeds, present the text extract of 350 characters as part of product attributes. If this option is not activated, all products of the XML product feed will be presented in search results, and the hits of the query string will be highlighted in all involved products. . . .
. . .
content of the database tables (those with the same table prefix) will be destroyed by the restore procedure. 16.5 Copy & Move This section of the Database Management will present only those databases, which are correctly configured, assigned and do have a set of installed tables as described in chapter Definition and configuration. This . . .
. . .
results. Never the less to find these x results, the complete database had to been browsed. Starting with version 2.5, an additional clean option is offered as part of the Admin backend. Main advantage of this option is a significant reduction of the search time for any query, because the content of the db could be limited to offer only x . . .
. . .
<link>http://www.abc.de/images/warp.gif</link> <title>warp.gif</title> <x_size>635</x_size> <y_size>98</y_size> </media_result> <media_result> <num>2</num> <type>audio</type> <url>http://www.abc.de/index.php</url> . . .
. . .
the following content: {"query":"colour","ip":"::1","host_name":"guard_007-hoster","query_time":"2016-03-18 10:56:23 AM","consumed":0.016,"total_results":2,"num_of_results":2,"from":1,"to":2,"text_results":[{"num":1,"weight":"100.0","url":"http:\/\/www.english.le-piaggie.info\/Info_eng.pdf","title":" Info_eng","fulltxt":" . . . . . .
. . .
placed on the ground floor, today's big kitchen with the heating fireplace for open and close mode of . . . ","page_size":"5,661.6kb","domain_name":"www.english.le-piaggie.info"},{"num":2,"weight":"50.0","url":"http:\/\/www.english.le-piaggie.info\/html\/description.html","title":" Description","fulltxt":" . . . cable. Additional active . . .
. . .
D.O.C.G. as far as to the Pratomagno mountains 160 olive-trees Detached house, about 800 m to reach the . . . ","page_size":"26.4 kb","domain_name":"www.english.le-piaggie.info"}]} Top . . .


5a2 All required information. Introduction Release and Legal Info Installation Documentation Change Log [ Documentation Summary ] Preamble: The info presented here is valid only for the latest release of Sphider-plus. At present version 4.2024a published February 12, 2024 is the actual release. 1. Settings, customizing and statistics 2. Indexing . . .
. . .
2. Indexing 2.1 Various options 2.2 Allow other hosts in same domain 2.3 Word stemming 2.4 Periodical Re-indexing 2.5 Preferred indexing 2.6 Multithreaded indexing 2.7 Create thumbnails during index procedure 2.8 Prevent indexing of known malware and pishing pages 2.9 Follow and create sitemap files 2.10 Use private sitemap instead of global . . .
. . .
Sitemap file 3. Using the indexer from command line 3.1 All options 3.2 Multithreaded indexing 4. Keeping pages, words and files from being indexed 4.1 robots.txt 4.2 Must include / must not include string list 4.3 Ignoring links 4.4 Ignoring parts of a page by <! sphider_noindex > 4.5 Ignoring parts of a page by <div id='abc'> . . .
. . .
charset' 6. Search modes 6.1 Search with wildcards * 6.2 Strict search ! 6.3 Tolerant search 6.4 Link search 6.5 Media search 6.6 Search only in one domain 6.7 Search in categories 6.8 Greek language support 6.9 Block queries 7. Chronological order for result listing 7.1 Text result listing 7.2 Media result listing 8. PDF converter 9. Clean . . .
. . .
of logging data 11. Error messages and Debug mode 12. Delete secondary characters 13. Media search for images, audio streams and videos 13.1 Media indexing 13.2 Not supported media content 13.3 Search for media content 13.4 Statistics for media content 14. Feed support 14.1 XML product feeds 14.2 RDF, RSD, RSS and Atom feeds 15. Result . . .
. . .
Overview 16.2 Definition and configuration 16.3 Activate / disable database 16.4 Backup & Restore of databases 16.5 Copy & and Move 16.6 Enhancing functionality of multiple database support 17. Search in categories 17.1 Hierachical structure 17.2 Parallel structure 18. User suggested sites 19. Vulnerability protection 19.1 Prevent queries . . .
. . .
templates 22.2 Embed the search engine into existing HTML code 22.3 The different style sheet files 23. JSON, XML and RSS result output [ Documentation ] 1. Settings, customizing and statistics If you want to change settings, behavior and design of Sphider-plus, you can do so by means of the Admin interface. There is a wide range of . . .
. . .
interface. There is a wide range of settings foreseen for Sphider-plus. Separated into different submenus like: Sites: - Add Site - Index only the new - Re-index all - Re-index only preferred URLs - Erase Re-index (available also for individual URLs) - Import/export URL list - Approve sites - Banned domains Categories: - Add, edit, delete - . . .
. . .
URL list - Approve sites - Banned domains Categories: - Add, edit, delete - Create new subcategory under Index: - Basic indexing options - Advanced options Clean: - Clean keywords not associated with any link - Clean links not associated with any site - Clean Category table not associated with any site - Clean Media links - Clear Temp table . . .
. . .
- Clear Search log - Clear 'Most Popular Page Links' log - Clear 'Most Popular Media Links' log - Clear Spider log, separate and bulk delete - Clear Thumbnail images, separate and bulk delete - Clear Text cache - Clear Media cache - Clear IDS log file - Clear flood attempts log file - Clear all entries in addurl or banned table - Truncate all . . .
. . .
flood attempts log file - Clear all entries in addurl or banned table - Truncate all tables in database Settings: - General Settings - Index Log Settings - Spider Settings - Search Settings - Order of Result listing - Suggest Options - Page Indexing Weights Database: - Configure up to 5 databases with unlimited number of table sets - Activate . . .
. . .
Database: - Configure up to 5 databases with unlimited number of table sets - Activate separately for 'Admin', 'Search' user and 'Suggest URL user' - Backup / Restore - Copy / Move - Optimize Templates: In order to enable customer's integration of Sphider-plus into existing sites, HTML templates are prepared for Search form Text result . . .
. . .
Search form Text result listing Media result listing Most popular queries etc. Three different designs are offered, which may be selected in submenu 'Settings'. If the layout does not fit the design of your site (which is normal), you may create new designs and modify the appropriate file /templates/My_template/adminstyle.css . . .
. . .
the appropriate file /templates/My_template/adminstyle.css /templates/My_template/userstyle.css Statistics output: - Top keywords (Top 50 with hit counter). - All indexed thumbnails w 3e80 ith ID3 and EXIF info. - Larges pages offering link URL and file size. - Most Popular Searches for text links offering: Link addr., total clicks, last . . .
. . .
Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Searches for media links offering: Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Links (click counter). - Search log offering: Query, Results, Queried at, Time taken, User IP, Country, Host name (Latest100) - Index log offering: . . .
. . .
Country, Host name (Latest100) - Index log offering: File-name, index date and delete option - sitemap log offering: sitemap.xml output sitemap list offering file/page suffixes - IDS log offering: IP, host, query, impact, involved tags, date and time of intrusion. - Flood attempts log offering: IP, query, date and time of flood attempt. - Auto . . .
. . .
attempts log offering: IP, query, date and time of flood attempt. - Auto Re-index log file - Server info offering: Server software, environment, MySQL, PDF-converter, image functions, php.ini file PHP integration, PHP security info. Each item holding lists of details. All text links, media links and thumbnails are active linked. As stated in . . .
. . .
individual Re-indexing, the periodical Re-indexer could be started and aborted in the "Options" menu of each site. 2.5 Preferred Re-indexing Each new URL added to the Admin backend, could be supplied with a priority level. This level will be used by the option 'Re-index only preferred sites'. Level 1 will be interpreted as most important, while . . .
. . .
less all index results will be stored in log files in sub folder /admin/log/ The names of the log files look like: db2_100524-2147561html-21.47.56_1.html (log file of first thread) db2_100524-2147561html-21.48.12_2.html (log file of second thread) and is build by the following items: db2 - Number of database. 100524-2147561html - Date (May 24, 2010) 21.47.56 - Time when this thread . . .
. . .
spider.php -new1 php spider.php -new2 etc. The IDs will be added to the name of the corresponding log files like: db2_100524-2147561html-21.47.56_ID1.html (log file of first thread) db2_100524-2147561html-21.48.12_ID2.html (log file of second thread) IDs could be defined by personal requirements, but the limitations for file names with respect to the OS should be taken . . .
. . .
but will not erase the content of all the other tables. So the check whether the content of a page has changed (MD5sum) is still available for a fast re-index procedure. Once prepared, multithreaded re-indexing could be invoked by starting several threads and adding individual IDs to the option parameter like: php spider.php -erased1 php . . .
. . .
4.4 Ignoring parts of a page Sphider-plus includes an option to exclude parts of pages from being indexed. Thi 775e s can for example be used to prevent search result flooding when certain keywords appear on certain part in most pages (like a header, footer or a menu). Any part of a page between <! sphider_noindex > and <! . . .
. . .
<! sphider_noindex > and <! /sphider_noindex > tags is not indexed, however links in it are followed. 4.5 Ignoring parts of a page by <div id='abc'> Ignoring parts of a page by the <! sphider_noindex > tags requires direct access to the page, because the tags need to be added (edited) to the page. A more flexible method, . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ menu[0-5] / 4.6 Indexing only parts of a page by <div id='abc'> If enabled in Admin settings, the values as defined in the list-file /include/common/divs_use.txt will be used to index only the content between <div . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ table[0-5] / 4.7 Ignore HTML elements defined by <tagname> . . </tagname> This option is foreseen to cooperate with the new HTML5 elements like section, nav, aside, hgroup, article, header, footer, etc. HTML elements . . .
. . .
contain a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */nav[0-5]/ Please keep in mind that element names placed in /include/common/elements_not.txt will be processed case-sensitive. 4.8 Index only HTML elements defined by <tagname> . . </tagname> This is the vice versa . . .
. . .
of all the duplicate content URLs: http://www.example.com/product.php?item=swedish-fish&category=gummy-candy http://www.example.com/product.php?item=swedish-fish&trackingid=1234&sessionid=5678 and Sphider-plus will understand that the duplicates all refer to the canonical URL: http://www.example.com/product.php?item=swedish-fish. The . . .
. . .
charset are supported by the ConvertCharset function and will be used to convert text into UTF-8 Unicode: WINDOWS windows-1250 - Central Europe windows-1251 - Cyrillic windows-1252 - Latin I windows-1253 - Greek windows-1254 - Turkish windows-1255 - Hebrew windows-1256 - Arabic windows-1257 - Baltic windows-1258 - Viet Nam cp874 - Thai - this . . .
. . .
- Baltic windows-1258 - Viet Nam cp874 - Thai - this file is also for DOS DOS cp437 - Latin US cp737 - Greek cp775 - BaltRim cp850 - Latin1 cp852 - Latin2 cp855 - Cyrylic cp857 - Turkish cp860 - Portuguese cp861 - Iceland cp862 - Hebrew cp863 - Canada cp864 - Arabic cp865 - Nordic cp866 - Cyrylic Russian (this is the one, used in IE . . .
. . .
IE Cyrillic (DOS) ) cp869 - Greek2 MAC (Apple) x-mac-cyrillic x-mac-greek x-mac-icelandic x-mac-ce x-mac-roman ISO iso-8859-1 iso-8859-2 iso-8859-3 iso-8859-4 iso-8859-5 iso-8859-6 iso-8859-7 iso-8859-8 iso-8859-9 iso-8859-10 iso-8859-11 iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 . . .
. . .
iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 cp500 cp856 cp875 cp1006 cp1026 koi8-r (Cyrillic) koi8-u (Cyrillic Ukrainian) nextstep us-ascii us-ascii-quotes DSP implementation for NeXT stdenc symbol zdingbat And specially for old Polish programs: mazovia This list is to be read . . .
. . .
Access denied; you need the RELOAD privilege. . . Top 10. Enable real-time output of logging data Up to version 1.5 of Sphider-plus, during index / re-index there was no printout available because: - Several servers, especially on Win32, buffer the output from the script until it terminates before transmitting the results to the browser. - . . .
. . .
is seen. - Some versions of Microsoft Internet Explorer only start to display the page after they have received 256 bytes of output. As progress was not presented during index / re- index procedure, waiting for results became a pain in the neck. Selectable in Admin setting together with the update interval (1 - 10 seconds), AJAX technology was . . .
. . .
characters in front of words Warning: This option should be used with special care and not be activated for non ISO-8859 charsets. Some special characters as part of the word ending might be erased by accidental. Top 13. Media search for images, audio streams and videos 13.1 Media indexing Index of media files is enabled by separated Admin . . .
. . .
and 'Found at' - Total clicks - Last clicked - Query input 'Indexed Image Thumbnails' presenting: - Thumbnail 150 x 100 pixel - Image details like title, filename size of original image, link- and thumb-id - Option to delete the thumbnail In order to open the media files all tables contain active links. Media results are also stored in . . .
. . .
Settings' menu of the admin backend: First option: For results of XML product feeds, present the text extract of 350 characters as part of product attributes. If this option is not activated, all products of the XML product feed will be presented in search results, and the hits of the query string will be highlighted in all involved products. . . .
. . .
content of the database tables (those with the same table prefix) will be destroyed by the restore procedure. 16.5 Copy & Move This section of the Database Management will present only those databases, which are correctly configured, assigned and do have a set of installed tables as described in chapter Definition and configuration. This . . .
. . .
results. Never the less to find these x results, the complete database had to been browsed. Starting with version 2.5, an additional clean option is offered as part of the Admin backend. Main advantage of this option is a significant reduction of the search time for any query, because the content of the db could be limited to offer only x . . .
. . .
<link>http://www.abc.de/images/warp.gif</link> <title>warp.gif</title> <x_size>635</x_size> <y_size>98</y_size> </media_result> <media_result> <num>2</num> <type>audio</type> <url>http://www.abc.de/index.php</url> . . .
. . .
the following content: {"query":"colour","ip":"::1","host_name":"guard_007-hoster","query_time":"2016-03-18 10:56:23 AM","consumed":0.016,"total_results":2,"num_of_results":2,"from":1,"to":2,"text_results":[{"num":1,"weight":"100.0","url":"http:\/\/www.english.le-piaggie.info\/Info_eng.pdf","title":" Info_eng","fulltxt":" . . . . . .
. . .
placed on the ground floor, today's big kitchen with the heating fireplace for open and close mode of . . . ","page_size":"5,661.6kb","domain_name":"www.english.le-piaggie.info"},{"num":2,"weight":"50.0","url":"http:\/\/www.english.le-piaggie.info\/html\/description.html","title":" Description","fulltxt":" . . . cable. Additional active . . .
. . .
D.O.C.G. as far as to the Pratomagno mountains 160 olive-trees Detached house, about 800 m to reach the . . . ","page_size":"26.4 kb","domain_name":"www.english.le-piaggie.info"}]} Top . . .


5a2 All required information. Introduction Release and Legal Info Installation Documentation Change Log [ Documentation Summary ] Preamble: The info presented here is valid only for the latest release of Sphider-plus. At present version 4.2024a published February 12, 2024 is the actual release. 1. Settings, customizing and statistics 2. Indexing . . .
. . .
2. Indexing 2.1 Various options 2.2 Allow other hosts in same domain 2.3 Word stemming 2.4 Periodical Re-indexing 2.5 Preferred indexing 2.6 Multithreaded indexing 2.7 Create thumbnails during index procedure 2.8 Prevent indexing of known malware and pishing pages 2.9 Follow and create sitemap files 2.10 Use private sitemap instead of global . . .
. . .
Sitemap file 3. Using the indexer from command line 3.1 All options 3.2 Multithreaded indexing 4. Keeping pages, words and files from being indexed 4.1 robots.txt 4.2 Must include / must not include string list 4.3 Ignoring links 4.4 Ignoring parts of a page by <! sphider_noindex > 4.5 Ignoring parts of a page by <div id='abc'> . . .
. . .
charset' 6. Search modes 6.1 Search with wildcards * 6.2 Strict search ! 6.3 Tolerant search 6.4 Link search 6.5 Media search 6.6 Search only in one domain 6.7 Search in categories 6.8 Greek language support 6.9 Block queries 7. Chronological order for result listing 7.1 Text result listing 7.2 Media result listing 8. PDF converter 9. Clean . . .
. . .
of logging data 11. Error messages and Debug mode 12. Delete secondary characters 13. Media search for images, audio streams and videos 13.1 Media indexing 13.2 Not supported media content 13.3 Search for media content 13.4 Statistics for media content 14. Feed support 14.1 XML product feeds 14.2 RDF, RSD, RSS and Atom feeds 15. Result . . .
. . .
Overview 16.2 Definition and configuration 16.3 Activate / disable database 16.4 Backup & Restore of databases 16.5 Copy & and Move 16.6 Enhancing functionality of multiple database support 17. Search in categories 17.1 Hierachical structure 17.2 Parallel structure 18. User suggested sites 19. Vulnerability protection 19.1 Prevent queries . . .
. . .
templates 22.2 Embed the search engine into existing HTML code 22.3 The different style sheet files 23. JSON, XML and RSS result output [ Documentation ] 1. Settings, customizing and statistics If you want to change settings, behavior and design of Sphider-plus, you can do so by means of the Admin interface. There is a wide range of . . .
. . .
interface. There is a wide range of settings foreseen for Sphider-plus. Separated into different submenus like: Sites: - Add Site - Index only the new - Re-index all - Re-index only preferred URLs - Erase Re-index (available also for individual URLs) - Import/export URL list - Approve sites - Banned domains Categories: - Add, edit, delete - . . .
. . .
URL list - Approve sites - Banned domains Categories: - Add, edit, delete - Create new subcategory under Index: - Basic indexing options - Advanced options Clean: - Clean keywords not associated with any link - Clean links not associated with any site - Clean Category table not associated with any site - Clean Media links - Clear Temp table . . .
. . .
- Clear Search log - Clear 'Most Popular Page Links' log - Clear 'Most Popular Media Links' log - Clear Spider log, separate and bulk delete - Clear Thumbnail images, separate and bulk delete - Clear Text cache - Clear Media cache - Clear IDS log file - Clear flood attempts log file - Clear all entries in addurl or banned table - Truncate all . . .
. . .
flood attempts log file - Clear all entries in addurl or banned table - Truncate all tables in database Settings: - General Settings - Index Log Settings - Spider Settings - Search Settings - Order of Result listing - Suggest Options - Page Indexing Weights Database: - Configure up to 5 databases with unlimited number of table sets - Activate . . .
. . .
Database: - Configure up to 5 databases with unlimited number of table sets - Activate separately for 'Admin', 'Search' user and 'Suggest URL user' - Backup / Restore - Copy / Move - Optimize Templates: In order to enable customer's integration of Sphider-plus into existing sites, HTML templates are prepared for Search form Text result . . .
. . .
Search form Text result listing Media result listing Most popular queries etc. Three different designs are offered, which may be selected in submenu 'Settings'. If the layout does not fit the design of your site (which is normal), you may create new designs and modify the appropriate file /templates/My_template/adminstyle.css . . .
. . .
the appropriate file /templates/My_template/adminstyle.css /templates/My_template/userstyle.css Statistics output: - Top keywords (Top 50 with hit counter). - All indexed thumbnails w 3e80 ith ID3 and EXIF info. - Larges pages offering link URL and file size. - Most Popular Searches for text links offering: Link addr., total clicks, last . . .
. . .
Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Searches for media links offering: Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Links (click counter). - Search log offering: Query, Results, Queried at, Time taken, User IP, Country, Host name (Latest100) - Index log offering: . . .
. . .
Country, Host name (Latest100) - Index log offering: File-name, index date and delete option - sitemap log offering: sitemap.xml output sitemap list offering file/page suffixes - IDS log offering: IP, host, query, impact, involved tags, date and time of intrusion. - Flood attempts log offering: IP, query, date and time of flood attempt. - Auto . . .
. . .
attempts log offering: IP, query, date and time of flood attempt. - Auto Re-index log file - Server info offering: Server software, environment, MySQL, PDF-converter, image functions, php.ini file PHP integration, PHP security info. Each item holding lists of details. All text links, media links and thumbnails are active linked. As stated in . . .
. . .
individual Re-indexing, the periodical Re-indexer could be started and aborted in the "Options" menu of each site. 2.5 Preferred Re-indexing Each new URL added to the Admin backend, could be supplied with a priority level. This level will be used by the option 'Re-index only preferred sites'. Level 1 will be interpreted as most important, while . . .
. . .
less all index results will be stored in log files in sub folder /admin/log/ The names of the log files look like: db2_100524-2147561html_1.html (log file of first thread) db2_100524-21.48.12_2.html (log file of second thread) and is build by the following items: db2 - Number of database. 100524 - Date (May 24, 2010) 2147561html - Time when this thread . . .
. . .
spider.php -new1 php spider.php -new2 etc. The IDs will be added to the name of the corresponding log files like: db2_100524-2147561html_ID1.html (log file of first thread) db2_100524-21.48.12_ID2.html (log file of second thread) IDs could be defined by personal requirements, but the limitations for file names with respect to the OS should be taken . . .
. . .
but will not erase the content of all the other tables. So the check whether the content of a page has changed (MD5sum) is still available for a fast re-index procedure. Once prepared, multithreaded re-indexing could be invoked by starting several threads and adding individual IDs to the option parameter like: php spider.php -erased1 php . . .
. . .
4.4 Ignoring parts of a page Sphider-plus includes an option to exclude parts of pages from being indexed. Thi 775e s can for example be used to prevent search result flooding when certain keywords appear on certain part in most pages (like a header, footer or a menu). Any part of a page between <! sphider_noindex > and <! . . .
. . .
<! sphider_noindex > and <! /sphider_noindex > tags is not indexed, however links in it are followed. 4.5 Ignoring parts of a page by <div id='abc'> Ignoring parts of a page by the <! sphider_noindex > tags requires direct access to the page, because the tags need to be added (edited) to the page. A more flexible method, . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ menu[0-5] / 4.6 Indexing only parts of a page by <div id='abc'> If enabled in Admin settings, the values as defined in the list-file /include/common/divs_use.txt will be used to index only the content between <div . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ table[0-5] / 4.7 Ignore HTML elements defined by <tagname> . . </tagname> This option is foreseen to cooperate with the new HTML5 elements like section, nav, aside, hgroup, article, header, footer, etc. HTML elements . . .
. . .
contain a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */nav[0-5]/ Please keep in mind that element names placed in /include/common/elements_not.txt will be processed case-sensitive. 4.8 Index only HTML elements defined by <tagname> . . </tagname> This is the vice versa . . .
. . .
of all the duplicate content URLs: http://www.example.com/product.php?item=swedish-fish&category=gummy-candy http://www.example.com/product.php?item=swedish-fish&trackingid=1234&sessionid=5678 and Sphider-plus will understand that the duplicates all refer to the canonical URL: http://www.example.com/product.php?item=swedish-fish. The . . .
. . .
charset are supported by the ConvertCharset function and will be used to convert text into UTF-8 Unicode: WINDOWS windows-1250 - Central Europe windows-1251 - Cyrillic windows-1252 - Latin I windows-1253 - Greek windows-1254 - Turkish windows-1255 - Hebrew windows-1256 - Arabic windows-1257 - Baltic windows-1258 - Viet Nam cp874 - Thai - this . . .
. . .
- Baltic windows-1258 - Viet Nam cp874 - Thai - this file is also for DOS DOS cp437 - Latin US cp737 - Greek cp775 - BaltRim cp850 - Latin1 cp852 - Latin2 cp855 - Cyrylic cp857 - Turkish cp860 - Portuguese cp861 - Iceland cp862 - Hebrew cp863 - Canada cp864 - Arabic cp865 - Nordic cp866 - Cyrylic Russian (this is the one, used in IE . . .
. . .
IE Cyrillic (DOS) ) cp869 - Greek2 MAC (Apple) x-mac-cyrillic x-mac-greek x-mac-icelandic x-mac-ce x-mac-roman ISO iso-8859-1 iso-8859-2 iso-8859-3 iso-8859-4 iso-8859-5 iso-8859-6 iso-8859-7 iso-8859-8 iso-8859-9 iso-8859-10 iso-8859-11 iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 . . .
. . .
iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 cp500 cp856 cp875 cp1006 cp1026 koi8-r (Cyrillic) koi8-u (Cyrillic Ukrainian) nextstep us-ascii us-ascii-quotes DSP implementation for NeXT stdenc symbol zdingbat And specially for old Polish programs: mazovia This list is to be read . . .
. . .
Access denied; you need the RELOAD privilege. . . Top 10. Enable real-time output of logging data Up to version 1.5 of Sphider-plus, during index / re-index there was no printout available because: - Several servers, especially on Win32, buffer the output from the script until it terminates before transmitting the results to the browser. - . . .
. . .
is seen. - Some versions of Microsoft Internet Explorer only start to display the page after they have received 256 bytes of output. As progress was not presented during index / re- index procedure, waiting for results became a pain in the neck. Selectable in Admin setting together with the update interval (1 - 10 seconds), AJAX technology was . . .
. . .
characters in front of words Warning: This option should be used with special care and not be activated for non ISO-8859 charsets. Some special characters as part of the word ending might be erased by accidental. Top 13. Media search for images, audio streams and videos 13.1 Media indexing Index of media files is enabled by separated Admin . . .
. . .
and 'Found at' - Total clicks - Last clicked - Query input 'Indexed Image Thumbnails' presenting: - Thumbnail 150 x 100 pixel - Image details like title, filename size of original image, link- and thumb-id - Option to delete the thumbnail In order to open the media files all tables contain active links. Media results are also stored in . . .
. . .
Settings' menu of the admin backend: First option: For results of XML product feeds, present the text extract of 350 characters as part of product attributes. If this option is not activated, all products of the XML product feed will be presented in search results, and the hits of the query string will be highlighted in all involved products. . . .
. . .
content of the database tables (those with the same table prefix) will be destroyed by the restore procedure. 16.5 Copy & Move This section of the Database Management will present only those databases, which are correctly configured, assigned and do have a set of installed tables as described in chapter Definition and configuration. This . . .
. . .
results. Never the less to find these x results, the complete database had to been browsed. Starting with version 2.5, an additional clean option is offered as part of the Admin backend. Main advantage of this option is a significant reduction of the search time for any query, because the content of the db could be limited to offer only x . . .
. . .
<link>http://www.abc.de/images/warp.gif</link> <title>warp.gif</title> <x_size>635</x_size> <y_size>98</y_size> </media_result> <media_result> <num>2</num> <type>audio</type> <url>http://www.abc.de/index.php</url> . . .
. . .
the following content: {"query":"colour","ip":"::1","host_name":"guard_007-hoster","query_time":"2016-03-18 10:56:23 AM","consumed":0.016,"total_results":2,"num_of_results":2,"from":1,"to":2,"text_results":[{"num":1,"weight":"100.0","url":"http:\/\/www.english.le-piaggie.info\/Info_eng.pdf","title":" Info_eng","fulltxt":" . . . . . .
. . .
placed on the ground floor, today's big kitchen with the heating fireplace for open and close mode of . . . ","page_size":"5,661.6kb","domain_name":"www.english.le-piaggie.info"},{"num":2,"weight":"50.0","url":"http:\/\/www.english.le-piaggie.info\/html\/description.html","title":" Description","fulltxt":" . . . cable. Additional active . . .
. . .
D.O.C.G. as far as to the Pratomagno mountains 160 olive-trees Detached house, about 800 m to reach the . . . ","page_size":"26.4 kb","domain_name":"www.english.le-piaggie.info"}]} Top . . .


5a2 All required information. Introduction Release and Legal Info Installation Documentation Change Log [ Documentation Summary ] Preamble: The info presented here is valid only for the latest release of Sphider-plus. At present version 4.2024a published February 12, 2024 is the actual release. 1. Settings, customizing and statistics 2. Indexing . . .
. . .
2. Indexing 2.1 Various options 2.2 Allow other hosts in same domain 2.3 Word stemming 2.4 Periodical Re-indexing 2.5 Preferred indexing 2.6 Multithreaded indexing 2.7 Create thumbnails during index procedure 2.8 Prevent indexing of known malware and pishing pages 2.9 Follow and create sitemap files 2.10 Use private sitemap instead of global . . .
. . .
Sitemap file 3. Using the indexer from command line 3.1 All options 3.2 Multithreaded indexing 4. Keeping pages, words and files from being indexed 4.1 robots.txt 4.2 Must include / must not include string list 4.3 Ignoring links 4.4 Ignoring parts of a page by <! sphider_noindex > 4.5 Ignoring parts of a page by <div id='abc'> . . .
. . .
charset' 6. Search modes 6.1 Search with wildcards * 6.2 Strict search ! 6.3 Tolerant search 6.4 Link search 6.5 Media search 6.6 Search only in one domain 6.7 Search in categories 6.8 Greek language support 6.9 Block queries 7. Chronological order for result listing 7.1 Text result listing 7.2 Media result listing 8. PDF converter 9. Clean . . .
. . .
of logging data 11. Error messages and Debug mode 12. Delete secondary characters 13. Media search for images, audio streams and videos 13.1 Media indexing 13.2 Not supported media content 13.3 Search for media content 13.4 Statistics for media content 14. Feed support 14.1 XML product feeds 14.2 RDF, RSD, RSS and Atom feeds 15. Result . . .
. . .
Overview 16.2 Definition and configuration 16.3 Activate / disable database 16.4 Backup & Restore of databases 16.5 Copy & and Move 16.6 Enhancing functionality of multiple database support 17. Search in categories 17.1 Hierachical structure 17.2 Parallel structure 18. User suggested sites 19. Vulnerability protection 19.1 Prevent queries . . .
. . .
templates 22.2 Embed the search engine into existing HTML code 22.3 The different style sheet files 23. JSON, XML and RSS result output [ Documentation ] 1. Settings, customizing and statistics If you want to change settings, behavior and design of Sphider-plus, you can do so by means of the Admin interface. There is a wide range of . . .
. . .
interface. There is a wide range of settings foreseen for Sphider-plus. Separated into different submenus like: Sites: - Add Site - Index only the new - Re-index all - Re-index only preferred URLs - Erase Re-index (available also for individual URLs) - Import/export URL list - Approve sites - Banned domains Categories: - Add, edit, delete - . . .
. . .
URL list - Approve sites - Banned domains Categories: - Add, edit, delete - Create new subcategory under Index: - Basic indexing options - Advanced options Clean: - Clean keywords not associated with any link - Clean links not associated with any site - Clean Category table not associated with any site - Clean Media links - Clear Temp table . . .
. . .
- Clear Search log - Clear 'Most Popular Page Links' log - Clear 'Most Popular Media Links' log - Clear Spider log, separate and bulk delete - Clear Thumbnail images, separate and bulk delete - Clear Text cache - Clear Media cache - Clear IDS log file - Clear flood attempts log file - Clear all entries in addurl or banned table - Truncate all . . .
. . .
flood attempts log file - Clear all entries in addurl or banned table - Truncate all tables in database Settings: - General Settings - Index Log Settings - Spider Settings - Search Settings - Order of Result listing - Suggest Options - Page Indexing Weights Database: - Configure up to 5 databases with unlimited number of table sets - Activate . . .
. . .
Database: - Configure up to 5 databases with unlimited number of table sets - Activate separately for 'Admin', 'Search' user and 'Suggest URL user' - Backup / Restore - Copy / Move - Optimize Templates: In order to enable customer's integration of Sphider-plus into existing sites, HTML templates are prepared for Search form Text result . . .
. . .
Search form Text result listing Media result listing Most popular queries etc. Three different designs are offered, which may be selected in submenu 'Settings'. If the layout does not fit the design of your site (which is normal), you may create new designs and modify the appropriate file /templates/My_template/adminstyle.css . . .
. . .
the appropriate file /templates/My_template/adminstyle.css /templates/My_template/userstyle.css Statistics output: - Top keywords (Top 50 with hit counter). - All indexed thumbnails w 3e80 ith ID3 and EXIF info. - Larges pages offering link URL and file size. - Most Popular Searches for text links offering: Link addr., total clicks, last . . .
. . .
Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Searches for media links offering: Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Links (click counter). - Search log offering: Query, Results, Queried at, Time taken, User IP, Country, Host name (Latest100) - Index log offering: . . .
. . .
Country, Host name (Latest100) - Index log offering: File-name, index date and delete option - sitemap log offering: sitemap.xml output sitemap list offering file/page suffixes - IDS log offering: IP, host, query, impact, involved tags, date and time of intrusion. - Flood attempts log offering: IP, query, date and time of flood attempt. - Auto . . .
. . .
attempts log offering: IP, query, date and time of flood attempt. - Auto Re-index log file - Server info offering: Server software, environment, MySQL, PDF-converter, image functions, php.ini file PHP integration, PHP security info. Each item holding lists of details. All text links, media links and thumbnails are active linked. As stated in . . .
. . .
individual Re-indexing, the periodical Re-indexer could be started and aborted in the "Options" menu of each site. 2.5 Preferred Re-indexing Each new URL added to the Admin backend, could be supplied with a priority level. This level will be used by the option 'Re-index only preferred sites'. Level 1 will be interpreted as most important, while . . .
. . .
less all index results will be stored in log files in sub folder /admin/log/ The names of the log files look like: db2_100524-2147561html (log file of first thread) db2_100524-21.48.12_2.html (log file of second thread) and is build by the following items: db2 - Number of database. 100524 - Date (May 24, 2010) 21.47.56 - Time when this thread . . .
. . .
spider.php -new1 php spider.php -new2 etc. The IDs will be added to the name of the corresponding log files like: db2_100524-21.47.56_ID1.html (log file of first thread) db2_100524-21.48.12_ID2.html (log file of second thread) IDs could be defined by personal requirements, but the limitations for file names with respect to the OS should be taken . . .
. . .
but will not erase the content of all the other tables. So the check whether the content of a page has changed (MD5sum) is still available for a fast re-index procedure. Once prepared, multithreaded re-indexing could be invoked by starting several threads and adding individual IDs to the option parameter like: php spider.php -erased1 php . . .
. . .
4.4 Ignoring parts of a page Sphider-plus includes an option to exclude parts of pages from being indexed. Thi 775e s can for example be used to prevent search result flooding when certain keywords appear on certain part in most pages (like a header, footer or a menu). Any part of a page between <! sphider_noindex > and <! . . .
. . .
<! sphider_noindex > and <! /sphider_noindex > tags is not indexed, however links in it are followed. 4.5 Ignoring parts of a page by <div id='abc'> Ignoring parts of a page by the <! sphider_noindex > tags requires direct access to the page, because the tags need to be added (edited) to the page. A more flexible method, . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ menu[0-5] / 4.6 Indexing only parts of a page by <div id='abc'> If enabled in Admin settings, the values as defined in the list-file /include/common/divs_use.txt will be used to index only the content between <div . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ table[0-5] / 4.7 Ignore HTML elements defined by <tagname> . . </tagname> This option is foreseen to cooperate with the new HTML5 elements like section, nav, aside, hgroup, article, header, footer, etc. HTML elements . . .
. . .
contain a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */nav[0-5]/ Please keep in mind that element names placed in /include/common/elements_not.txt will be processed case-sensitive. 4.8 Index only HTML elements defined by <tagname> . . </tagname> This is the vice versa . . .
. . .
of all the duplicate content URLs: http://www.example.com/product.php?item=swedish-fish&category=gummy-candy http://www.example.com/product.php?item=swedish-fish&trackingid=1234&sessionid=5678 and Sphider-plus will understand that the duplicates all refer to the canonical URL: http://www.example.com/product.php?item=swedish-fish. The . . .
. . .
charset are supported by the ConvertCharset function and will be used to convert text into UTF-8 Unicode: WINDOWS windows-1250 - Central Europe windows-1251 - Cyrillic windows-1252 - Latin I windows-1253 - Greek windows-1254 - Turkish windows-1255 - Hebrew windows-1256 - Arabic windows-1257 - Baltic windows-1258 - Viet Nam cp874 - Thai - this . . .
. . .
- Baltic windows-1258 - Viet Nam cp874 - Thai - this file is also for DOS DOS cp437 - Latin US cp737 - Greek cp775 - BaltRim cp850 - Latin1 cp852 - Latin2 cp855 - Cyrylic cp857 - Turkish cp860 - Portuguese cp861 - Iceland cp862 - Hebrew cp863 - Canada cp864 - Arabic cp865 - Nordic cp866 - Cyrylic Russian (this is the one, used in IE . . .
. . .
IE Cyrillic (DOS) ) cp869 - Greek2 MAC (Apple) x-mac-cyrillic x-mac-greek x-mac-icelandic x-mac-ce x-mac-roman ISO iso-8859-1 iso-8859-2 iso-8859-3 iso-8859-4 iso-8859-5 iso-8859-6 iso-8859-7 iso-8859-8 iso-8859-9 iso-8859-10 iso-8859-11 iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 . . .
. . .
iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 cp500 cp856 cp875 cp1006 cp1026 koi8-r (Cyrillic) koi8-u (Cyrillic Ukrainian) nextstep us-ascii us-ascii-quotes DSP implementation for NeXT stdenc symbol zdingbat And specially for old Polish programs: mazovia This list is to be read . . .
. . .
Access denied; you need the RELOAD privilege. . . Top 10. Enable real-time output of logging data Up to version 1.5 of Sphider-plus, during index / re-index there was no printout available because: - Several servers, especially on Win32, buffer the output from the script until it terminates before transmitting the results to the browser. - . . .
. . .
is seen. - Some versions of Microsoft Internet Explorer only start to display the page after they have received 256 bytes of output. As progress was not presented during index / re- index procedure, waiting for results became a pain in the neck. Selectable in Admin setting together with the update interval (1 - 10 seconds), AJAX technology was . . .
. . .
characters in front of words Warning: This option should be used with special care and not be activated for non ISO-8859 charsets. Some special characters as part of the word ending might be erased by accidental. Top 13. Media search for images, audio streams and videos 13.1 Media indexing Index of media files is enabled by separated Admin . . .
. . .
and 'Found at' - Total clicks - Last clicked - Query input 'Indexed Image Thumbnails' presenting: - Thumbnail 150 x 100 pixel - Image details like title, filename size of original image, link- and thumb-id - Option to delete the thumbnail In order to open the media files all tables contain active links. Media results are also stored in . . .
. . .
Settings' menu of the admin backend: First option: For results of XML product feeds, present the text extract of 350 characters as part of product attributes. If this option is not activated, all products of the XML product feed will be presented in search results, and the hits of the query string will be highlighted in all involved products. . . .
. . .
content of the database tables (those with the same table prefix) will be destroyed by the restore procedure. 16.5 Copy & Move This section of the Database Management will present only those databases, which are correctly configured, assigned and do have a set of installed tables as described in chapter Definition and configuration. This . . .
. . .
results. Never the less to find these x results, the complete database had to been browsed. Starting with version 2.5, an additional clean option is offered as part of the Admin backend. Main advantage of this option is a significant reduction of the search time for any query, because the content of the db could be limited to offer only x . . .
. . .
<link>http://www.abc.de/images/warp.gif</link> <title>warp.gif</title> <x_size>635</x_size> <y_size>98</y_size> </media_result> <media_result> <num>2</num> <type>audio</type> <url>http://www.abc.de/index.php</url> . . .
. . .
the following content: {"query":"colour","ip":"::1","host_name":"guard_007-hoster","query_time":"2016-03-18 10:56:23 AM","consumed":0.016,"total_results":2,"num_of_results":2,"from":1,"to":2,"text_results":[{"num":1,"weight":"100.0","url":"http:\/\/www.english.le-piaggie.info\/Info_eng.pdf","title":" Info_eng","fulltxt":" . . . . . .
. . .
placed on the ground floor, today's big kitchen with the heating fireplace for open and close mode of . . . ","page_size":"5,661.6kb","domain_name":"www.english.le-piaggie.info"},{"num":2,"weight":"50.0","url":"http:\/\/www.english.le-piaggie.info\/html\/description.html","title":" Description","fulltxt":" . . . cable. Additional active . . .
. . .
D.O.C.G. as far as to the Pratomagno mountains 160 olive-trees Detached house, about 800 m to reach the . . . ","page_size":"26.4 kb","domain_name":"www.english.le-piaggie.info"}]} Top . . .


5a2 All required information. Introduction Release and Legal Info Installation Documentation Change Log [ Documentation Summary ] Preamble: The info presented here is valid only for the latest release of Sphider-plus. At present version 4.2024a published February 12, 2024 is the actual release. 1. Settings, customizing and statistics 2. Indexing . . .
. . .
2. Indexing 2.1 Various options 2.2 Allow other hosts in same domain 2.3 Word stemming 2.4 Periodical Re-indexing 2.5 Preferred indexing 2.6 Multithreaded indexing 2.7 Create thumbnails during index procedure 2.8 Prevent indexing of known malware and pishing pages 2.9 Follow and create sitemap files 2.10 Use private sitemap instead of global . . .
. . .
Sitemap file 3. Using the indexer from command line 3.1 All options 3.2 Multithreaded indexing 4. Keeping pages, words and files from being indexed 4.1 robots.txt 4.2 Must include / must not include string list 4.3 Ignoring links 4.4 Ignoring parts of a page by <! sphider_noindex > 4.5 Ignoring parts of a page by <div id='abc'> . . .
. . .
charset' 6. Search modes 6.1 Search with wildcards * 6.2 Strict search ! 6.3 Tolerant search 6.4 Link search 6.5 Media search 6.6 Search only in one domain 6.7 Search in categories 6.8 Greek language support 6.9 Block queries 7. Chronological order for result listing 7.1 Text result listing 7.2 Media result listing 8. PDF converter 9. Clean . . .
. . .
of logging data 11. Error messages and Debug mode 12. Delete secondary characters 13. Media search for images, audio streams and videos 13.1 Media indexing 13.2 Not supported media content 13.3 Search for media content 13.4 Statistics for media content 14. Feed support 14.1 XML product feeds 14.2 RDF, RSD, RSS and Atom feeds 15. Result . . .
. . .
Overview 16.2 Definition and configuration 16.3 Activate / disable database 16.4 Backup & Restore of databases 16.5 Copy & and Move 16.6 Enhancing functionality of multiple database support 17. Search in categories 17.1 Hierachical structure 17.2 Parallel structure 18. User suggested sites 19. Vulnerability protection 19.1 Prevent queries . . .
. . .
templates 22.2 Embed the search engine into existing HTML code 22.3 The different style sheet files 23. JSON, XML and RSS result output [ Documentation ] 1. Settings, customizing and statistics If you want to change settings, behavior and design of Sphider-plus, you can do so by means of the Admin interface. There is a wide range of . . .
. . .
interface. There is a wide range of settings foreseen for Sphider-plus. Separated into different submenus like: Sites: - Add Site - Index only the new - Re-index all - Re-index only preferred URLs - Erase Re-index (available also for individual URLs) - Import/export URL list - Approve sites - Banned domains Categories: - Add, edit, delete - . . .
. . .
URL list - Approve sites - Banned domains Categories: - Add, edit, delete - Create new subcategory under Index: - Basic indexing options - Advanced options Clean: - Clean keywords not associated with any link - Clean links not associated with any site - Clean Category table not associated with any site - Clean Media links - Clear Temp table . . .
. . .
- Clear Search log - Clear 'Most Popular Page Links' log - Clear 'Most Popular Media Links' log - Clear Spider log, separate and bulk delete - Clear Thumbnail images, separate and bulk delete - Clear Text cache - Clear Media cache - Clear IDS log file - Clear flood attempts log file - Clear all entries in addurl or banned table - Truncate all . . .
. . .
flood attempts log file - Clear all entries in addurl or banned table - Truncate all tables in database Settings: - General Settings - Index Log Settings - Spider Settings - Search Settings - Order of Result listing - Suggest Options - Page Indexing Weights Database: - Configure up to 5 databases with unlimited number of table sets - Activate . . .
. . .
Database: - Configure up to 5 databases with unlimited number of table sets - Activate separately for 'Admin', 'Search' user and 'Suggest URL user' - Backup / Restore - Copy / Move - Optimize Templates: In order to enable customer's integration of Sphider-plus into existing sites, HTML templates are prepared for Search form Text result . . .
. . .
Search form Text result listing Media result listing Most popular queries etc. Three different designs are offered, which may be selected in submenu 'Settings'. If the layout does not fit the design of your site (which is normal), you may create new designs and modify the appropriate file /templates/My_template/adminstyle.css . . .
. . .
the appropriate file /templates/My_template/adminstyle.css /templates/My_template/userstyle.css Statistics output: - Top keywords (Top 50 with hit counter). - All indexed thumbnails w 3e80 ith ID3 and EXIF info. - Larges pages offering link URL and file size. - Most Popular Searches for text links offering: Link addr., total clicks, last . . .
. . .
Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Searches for media links offering: Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Links (click counter). - Search log offering: Query, Results, Queried at, Time taken, User IP, Country, Host name (Latest100) - Index log offering: . . .
. . .
Country, Host name (Latest100) - Index log offering: File-name, index date and delete option - sitemap log offering: sitemap.xml output sitemap list offering file/page suffixes - IDS log offering: IP, host, query, impact, involved tags, date and time of intrusion. - Flood attempts log offering: IP, query, date and time of flood attempt. - Auto . . .
. . .
attempts log offering: IP, query, date and time of flood attempt. - Auto Re-index log file - Server info offering: Server software, environment, MySQL, PDF-converter, image functions, php.ini file PHP integration, PHP security info. Each item holding lists of details. All text links, media links and thumbnails are active linked. As stated in . . .
. . .
individual Re-indexing, the periodical Re-indexer could be started and aborted in the "Options" menu of each site. 2.5 Preferred Re-indexing Each new URL added to the Admin backend, could be supplied with a priority level. This level will be used by the option 'Re-index only preferred sites'. Level 1 will be interpreted as most important, while . . .
. . .
less all index results will be stored in log files in sub folder /admin/log/ The names of the log files look like: db2_100524-21.47.56_1.html (log file of first thread) db2_100524-21.48.12_2.html (log file of second thread) and is build by the following items: db2 - Number of database. 100524 - Date (May 24, 2010) 21.47.56 - Time when this thread . . .
. . .
spider.php -new1 php spider.php -new2 etc. The IDs will be added to the name of the corresponding log files like: db2_100524-214756ID1html (log file of first thread) db2_100524-21.48.12_ID2.html (log file of second thread) IDs could be defined by personal requirements, but the limitations for file names with respect to the OS should be taken . . .
. . .
but will not erase the content of all the other tables. So the check whether the content of a page has changed (MD5sum) is still available for a fast re-index procedure. Once prepared, multithreaded re-indexing could be invoked by starting several threads and adding individual IDs to the option parameter like: php spider.php -erased1 php . . .
. . .
4.4 Ignoring parts of a page Sphider-plus includes an option to exclude parts of pages from being indexed. Thi 775e s can for example be used to prevent search result flooding when certain keywords appear on certain part in most pages (like a header, footer or a menu). Any part of a page between <! sphider_noindex > and <! . . .
. . .
<! sphider_noindex > and <! /sphider_noindex > tags is not indexed, however links in it are followed. 4.5 Ignoring parts of a page by <div id='abc'> Ignoring parts of a page by the <! sphider_noindex > tags requires direct access to the page, because the tags need to be added (edited) to the page. A more flexible method, . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ menu[0-5] / 4.6 Indexing only parts of a page by <div id='abc'> If enabled in Admin settings, the values as defined in the list-file /include/common/divs_use.txt will be used to index only the content between <div . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ table[0-5] / 4.7 Ignore HTML elements defined by <tagname> . . </tagname> This option is foreseen to cooperate with the new HTML5 elements like section, nav, aside, hgroup, article, header, footer, etc. HTML elements . . .
. . .
contain a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */nav[0-5]/ Please keep in mind that element names placed in /include/common/elements_not.txt will be processed case-sensitive. 4.8 Index only HTML elements defined by <tagname> . . </tagname> This is the vice versa . . .
. . .
of all the duplicate content URLs: http://www.example.com/product.php?item=swedish-fish&category=gummy-candy http://www.example.com/product.php?item=swedish-fish&trackingid=1234&sessionid=5678 and Sphider-plus will understand that the duplicates all refer to the canonical URL: http://www.example.com/product.php?item=swedish-fish. The . . .
. . .
charset are supported by the ConvertCharset function and will be used to convert text into UTF-8 Unicode: WINDOWS windows-1250 - Central Europe windows-1251 - Cyrillic windows-1252 - Latin I windows-1253 - Greek windows-1254 - Turkish windows-1255 - Hebrew windows-1256 - Arabic windows-1257 - Baltic windows-1258 - Viet Nam cp874 - Thai - this . . .
. . .
- Baltic windows-1258 - Viet Nam cp874 - Thai - this file is also for DOS DOS cp437 - Latin US cp737 - Greek cp775 - BaltRim cp850 - Latin1 cp852 - Latin2 cp855 - Cyrylic cp857 - Turkish cp860 - Portuguese cp861 - Iceland cp862 - Hebrew cp863 - Canada cp864 - Arabic cp865 - Nordic cp866 - Cyrylic Russian (this is the one, used in IE . . .
. . .
IE Cyrillic (DOS) ) cp869 - Greek2 MAC (Apple) x-mac-cyrillic x-mac-greek x-mac-icelandic x-mac-ce x-mac-roman ISO iso-8859-1 iso-8859-2 iso-8859-3 iso-8859-4 iso-8859-5 iso-8859-6 iso-8859-7 iso-8859-8 iso-8859-9 iso-8859-10 iso-8859-11 iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 . . .
. . .
iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 cp500 cp856 cp875 cp1006 cp1026 koi8-r (Cyrillic) koi8-u (Cyrillic Ukrainian) nextstep us-ascii us-ascii-quotes DSP implementation for NeXT stdenc symbol zdingbat And specially for old Polish programs: mazovia This list is to be read . . .
. . .
Access denied; you need the RELOAD privilege. . . Top 10. Enable real-time output of logging data Up to version 1.5 of Sphider-plus, during index / re-index there was no printout available because: - Several servers, especially on Win32, buffer the output from the script until it terminates before transmitting the results to the browser. - . . .
. . .
is seen. - Some versions of Microsoft Internet Explorer only start to display the page after they have received 256 bytes of output. As progress was not presented during index / re- index procedure, waiting for results became a pain in the neck. Selectable in Admin setting together with the update interval (1 - 10 seconds), AJAX technology was . . .
. . .
characters in front of words Warning: This option should be used with special care and not be activated for non ISO-8859 charsets. Some special characters as part of the word ending might be erased by accidental. Top 13. Media search for images, audio streams and videos 13.1 Media indexing Index of media files is enabled by separated Admin . . .
. . .
and 'Found at' - Total clicks - Last clicked - Query input 'Indexed Image Thumbnails' presenting: - Thumbnail 150 x 100 pixel - Image details like title, filename size of original image, link- and thumb-id - Option to delete the thumbnail In order to open the media files all tables contain active links. Media results are also stored in . . .
. . .
Settings' menu of the admin backend: First option: For results of XML product feeds, present the text extract of 350 characters as part of product attributes. If this option is not activated, all products of the XML product feed will be presented in search results, and the hits of the query string will be highlighted in all involved products. . . .
. . .
content of the database tables (those with the same table prefix) will be destroyed by the restore procedure. 16.5 Copy & Move This section of the Database Management will present only those databases, which are correctly configured, assigned and do have a set of installed tables as described in chapter Definition and configuration. This . . .
. . .
results. Never the less to find these x results, the complete database had to been browsed. Starting with version 2.5, an additional clean option is offered as part of the Admin backend. Main advantage of this option is a significant reduction of the search time for any query, because the content of the db could be limited to offer only x . . .
. . .
<link>http://www.abc.de/images/warp.gif</link> <title>warp.gif</title> <x_size>635</x_size> <y_size>98</y_size> </media_result> <media_result> <num>2</num> <type>audio</type> <url>http://www.abc.de/index.php</url> . . .
. . .
the following content: {"query":"colour","ip":"::1","host_name":"guard_007-hoster","query_time":"2016-03-18 10:56:23 AM","consumed":0.016,"total_results":2,"num_of_results":2,"from":1,"to":2,"text_results":[{"num":1,"weight":"100.0","url":"http:\/\/www.english.le-piaggie.info\/Info_eng.pdf","title":" Info_eng","fulltxt":" . . . . . .
. . .
placed on the ground floor, today's big kitchen with the heating fireplace for open and close mode of . . . ","page_size":"5,661.6kb","domain_name":"www.english.le-piaggie.info"},{"num":2,"weight":"50.0","url":"http:\/\/www.english.le-piaggie.info\/html\/description.html","title":" Description","fulltxt":" . . . cable. Additional active . . .
. . .
D.O.C.G. as far as to the Pratomagno mountains 160 olive-trees Detached house, about 800 m to reach the . . . ","page_size":"26.4 kb","domain_name":"www.english.le-piaggie.info"}]} Top . . .


5a2 All required information. Introduction Release and Legal Info Installation Documentation Change Log [ Documentation Summary ] Preamble: The info presented here is valid only for the latest release of Sphider-plus. At present version 4.2024a published February 12, 2024 is the actual release. 1. Settings, customizing and statistics 2. Indexing . . .
. . .
2. Indexing 2.1 Various options 2.2 Allow other hosts in same domain 2.3 Word stemming 2.4 Periodical Re-indexing 2.5 Preferred indexing 2.6 Multithreaded indexing 2.7 Create thumbnails during index procedure 2.8 Prevent indexing of known malware and pishing pages 2.9 Follow and create sitemap files 2.10 Use private sitemap instead of global . . .
. . .
Sitemap file 3. Using the indexer from command line 3.1 All options 3.2 Multithreaded indexing 4. Keeping pages, words and files from being indexed 4.1 robots.txt 4.2 Must include / must not include string list 4.3 Ignoring links 4.4 Ignoring parts of a page by <! sphider_noindex > 4.5 Ignoring parts of a page by <div id='abc'> . . .
. . .
charset' 6. Search modes 6.1 Search with wildcards * 6.2 Strict search ! 6.3 Tolerant search 6.4 Link search 6.5 Media search 6.6 Search only in one domain 6.7 Search in categories 6.8 Greek language support 6.9 Block queries 7. Chronological order for result listing 7.1 Text result listing 7.2 Media result listing 8. PDF converter 9. Clean . . .
. . .
of logging data 11. Error messages and Debug mode 12. Delete secondary characters 13. Media search for images, audio streams and videos 13.1 Media indexing 13.2 Not supported media content 13.3 Search for media content 13.4 Statistics for media content 14. Feed support 14.1 XML product feeds 14.2 RDF, RSD, RSS and Atom feeds 15. Result . . .
. . .
Overview 16.2 Definition and configuration 16.3 Activate / disable database 16.4 Backup & Restore of databases 16.5 Copy & and Move 16.6 Enhancing functionality of multiple database support 17. Search in categories 17.1 Hierachical structure 17.2 Parallel structure 18. User suggested sites 19. Vulnerability protection 19.1 Prevent queries . . .
. . .
templates 22.2 Embed the search engine into existing HTML code 22.3 The different style sheet files 23. JSON, XML and RSS result output [ Documentation ] 1. Settings, customizing and statistics If you want to change settings, behavior and design of Sphider-plus, you can do so by means of the Admin interface. There is a wide range of . . .
. . .
interface. There is a wide range of settings foreseen for Sphider-plus. Separated into different submenus like: Sites: - Add Site - Index only the new - Re-index all - Re-index only preferred URLs - Erase Re-index (available also for individual URLs) - Import/export URL list - Approve sites - Banned domains Categories: - Add, edit, delete - . . .
. . .
URL list - Approve sites - Banned domains Categories: - Add, edit, delete - Create new subcategory under Index: - Basic indexing options - Advanced options Clean: - Clean keywords not associated with any link - Clean links not associated with any site - Clean Category table not associated with any site - Clean Media links - Clear Temp table . . .
. . .
- Clear Search log - Clear 'Most Popular Page Links' log - Clear 'Most Popular Media Links' log - Clear Spider log, separate and bulk delete - Clear Thumbnail images, separate and bulk delete - Clear Text cache - Clear Media cache - Clear IDS log file - Clear flood attempts log file - Clear all entries in addurl or banned table - Truncate all . . .
. . .
flood attempts log file - Clear all entries in addurl or banned table - Truncate all tables in database Settings: - General Settings - Index Log Settings - Spider Settings - Search Settings - Order of Result listing - Suggest Options - Page Indexing Weights Database: - Configure up to 5 databases with unlimited number of table sets - Activate . . .
. . .
Database: - Configure up to 5 databases with unlimited number of table sets - Activate separately for 'Admin', 'Search' user and 'Suggest URL user' - Backup / Restore - Copy / Move - Optimize Templates: In order to enable customer's integration of Sphider-plus into existing sites, HTML templates are prepared for Search form Text result . . .
. . .
Search form Text result listing Media result listing Most popular queries etc. Three different designs are offered, which may be selected in submenu 'Settings'. If the layout does not fit the design of your site (which is normal), you may create new designs and modify the appropriate file /templates/My_template/adminstyle.css . . .
. . .
the appropriate file /templates/My_template/adminstyle.css /templates/My_template/userstyle.css Statistics output: - Top keywords (Top 50 with hit counter). - All indexed thumbnails w 3e80 ith ID3 and EXIF info. - Larges pages offering link URL and file size. - Most Popular Searches for text links offering: Link addr., total clicks, last . . .
. . .
Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Searches for media links offering: Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Links (click counter). - Search log offering: Query, Results, Queried at, Time taken, User IP, Country, Host name (Latest100) - Index log offering: . . .
. . .
Country, Host name (Latest100) - Index log offering: File-name, index date and delete option - sitemap log offering: sitemap.xml output sitemap list offering file/page suffixes - IDS log offering: IP, host, query, impact, involved tags, date and time of intrusion. - Flood attempts log offering: IP, query, date and time of flood attempt. - Auto . . .
. . .
attempts log offering: IP, query, date and time of flood attempt. - Auto Re-index log file - Server info offering: Server software, environment, MySQL, PDF-converter, image functions, php.ini file PHP integration, PHP security info. Each item holding lists of details. All text links, media links and thumbnails are active linked. As stated in . . .
. . .
individual Re-indexing, the periodical Re-indexer could be started and aborted in the "Options" menu of each site. 2.5 Preferred Re-indexing Each new URL added to the Admin backend, could be supplied with a priority level. This level will be used by the option 'Re-index only preferred sites'. Level 1 will be interpreted as most important, while . . .
. . .
less all index results will be stored in log files in sub folder /admin/log/ The names of the log files look like: db2_100524-21.47.56_1.html (log file of first thread) db2_100524-21.48.12_2.html (log file of second thread) and is build by the following items: db2 - Number of database. 100524 - Date (May 24, 2010) 21.47.56 - Time when this thread . . .
. . .
spider.php -new1 php spider.php -new2 etc. The IDs will be added to the name of the corresponding log files like: db2_100524-21.47.56_ID1.html (log file of first thread) db2_100524-21.48.12_ID2.html (log file of second thread) IDs could be defined by personal requirements, but the limitations for file names with respect to the OS should be taken . . .
. . .
but will not erase the content of all the other tables. So the check whether the content of a page has changed (MD5sum) is still available for a fast re-index procedure. Once prepared, multithreaded re-indexing could be invoked by starting several threads and adding individual IDs to the option parameter like: php spider.php -erased1 php . . .
. . .
4.4 Ignoring parts of a page Sphider-plus includes an option to exclude parts of pages from being indexed. Thi 775e s can for example be used to prevent search result flooding when certain keywords appear on certain part in most pages (like a header, footer or a menu). Any part of a page between <! sphider_noindex > and <! . . .
. . .
<! sphider_noindex > and <! /sphider_noindex > tags is not indexed, however links in it are followed. 4.5 Ignoring parts of a page by <div id='abc'> Ignoring parts of a page by the <! sphider_noindex > tags requires direct access to the page, because the tags need to be added (edited) to the page. A more flexible method, . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ menu[0-5] / 4.6 Indexing only parts of a page by <div id='abc'> If enabled in Admin settings, the values as defined in the list-file /include/common/divs_use.txt will be used to index only the content between <div . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ table[0-5] / 4.7 Ignore HTML elements defined by <tagname> . . </tagname> This option is foreseen to cooperate with the new HTML5 elements like section, nav, aside, hgroup, article, header, footer, etc. HTML elements . . .
. . .
contain a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */nav[0-5]/ Please keep in mind that element names placed in /include/common/elements_not.txt will be processed case-sensitive. 4.8 Index only HTML elements defined by <tagname> . . </tagname> This is the vice versa . . .
. . .
of all the duplicate content URLs: http://www.example.com/product.php?item=swedish-fish&category=gummy-candy http://www.example.com/product.php?item=swedish-fish&trackingid=1234&sessionid=5678 and Sphider-plus will understand that the duplicates all refer to the canonical URL: http://www.example.com/product.php?item=swedish-fish. The . . .
. . .
charset are supported by the ConvertCharset function and will be used to convert text into UTF-8 Unicode: WINDOWS windows-1250 - Central Europe windows-1251 - Cyrillic windows-1252 - Latin I windows-1253 - Greek windows-1254 - Turkish windows-1255 - Hebrew windows-1256 - Arabic windows-1257 - Baltic windows-1258 - Viet Nam cp874 - Thai - this . . .
. . .
- Baltic windows-1258 - Viet Nam cp874 - Thai - this file is also for DOS DOS cp437 - Latin US cp737 - Greek cp775 - BaltRim cp850 - Latin1 cp852 - Latin2 cp855 - Cyrylic cp857 - Turkish cp860 - Portuguese cp861 - Iceland cp862 - Hebrew cp863 - Canada cp864 - Arabic cp865 - Nordic cp866 - Cyrylic Russian (this is the one, used in IE . . .
. . .
IE Cyrillic (DOS) ) cp869 - Greek2 MAC (Apple) x-mac-cyrillic x-mac-greek x-mac-icelandic x-mac-ce x-mac-roman ISO iso-8859-1 iso-8859-2 iso-8859-3 iso-8859-4 iso-8859-5 iso-8859-6 iso-8859-7 iso-8859-8 iso-8859-9 iso-8859-10 iso-8859-11 iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 . . .
. . .
iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 cp500 cp856 cp875 cp1006 cp1026 koi8-r (Cyrillic) koi8-u (Cyrillic Ukrainian) nextstep us-ascii us-ascii-quotes DSP implementation for NeXT stdenc symbol zdingbat And specially for old Polish programs: mazovia This list is to be read . . .
. . .
Access denied; you need the RELOAD privilege. . . Top 10. Enable real-time output of logging data Up to version 1.5 of Sphider-plus, during index / re-index there was no printout available because: - Several servers, especially on Win32, buffer the output from the script until it terminates before transmitting the results to the browser. - . . .
. . .
is seen. - Some versions of Microsoft Internet Explorer only start to display the page after they have received 256 bytes of output. As progress was not presented during index / re- index procedure, waiting for results became a pain in the neck. Selectable in Admin setting together with the update interval (1 - 10 seconds), AJAX technology was . . .
. . .
characters in front of words Warning: This option should be used with special care and not be activated for non ISO-8859 charsets. Some special characters as part of the word ending might be erased by accidental. Top 13. Media search for images, audio streams and videos 13.1 Media indexing Index of media files is enabled by separated Admin . . .
. . .
and 'Found at' - Total clicks - Last clicked - Query input 'Indexed Image Thumbnails' presenting: - Thumbnail 150 x 100 pixel - Image details like title, filename size of original image, link- and thumb-id - Option to delete the thumbnail In order to open the media files all tables contain active links. Media results are also stored in . . .
. . .
Settings' menu of the admin backend: First option: For results of XML product feeds, present the text extract of 350 characters as part of product attributes. If this option is not activated, all products of the XML product feed will be presented in search results, and the hits of the query string will be highlighted in all involved products. . . .
. . .
content of the database tables (those with the same table prefix) will be destroyed by the restore procedure. 16.5 Copy & Move This section of the Database Management will present only those databases, which are correctly configured, assigned and do have a set of installed tables as described in chapter Definition and configuration. This . . .
. . .
results. Never the less to find these x results, the complete database had to been browsed. Starting with version 2.5, an additional clean option is offered as part of the Admin backend. Main advantage of this option is a significant reduction of the search time for any query, because the content of the db could be limited to offer only x . . .
. . .
<link>http://www.abc.de/images/warp.gif</link> <title>warp.gif</title> <x_size>635</x_size> <y_size>98</y_size> </media_result> <media_result> <num>2</num> <type>audio</type> <url>http://www.abc.de/index.php</url> . . .
. . .
the following content: {"query":"colour","ip":"::1","host_name":"guard_007-hoster","query_time":"2016-03-18 10:56:23 AM","consumed":0.016,"total_results":2,"num_of_results":2,"from":1,"to":2,"text_results":[{"num":1,"weight":"100.0","url":"http:\/\/www.english.le-piaggie.info\/Info_eng.pdf","title":" Info_eng","fulltxt":" . . . . . .
. . .
placed on the ground floor, today's big kitchen with the heating fireplace for open and close mode of . . . ","page_size":"56616kbdomainnamewwwenglishle-piaggieinfo}{num2weight500urlhttp\/\/wwwenglishle-piaggieinfo\/html\/descriptionhtmltitle","domain_name":"www.english.le-piaggie.info"},{"num":2,"weight":"50.0","url":"http:\/\/www.english.le-piaggie.info\/html\/description.html","title":" Description","fulltxt":" . . . cable. Additional active . . .
. . .
D.O.C.G. as far as to the Pratomagno mountains 160 olive-trees Detached house, about 800 m to reach the . . . ","page_size":"26.4 kb","domain_name":"www.english.le-piaggie.info"}]} Top . . .


5a2 All required information. Introduction Release and Legal Info Installation Documentation Change Log [ Documentation Summary ] Preamble: The info presented here is valid only for the latest release of Sphider-plus. At present version 4.2024a published February 12, 2024 is the actual release. 1. Settings, customizing and statistics 2. Indexing . . .
. . .
2. Indexing 2.1 Various options 2.2 Allow other hosts in same domain 2.3 Word stemming 2.4 Periodical Re-indexing 2.5 Preferred indexing 2.6 Multithreaded indexing 2.7 Create thumbnails during index procedure 2.8 Prevent indexing of known malware and pishing pages 2.9 Follow and create sitemap files 2.10 Use private sitemap instead of global . . .
. . .
Sitemap file 3. Using the indexer from command line 3.1 All options 3.2 Multithreaded indexing 4. Keeping pages, words and files from being indexed 4.1 robots.txt 4.2 Must include / must not include string list 4.3 Ignoring links 4.4 Ignoring parts of a page by <! sphider_noindex > 4.5 Ignoring parts of a page by <div id='abc'> . . .
. . .
charset' 6. Search modes 6.1 Search with wildcards * 6.2 Strict search ! 6.3 Tolerant search 6.4 Link search 6.5 Media search 6.6 Search only in one domain 6.7 Search in categories 6.8 Greek language support 6.9 Block queries 7. Chronological order for result listing 7.1 Text result listing 7.2 Media result listing 8. PDF converter 9. Clean . . .
. . .
of logging data 11. Error messages and Debug mode 12. Delete secondary characters 13. Media search for images, audio streams and videos 13.1 Media indexing 13.2 Not supported media content 13.3 Search for media content 13.4 Statistics for media content 14. Feed support 14.1 XML product feeds 14.2 RDF, RSD, RSS and Atom feeds 15. Result . . .
. . .
Overview 16.2 Definition and configuration 16.3 Activate / disable database 16.4 Backup & Restore of databases 16.5 Copy & and Move 16.6 Enhancing functionality of multiple database support 17. Search in categories 17.1 Hierachical structure 17.2 Parallel structure 18. User suggested sites 19. Vulnerability protection 19.1 Prevent queries . . .
. . .
templates 22.2 Embed the search engine into existing HTML code 22.3 The different style sheet files 23. JSON, XML and RSS result output [ Documentation ] 1. Settings, customizing and statistics If you want to change settings, behavior and design of Sphider-plus, you can do so by means of the Admin interface. There is a wide range of . . .
. . .
interface. There is a wide range of settings foreseen for Sphider-plus. Separated into different submenus like: Sites: - Add Site - Index only the new - Re-index all - Re-index only preferred URLs - Erase Re-index (available also for individual URLs) - Import/export URL list - Approve sites - Banned domains Categories: - Add, edit, delete - . . .
. . .
URL list - Approve sites - Banned domains Categories: - Add, edit, delete - Create new subcategory under Index: - Basic indexing options - Advanced options Clean: - Clean keywords not associated with any link - Clean links not associated with any site - Clean Category table not associated with any site - Clean Media links - Clear Temp table . . .
. . .
- Clear Search log - Clear 'Most Popular Page Links' log - Clear 'Most Popular Media Links' log - Clear Spider log, separate and bulk delete - Clear Thumbnail images, separate and bulk delete - Clear Text cache - Clear Media cache - Clear IDS log file - Clear flood attempts log file - Clear all entries in addurl or banned table - Truncate all . . .
. . .
flood attempts log file - Clear all entries in addurl or banned table - Truncate all tables in database Settings: - General Settings - Index Log Settings - Spider Settings - Search Settings - Order of Result listing - Suggest Options - Page Indexing Weights Database: - Configure up to 5 databases with unlimited number of table sets - Activate . . .
. . .
Database: - Configure up to 5 databases with unlimited number of table sets - Activate separately for 'Admin', 'Search' user and 'Suggest URL user' - Backup / Restore - Copy / Move - Optimize Templates: In order to enable customer's integration of Sphider-plus into existing sites, HTML templates are prepared for Search form Text result . . .
. . .
Search form Text result listing Media result listing Most popular queries etc. Three different designs are offered, which may be selected in submenu 'Settings'. If the layout does not fit the design of your site (which is normal), you may create new designs and modify the appropriate file /templates/My_template/adminstyle.css . . .
. . .
the appropriate file /templates/My_template/adminstyle.css /templates/My_template/userstyle.css Statistics output: - Top keywords (Top 50 with hit counter). - All indexed thumbnails w 3e80 ith ID3 and EXIF info. - Larges pages offering link URL and file size. - Most Popular Searches for text links offering: Link addr., total clicks, last . . .
. . .
Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Searches for media links offering: Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Links (click counter). - Search log offering: Query, Results, Queried at, Time taken, User IP, Country, Host name (Latest100) - Index log offering: . . .
. . .
Country, Host name (Latest100) - Index log offering: File-name, index date and delete option - sitemap log offering: sitemap.xml output sitemap list offering file/page suffixes - IDS log offering: IP, host, query, impact, involved tags, date and time of intrusion. - Flood attempts log offering: IP, query, date and time of flood attempt. - Auto . . .
. . .
attempts log offering: IP, query, date and time of flood attempt. - Auto Re-index log file - Server info offering: Server software, environment, MySQL, PDF-converter, image functions, php.ini file PHP integration, PHP security info. Each item holding lists of details. All text links, media links and thumbnails are active linked. As stated in . . .
. . .
individual Re-indexing, the periodical Re-indexer could be started and aborted in the "Options" menu of each site. 2.5 Preferred Re-indexing Each new URL added to the Admin backend, could be supplied with a priority level. This level will be used by the option 'Re-index only preferred sites'. Level 1 will be interpreted as most important, while . . .
. . .
less all index results will be stored in log files in sub folder /admin/log/ The names of the log files look like: db2_100524-21.47.56_1.html (log file of first thread) db2_100524-21.48.12_2.html (log file of second thread) and is build by the following items: db2 - Number of database. 100524 - Date (May 24, 2010) 21.47.56 - Time when this thread . . .
. . .
spider.php -new1 php spider.php -new2 etc. The IDs will be added to the name of the corresponding log files like: db2_100524-21.47.56_ID1.html (log file of first thread) db2_100524-21.48.12_ID2.html (log file of second thread) IDs could be defined by personal requirements, but the limitations for file names with respect to the OS should be taken . . .
. . .
but will not erase the content of all the other tables. So the check whether the content of a page has changed (MD5sum) is still available for a fast re-index procedure. Once prepared, multithreaded re-indexing could be invoked by starting several threads and adding individual IDs to the option parameter like: php spider.php -erased1 php . . .
. . .
4.4 Ignoring parts of a page Sphider-plus includes an option to exclude parts of pages from being indexed. Thi 775e s can for example be used to prevent search result flooding when certain keywords appear on certain part in most pages (like a header, footer or a menu). Any part of a page between <! sphider_noindex > and <! . . .
. . .
<! sphider_noindex > and <! /sphider_noindex > tags is not indexed, however links in it are followed. 4.5 Ignoring parts of a page by <div id='abc'> Ignoring parts of a page by the <! sphider_noindex > tags requires direct access to the page, because the tags need to be added (edited) to the page. A more flexible method, . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ menu[0-5] / 4.6 Indexing only parts of a page by <div id='abc'> If enabled in Admin settings, the values as defined in the list-file /include/common/divs_use.txt will be used to index only the content between <div . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ table[0-5] / 4.7 Ignore HTML elements defined by <tagname> . . </tagname> This option is foreseen to cooperate with the new HTML5 elements like section, nav, aside, hgroup, article, header, footer, etc. HTML elements . . .
. . .
contain a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */nav[0-5]/ Please keep in mind that element names placed in /include/common/elements_not.txt will be processed case-sensitive. 4.8 Index only HTML elements defined by <tagname> . . </tagname> This is the vice versa . . .
. . .
of all the duplicate content URLs: http://www.example.com/product.php?item=swedish-fish&category=gummy-candy http://www.example.com/product.php?item=swedish-fish&trackingid=1234&sessionid=5678 and Sphider-plus will understand that the duplicates all refer to the canonical URL: http://www.example.com/product.php?item=swedish-fish. The . . .
. . .
charset are supported by the ConvertCharset function and will be used to convert text into UTF-8 Unicode: WINDOWS windows-1250 - Central Europe windows-1251 - Cyrillic windows-1252 - Latin I windows-1253 - Greek windows-1254 - Turkish windows-1255 - Hebrew windows-1256 - Arabic windows-1257 - Baltic windows-1258 - Viet Nam cp874 - Thai - this . . .
. . .
- Baltic windows-1258 - Viet Nam cp874 - Thai - this file is also for DOS DOS cp437 - Latin US cp737 - Greek cp775 - BaltRim cp850 - Latin1 cp852 - Latin2 cp855 - Cyrylic cp857 - Turkish cp860 - Portuguese cp861 - Iceland cp862 - Hebrew cp863 - Canada cp864 - Arabic cp865 - Nordic cp866 - Cyrylic Russian (this is the one, used in IE . . .
. . .
IE Cyrillic (DOS) ) cp869 - Greek2 MAC (Apple) x-mac-cyrillic x-mac-greek x-mac-icelandic x-mac-ce x-mac-roman ISO iso-8859-1 iso-8859-2 iso-8859-3 iso-8859-4 iso-8859-5 iso-8859-6 iso-8859-7 iso-8859-8 iso-8859-9 iso-8859-10 iso-8859-11 iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 . . .
. . .
iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 cp500 cp856 cp875 cp1006 cp1026 koi8-r (Cyrillic) koi8-u (Cyrillic Ukrainian) nextstep us-ascii us-ascii-quotes DSP implementation for NeXT stdenc symbol zdingbat And specially for old Polish programs: mazovia This list is to be read . . .
. . .
Access denied; you need the RELOAD privilege. . . Top 10. Enable real-time output of logging data Up to version 1.5 of Sphider-plus, during index / re-index there was no printout available because: - Several servers, especially on Win32, buffer the output from the script until it terminates before transmitting the results to the browser. - . . .
. . .
is seen. - Some versions of Microsoft Internet Explorer only start to display the page after they have received 256 bytes of output. As progress was not presented during index / re- index procedure, waiting for results became a pain in the neck. Selectable in Admin setting together with the update interval (1 - 10 seconds), AJAX technology was . . .
. . .
characters in front of words Warning: This option should be used with special care and not be activated for non ISO-8859 charsets. Some special characters as part of the word ending might be erased by accidental. Top 13. Media search for images, audio streams and videos 13.1 Media indexing Index of media files is enabled by separated Admin . . .
. . .
and 'Found at' - Total clicks - Last clicked - Query input 'Indexed Image Thumbnails' presenting: - Thumbnail 150 x 100 pixel - Image details like title, filename size of original image, link- and thumb-id - Option to delete the thumbnail In order to open the media files all tables contain active links. Media results are also stored in . . .
. . .
Settings' menu of the admin backend: First option: For results of XML product feeds, present the text extract of 350 characters as part of product attributes. If this option is not activated, all products of the XML product feed will be presented in search results, and the hits of the query string will be highlighted in all involved products. . . .
. . .
content of the database tables (those with the same table prefix) will be destroyed by the restore procedure. 16.5 Copy & Move This section of the Database Management will present only those databases, which are correctly configured, assigned and do have a set of installed tables as described in chapter Definition and configuration. This . . .
. . .
results. Never the less to find these x results, the complete database had to been browsed. Starting with version 2.5, an additional clean option is offered as part of the Admin backend. Main advantage of this option is a significant reduction of the search time for any query, because the content of the db could be limited to offer only x . . .
. . .
<link>http://www.abc.de/images/warp.gif</link> <title>warp.gif</title> <x_size>635</x_size> <y_size>98</y_size> </media_result> <media_result> <num>2</num> <type>audio</type> <url>http://www.abc.de/index.php</url> . . .
. . .
the following content: {"query":"colour","ip":"::1","host_name":"guard_007-hoster","query_time":"2016-03-18 10:56:23 AM","consumed":0.016,"total_results":2,"num_of_results":2,"from":1,"to":2,"text_results":[{"num":1,"weight":"100.0","url":"http:\/\/www.english.le-piaggie.info\/Info_eng.pdf","title":" Info_eng","fulltxt":" . . . . . .
. . .
placed on the ground floor, today's big kitchen with the heating fireplace for open and close mode of . . . ","page_size":"5,661.6kb","domain_name":"www.english.le-piaggie.info"},{"num":2,"weight":"50.0","url":"http:\/\/www.english.le-piaggie.info\/html\/description.html","title":" Description","fulltxt":" . . . cable. Additional active . . .
. . .
D.O.C.G. as far as to the Pratomagno mountains 160 olive-trees Detached house, about 800 m to reach the . . . ","page_size":"26.4 kb","domain_name":"www.english.le-piaggie.info"}]} Top . . .


5a2 All required information. Introduction Release and Legal Info Installation Documentation Change Log [ Documentation Summary ] Preamble: The info presented here is valid only for the latest release of Sphider-plus. At present version 4.2024a published February 12, 2024 is the actual release. 1. Settings, customizing and statistics 2. Indexing . . .
. . .
2. Indexing 2.1 Various options 2.2 Allow other hosts in same domain 2.3 Word stemming 2.4 Periodical Re-indexing 2.5 Preferred indexing 2.6 Multithreaded indexing 2.7 Create thumbnails during index procedure 2.8 Prevent indexing of known malware and pishing pages 2.9 Follow and create sitemap files 2.10 Use private sitemap instead of global . . .
. . .
Sitemap file 3. Using the indexer from command line 3.1 All options 3.2 Multithreaded indexing 4. Keeping pages, words and files from being indexed 4.1 robots.txt 4.2 Must include / must not include string list 4.3 Ignoring links 4.4 Ignoring parts of a page by <! sphider_noindex > 4.5 Ignoring parts of a page by <div id='abc'> . . .
. . .
charset' 6. Search modes 6.1 Search with wildcards * 6.2 Strict search ! 6.3 Tolerant search 6.4 Link search 6.5 Media search 6.6 Search only in one domain 6.7 Search in categories 6.8 Greek language support 6.9 Block queries 7. Chronological order for result listing 7.1 Text result listing 7.2 Media result listing 8. PDF converter 9. Clean . . .
. . .
of logging data 11. Error messages and Debug mode 12. Delete secondary characters 13. Media search for images, audio streams and videos 13.1 Media indexing 13.2 Not supported media content 13.3 Search for media content 13.4 Statistics for media content 14. Feed support 14.1 XML product feeds 14.2 RDF, RSD, RSS and Atom feeds 15. Result . . .
. . .
Overview 16.2 Definition and configuration 16.3 Activate / disable database 16.4 Backup & Restore of databases 16.5 Copy & and Move 16.6 Enhancing functionality of multiple database support 17. Search in categories 17.1 Hierachical structure 17.2 Parallel structure 18. User suggested sites 19. Vulnerability protection 19.1 Prevent queries . . .
. . .
templates 22.2 Embed the search engine into existing HTML code 22.3 The different style sheet files 23. JSON, XML and RSS result output [ Documentation ] 1. Settings, customizing and statistics If you want to change settings, behavior and design of Sphider-plus, you can do so by means of the Admin interface. There is a wide range of . . .
. . .
interface. There is a wide range of settings foreseen for Sphider-plus. Separated into different submenus like: Sites: - Add Site - Index only the new - Re-index all - Re-index only preferred URLs - Erase Re-index (available also for individual URLs) - Import/export URL list - Approve sites - Banned domains Categories: - Add, edit, delete - . . .
. . .
URL list - Approve sites - Banned domains Categories: - Add, edit, delete - Create new subcategory under Index: - Basic indexing options - Advanced options Clean: - Clean keywords not associated with any link - Clean links not associated with any site - Clean Category table not associated with any site - Clean Media links - Clear Temp table . . .
. . .
- Clear Search log - Clear 'Most Popular Page Links' log - Clear 'Most Popular Media Links' log - Clear Spider log, separate and bulk delete - Clear Thumbnail images, separate and bulk delete - Clear Text cache - Clear Media cache - Clear IDS log file - Clear flood attempts log file - Clear all entries in addurl or banned table - Truncate all . . .
. . .
flood attempts log file - Clear all entries in addurl or banned table - Truncate all tables in database Settings: - General Settings - Index Log Settings - Spider Settings - Search Settings - Order of Result listing - Suggest Options - Page Indexing Weights Database: - Configure up to 5 databases with unlimited number of table sets - Activate . . .
. . .
Database: - Configure up to 5 databases with unlimited number of table sets - Activate separately for 'Admin', 'Search' user and 'Suggest URL user' - Backup / Restore - Copy / Move - Optimize Templates: In order to enable customer's integration of Sphider-plus into existing sites, HTML templates are prepared for Search form Text result . . .
. . .
Search form Text result listing Media result listing Most popular queries etc. Three different designs are offered, which may be selected in submenu 'Settings'. If the layout does not fit the design of your site (which is normal), you may create new designs and modify the appropriate file /templates/My_template/adminstyle.css . . .
. . .
the appropriate file /templates/My_template/adminstyle.css /templates/My_template/userstyle.css Statistics output: - Top keywords (Top 50 with hit counter). - All indexed thumbnails w 3e80 ith ID3 and EXIF info. - Larges pages offering link URL and file size. - Most Popular Searches for text links offering: Link addr., total clicks, last . . .
. . .
Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Searches for media links offering: Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Links (click counter). - Search log offering: Query, Results, Queried at, Time taken, User IP, Country, Host name (Latest100) - Index log offering: . . .
. . .
Country, Host name (Latest100) - Index log offering: File-name, index date and delete option - sitemap log offering: sitemap.xml output sitemap list offering file/page suffixes - IDS log offering: IP, host, query, impact, involved tags, date and time of intrusion. - Flood attempts log offering: IP, query, date and time of flood attempt. - Auto . . .
. . .
attempts log offering: IP, query, date and time of flood attempt. - Auto Re-index log file - Server info offering: Server software, environment, MySQL, PDF-converter, image functions, php.ini file PHP integration, PHP security info. Each item holding lists of details. All text links, media links and thumbnails are active linked. As stated in . . .
. . .
individual Re-indexing, the periodical Re-indexer could be started and aborted in the "Options" menu of each site. 2.5 Preferred Re-indexing Each new URL added to the Admin backend, could be supplied with a priority level. This level will be used by the option 'Re-index only preferred sites'. Level 1 will be interpreted as most important, while . . .
. . .
less all index results will be stored in log files in sub folder /admin/log/ The names of the log files look like: db2_100524-21.47.56_1.html (log file of first thread) db2_100524-21.48.12_2.html (log file of second thread) and is build by the following items: db2 - Number of database. 100524 - Date (May 24, 2010) 21.47.56 - Time when this thread . . .
. . .
spider.php -new1 php spider.php -new2 etc. The IDs will be added to the name of the corresponding log files like: db2_100524-21.47.56_ID1.html (log file of first thread) db2_100524-21.48.12_ID2.html (log file of second thread) IDs could be defined by personal requirements, but the limitations for file names with respect to the OS should be taken . . .
. . .
but will not erase the content of all the other tables. So the check whether the content of a page has changed (MD5sum) is still available for a fast re-index procedure. Once prepared, multithreaded re-indexing could be invoked by starting several threads and adding individual IDs to the option parameter like: php spider.php -erased1 php . . .
. . .
4.4 Ignoring parts of a page Sphider-plus includes an option to exclude parts of pages from being indexed. Thi 775e s can for example be used to prevent search result flooding when certain keywords appear on certain part in most pages (like a header, footer or a menu). Any part of a page between <! sphider_noindex > and <! . . .
. . .
<! sphider_noindex > and <! /sphider_noindex > tags is not indexed, however links in it are followed. 4.5 Ignoring parts of a page by <div id='abc'> Ignoring parts of a page by the <! sphider_noindex > tags requires direct access to the page, because the tags need to be added (edited) to the page. A more flexible method, . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ menu[0-5] / 4.6 Indexing only parts of a page by <div id='abc'> If enabled in Admin settings, the values as defined in the list-file /include/common/divs_use.txt will be used to index only the content between <div . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ table[0-5] / 4.7 Ignore HTML elements defined by <tagname> . . </tagname> This option is foreseen to cooperate with the new HTML5 elements like section, nav, aside, hgroup, article, header, footer, etc. HTML elements . . .
. . .
contain a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */nav[0-5]/ Please keep in mind that element names placed in /include/common/elements_not.txt will be processed case-sensitive. 4.8 Index only HTML elements defined by <tagname> . . </tagname> This is the vice versa . . .
. . .
of all the duplicate content URLs: http://www.example.com/product.php?item=swedish-fish&category=gummy-candy http://www.example.com/product.php?item=swedish-fish&trackingid=1234&sessionid=5678 and Sphider-plus will understand that the duplicates all refer to the canonical URL: http://www.example.com/product.php?item=swedish-fish. The . . .
. . .
charset are supported by the ConvertCharset function and will be used to convert text into UTF-8 Unicode: WINDOWS windows-1250 - Central Europe windows-1251 - Cyrillic windows-1252 - Latin I windows-1253 - Greek windows-1254 - Turkish windows-1255 - Hebrew windows-1256 - Arabic windows-1257 - Baltic windows-1258 - Viet Nam cp874 - Thai - this . . .
. . .
- Baltic windows-1258 - Viet Nam cp874 - Thai - this file is also for DOS DOS cp437 - Latin US cp737 - Greek cp775 - BaltRim cp850 - Latin1 cp852 - Latin2 cp855 - Cyrylic cp857 - Turkish cp860 - Portuguese cp861 - Iceland cp862 - Hebrew cp863 - Canada cp864 - Arabic cp865 - Nordic cp866 - Cyrylic Russian (this is the one, used in IE . . .
. . .
IE Cyrillic (DOS) ) cp869 - Greek2 MAC (Apple) x-mac-cyrillic x-mac-greek x-mac-icelandic x-mac-ce x-mac-roman ISO iso-8859-1 iso-8859-2 iso-8859-3 iso-8859-4 iso-8859-5 iso-8859-6 iso-8859-7 iso-8859-8 iso-8859-9 iso-8859-10 iso-8859-11 iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 . . .
. . .
iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 cp500 cp856 cp875 cp1006 cp1026 koi8-r (Cyrillic) koi8-u (Cyrillic Ukrainian) nextstep us-ascii us-ascii-quotes DSP implementation for NeXT stdenc symbol zdingbat And specially for old Polish programs: mazovia This list is to be read . . .
. . .
Access denied; you need the RELOAD privilege. . . Top 10. Enable real-time output of logging data Up to version 1.5 of Sphider-plus, during index / re-index there was no printout available because: - Several servers, especially on Win32, buffer the output from the script until it terminates before transmitting the results to the browser. - . . .
. . .
is seen. - Some versions of Microsoft Internet Explorer only start to display the page after they have received 256 bytes of output. As progress was not presented during index / re- index procedure, waiting for results became a pain in the neck. Selectable in Admin setting together with the update interval (1 - 10 seconds), AJAX technology was . . .
. . .
characters in front of words Warning: This option should be used with special care and not be activated for non ISO-8859 charsets. Some special characters as part of the word ending might be erased by accidental. Top 13. Media search for images, audio streams and videos 13.1 Media indexing Index of media files is enabled by separated Admin . . .
. . .
and 'Found at' - Total clicks - Last clicked - Query input 'Indexed Image Thumbnails' presenting: - Thumbnail 150 x 100 pixel - Image details like title, filename size of original image, link- and thumb-id - Option to delete the thumbnail In order to open the media files all tables contain active links. Media results are also stored in . . .
. . .
Settings' menu of the admin backend: First option: For results of XML product feeds, present the text extract of 350 characters as part of product attributes. If this option is not activated, all products of the XML product feed will be presented in search results, and the hits of the query string will be highlighted in all involved products. . . .
. . .
content of the database tables (those with the same table prefix) will be destroyed by the restore procedure. 16.5 Copy & Move This section of the Database Management will present only those databases, which are correctly configured, assigned and do have a set of installed tables as described in chapter Definition and configuration. This . . .
. . .
results. Never the less to find these x results, the complete database had to been browsed. Starting with version 2.5, an additional clean option is offered as part of the Admin backend. Main advantage of this option is a significant reduction of the search time for any query, because the content of the db could be limited to offer only x . . .
. . .
<link>http://www.abc.de/images/warp.gif</link> <title>warp.gif</title> <x_size>635</x_size> <y_size>98</y_size> </media_result> <media_result> <num>2</num> <type>audio</type> <url>http://www.abc.de/index.php</url> . . .
. . .
the following content: {"query":"colour","ip":"::1","host_name":"guard_007-hoster","query_time":"2016-03-18 10:56:23 AM","consumed":0.016,"total_results":2,"num_of_results":2,"from":1,"to":2,"text_results":[{"num":1,"weight":"100.0","url":"http:\/\/www.english.le-piaggie.info\/Info_eng.pdf","title":" Info_eng","fulltxt":" . . . . . .
. . .
placed on the ground floor, today's big kitchen with the heating fireplace for open and close mode of . . . ","page_size":"5,661.6kb","domain_name":"www.english.le-piaggie.info"},{"num":2,"weight":"50.0","url":"http:\/\/www.english.le-piaggie.info\/html\/description.html","title":" Description","fulltxt":" . . . cable. Additional active . . .
. . .
D.O.C.G. as far as to the Pratomagno mountains 160 olive-trees Detached house, about 800 m to reach the . . . ","page_size":"26.4 kb","domain_name":"www.english.le-piaggie.info"}]} Top . . .


5a2 All required information. Introduction Release and Legal Info Installation Documentation Change Log [ Documentation Summary ] Preamble: The info presented here is valid only for the latest release of Sphider-plus. At present version 4.2024a published February 12, 2024 is the actual release. 1. Settings, customizing and statistics 2. Indexing . . .
. . .
2. Indexing 2.1 Various options 2.2 Allow other hosts in same domain 2.3 Word stemming 2.4 Periodical Re-indexing 2.5 Preferred indexing 2.6 Multithreaded indexing 2.7 Create thumbnails during index procedure 2.8 Prevent indexing of known malware and pishing pages 2.9 Follow and create sitemap files 2.10 Use private sitemap instead of global . . .
. . .
Sitemap file 3. Using the indexer from command line 3.1 All options 3.2 Multithreaded indexing 4. Keeping pages, words and files from being indexed 4.1 robots.txt 4.2 Must include / must not include string list 4.3 Ignoring links 4.4 Ignoring parts of a page by <! sphider_noindex > 4.5 Ignoring parts of a page by <div id='abc'> . . .
. . .
charset' 6. Search modes 6.1 Search with wildcards * 6.2 Strict search ! 6.3 Tolerant search 6.4 Link search 6.5 Media search 6.6 Search only in one domain 6.7 Search in categories 6.8 Greek language support 6.9 Block queries 7. Chronological order for result listing 7.1 Text result listing 7.2 Media result listing 8. PDF converter 9. Clean . . .
. . .
of logging data 11. Error messages and Debug mode 12. Delete secondary characters 13. Media search for images, audio streams and videos 13.1 Media indexing 13.2 Not supported media content 13.3 Search for media content 13.4 Statistics for media content 14. Feed support 14.1 XML product feeds 14.2 RDF, RSD, RSS and Atom feeds 15. Result . . .
. . .
Overview 16.2 Definition and configuration 16.3 Activate / disable database 16.4 Backup & Restore of databases 16.5 Copy & and Move 16.6 Enhancing functionality of multiple database support 17. Search in categories 17.1 Hierachical structure 17.2 Parallel structure 18. User suggested sites 19. Vulnerability protection 19.1 Prevent queries . . .
. . .
templates 22.2 Embed the search engine into existing HTML code 22.3 The different style sheet files 23. JSON, XML and RSS result output [ Documentation ] 1. Settings, customizing and statistics If you want to change settings, behavior and design of Sphider-plus, you can do so by means of the Admin interface. There is a wide range of . . .
. . .
interface. There is a wide range of settings foreseen for Sphider-plus. Separated into different submenus like: Sites: - Add Site - Index only the new - Re-index all - Re-index only preferred URLs - Erase Re-index (available also for individual URLs) - Import/export URL list - Approve sites - Banned domains Categories: - Add, edit, delete - . . .
. . .
URL list - Approve sites - Banned domains Categories: - Add, edit, delete - Create new subcategory under Index: - Basic indexing options - Advanced options Clean: - Clean keywords not associated with any link - Clean links not associated with any site - Clean Category table not associated with any site - Clean Media links - Clear Temp table . . .
. . .
- Clear Search log - Clear 'Most Popular Page Links' log - Clear 'Most Popular Media Links' log - Clear Spider log, separate and bulk delete - Clear Thumbnail images, separate and bulk delete - Clear Text cache - Clear Media cache - Clear IDS log file - Clear flood attempts log file - Clear all entries in addurl or banned table - Truncate all . . .
. . .
flood attempts log file - Clear all entries in addurl or banned table - Truncate all tables in database Settings: - General Settings - Index Log Settings - Spider Settings - Search Settings - Order of Result listing - Suggest Options - Page Indexing Weights Database: - Configure up to 5 databases with unlimited number of table sets - Activate . . .
. . .
Database: - Configure up to 5 databases with unlimited number of table sets - Activate separately for 'Admin', 'Search' user and 'Suggest URL user' - Backup / Restore - Copy / Move - Optimize Templates: In order to enable customer's integration of Sphider-plus into existing sites, HTML templates are prepared for Search form Text result . . .
. . .
Search form Text result listing Media result listing Most popular queries etc. Three different designs are offered, which may be selected in submenu 'Settings'. If the layout does not fit the design of your site (which is normal), you may create new designs and modify the appropriate file /templates/My_template/adminstyle.css . . .
. . .
the appropriate file /templates/My_template/adminstyle.css /templates/My_template/userstyle.css Statistics output: - Top keywords (Top 50 with hit counter). - All indexed thumbnails w 3e80 ith ID3 and EXIF info. - Larges pages offering link URL and file size. - Most Popular Searches for text links offering: Link addr., total clicks, last . . .
. . .
Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Searches for media links offering: Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Links (click counter). - Search log offering: Query, Results, Queried at, Time taken, User IP, Country, Host name (Latest100) - Index log offering: . . .
. . .
Country, Host name (Latest100) - Index log offering: File-name, index date and delete option - sitemap log offering: sitemap.xml output sitemap list offering file/page suffixes - IDS log offering: IP, host, query, impact, involved tags, date and time of intrusion. - Flood attempts log offering: IP, query, date and time of flood attempt. - Auto . . .
. . .
attempts log offering: IP, query, date and time of flood attempt. - Auto Re-index log file - Server info offering: Server software, environment, MySQL, PDF-converter, image functions, php.ini file PHP integration, PHP security info. Each item holding lists of details. All text links, media links and thumbnails are active linked. As stated in . . .
. . .
individual Re-indexing, the periodical Re-indexer could be started and aborted in the "Options" menu of each site. 2.5 Preferred Re-indexing Each new URL added to the Admin backend, could be supplied with a priority level. This level will be used by the option 'Re-index only preferred sites'. Level 1 will be interpreted as most important, while . . .
. . .
less all index results will be stored in log files in sub folder /admin/log/ The names of the log files look like: db2_100524-21.47.56_1.html (log file of first thread) db2_100524-21.48.12_2.html (log file of second thread) and is build by the following items: db2 - Number of database. 100524 - Date (May 24, 2010) 21.47.56 - Time when this thread . . .
. . .
spider.php -new1 php spider.php -new2 etc. The IDs will be added to the name of the corresponding log files like: db2_100524-21.47.56_ID1.html (log file of first thread) db2_100524-21.48.12_ID2.html (log file of second thread) IDs could be defined by personal requirements, but the limitations for file names with respect to the OS should be taken . . .
. . .
but will not erase the content of all the other tables. So the check whether the content of a page has changed (MD5sum) is still available for a fast re-index procedure. Once prepared, multithreaded re-indexing could be invoked by starting several threads and adding individual IDs to the option parameter like: php spider.php -erased1 php . . .
. . .
4.4 Ignoring parts of a page Sphider-plus includes an option to exclude parts of pages from being indexed. Thi 775e s can for example be used to prevent search result flooding when certain keywords appear on certain part in most pages (like a header, footer or a menu). Any part of a page between <! sphider_noindex > and <! . . .
. . .
<! sphider_noindex > and <! /sphider_noindex > tags is not indexed, however links in it are followed. 4.5 Ignoring parts of a page by <div id='abc'> Ignoring parts of a page by the <! sphider_noindex > tags requires direct access to the page, because the tags need to be added (edited) to the page. A more flexible method, . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ menu[0-5] / 4.6 Indexing only parts of a page by <div id='abc'> If enabled in Admin settings, the values as defined in the list-file /include/common/divs_use.txt will be used to index only the content between <div . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ table[0-5] / 4.7 Ignore HTML elements defined by <tagname> . . </tagname> This option is foreseen to cooperate with the new HTML5 elements like section, nav, aside, hgroup, article, header, footer, etc. HTML elements . . .
. . .
contain a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */nav[0-5]/ Please keep in mind that element names placed in /include/common/elements_not.txt will be processed case-sensitive. 4.8 Index only HTML elements defined by <tagname> . . </tagname> This is the vice versa . . .
. . .
of all the duplicate content URLs: http://www.example.com/product.php?item=swedish-fish&category=gummy-candy http://www.example.com/product.php?item=swedish-fish&trackingid=1234&sessionid=5678 and Sphider-plus will understand that the duplicates all refer to the canonical URL: http://www.example.com/product.php?item=swedish-fish. The . . .
. . .
charset are supported by the ConvertCharset function and will be used to convert text into UTF-8 Unicode: WINDOWS windows-1250 - Central Europe windows-1251 - Cyrillic windows-1252 - Latin I windows-1253 - Greek windows-1254 - Turkish windows-1255 - Hebrew windows-1256 - Arabic windows-1257 - Baltic windows-1258 - Viet Nam cp874 - Thai - this . . .
. . .
- Baltic windows-1258 - Viet Nam cp874 - Thai - this file is also for DOS DOS cp437 - Latin US cp737 - Greek cp775 - BaltRim cp850 - Latin1 cp852 - Latin2 cp855 - Cyrylic cp857 - Turkish cp860 - Portuguese cp861 - Iceland cp862 - Hebrew cp863 - Canada cp864 - Arabic cp865 - Nordic cp866 - Cyrylic Russian (this is the one, used in IE . . .
. . .
IE Cyrillic (DOS) ) cp869 - Greek2 MAC (Apple) x-mac-cyrillic x-mac-greek x-mac-icelandic x-mac-ce x-mac-roman ISO iso-8859-1-1 iso-8859-1-2 iso-8859-1-3 iso-8859-1-4 iso-8859-1-5 iso-8859-1-6 iso-8859-1-7 iso-8859-1-8 iso-8859-1-9 iso-8859-1-10 iso-8859-1-11 iso-8859-1-12 iso-8859-1-13 iso-8859-1-14 iso-8859-1-15 iso-8859-1-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 . . .
. . .
iso-8859-1-12 iso-8859-1-13 iso-8859-1-14 iso-8859-1-15 iso-8859-1-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 cp500 cp856 cp875 cp1006 cp1026 koi8-r (Cyrillic) koi8-u (Cyrillic Ukrainian) nextstep us-ascii us-ascii-quotes DSP implementation for NeXT stdenc symbol zdingbat And specially for old Polish programs: mazovia This list is to be read . . .
. . .
Access denied; you need the RELOAD privilege. . . Top 10. Enable real-time output of logging data Up to version 1.5 of Sphider-plus, during index / re-index there was no printout available because: - Several servers, especially on Win32, buffer the output from the script until it terminates before transmitting the results to the browser. - . . .
. . .
is seen. - Some versions of Microsoft Internet Explorer only start to display the page after they have received 256 bytes of output. As progress was not presented during index / re- index procedure, waiting for results became a pain in the neck. Selectable in Admin setting together with the update interval (1 - 10 seconds), AJAX technology was . . .
. . .
characters in front of words Warning: This option should be used with special care and not be activated for non ISO-8859-1 charsets. Some special characters as part of the word ending might be erased by accidental. Top 13. Media search for images, audio streams and videos 13.1 Media indexing Index of media files is enabled by separated Admin . . .
. . .
and 'Found at' - Total clicks - Last clicked - Query input 'Indexed Image Thumbnails' presenting: - Thumbnail 150 x 100 pixel - Image details like title, filename size of original image, link- and thumb-id - Option to delete the thumbnail In order to open the media files all tables contain active links. Media results are also stored in . . .
. . .
Settings' menu of the admin backend: First option: For results of XML product feeds, present the text extract of 350 characters as part of product attributes. If this option is not activated, all products of the XML product feed will be presented in search results, and the hits of the query string will be highlighted in all involved products. . . .
. . .
content of the database tables (those with the same table prefix) will be destroyed by the restore procedure. 16.5 Copy & Move This section of the Database Management will present only those databases, which are correctly configured, assigned and do have a set of installed tables as described in chapter Definition and configuration. This . . .
. . .
results. Never the less to find these x results, the complete database had to been browsed. Starting with version 2.5, an additional clean option is offered as part of the Admin backend. Main advantage of this option is a significant reduction of the search time for any query, because the content of the db could be limited to offer only x . . .
. . .
<link>http://www.abc.de/images/warp.gif</link> <title>warp.gif</title> <x_size>635</x_size> <y_size>98</y_size> </media_result> <media_result> <num>2</num> <type>audio</type> <url>http://www.abc.de/index.php</url> . . .
. . .
the following content: {"query":"colour","ip":"::1","host_name":"guard_007-hoster","query_time":"2016-03-18 10:56:23 AM","consumed":0.016,"total_results":2,"num_of_results":2,"from":1,"to":2,"text_results":[{"num":1,"weight":"100.0","url":"http:\/\/www.english.le-piaggie.info\/Info_eng.pdf","title":" Info_eng","fulltxt":" . . . . . .
. . .
placed on the ground floor, today's big kitchen with the heating fireplace for open and close mode of . . . ","page_size":"5,661.6kb","domain_name":"www.english.le-piaggie.info"},{"num":2,"weight":"50.0","url":"http:\/\/www.english.le-piaggie.info\/html\/description.html","title":" Description","fulltxt":" . . . cable. Additional active . . .
. . .
D.O.C.G. as far as to the Pratomagno mountains 160 olive-trees Detached house, about 800 m to reach the . . . ","page_size":"26.4 kb","domain_name":"www.english.le-piaggie.info"}]} Top . . .


5a2 All required information. Introduction Release and Legal Info Installation Documentation Change Log [ Documentation Summary ] Preamble The info presented here is valid only for the latest release of Sphider-plus. At present version 4.2024a published February 12, 2024 is the actual release. 1. Settings, customizing and statistics 2. Indexing . . .
. . .
2. Indexing 2.1 Various options 2.2 Allow other hosts in same domain 2.3 Word stemming 2.4 Periodical Re-indexing 2.5 Preferred indexing 2.6 Multithreaded indexing 2.7 Create thumbnails during index procedure 2.8 Prevent indexing of known malware and pishing pages 2.9 Follow and create sitemap files 2.10 Use private sitemap instead of global . . .
. . .
Sitemap file 3. Using the indexer from command line 3.1 All options 3.2 Multithreaded indexing 4. Keeping pages, words and files from being indexed 4.1 robots.txt 4.2 Must include / must not include string list 4.3 Ignoring links 4.4 Ignoring parts of a page by <! sphider_noindex > 4.5 Ignoring parts of a page by <div id='abc'> . . .
. . .
charset' 6. Search modes 6.1 Search with wildcards * 6.2 Strict search ! 6.3 Tolerant search 6.4 Link search 6.5 Media search 6.6 Search only in one domain 6.7 Search in categories 6.8 Greek language support 6.9 Block queries 7. Chronological order for result listing 7.1 Text result listing 7.2 Media result listing 8. PDF converter 9. Clean . . .
. . .
of logging data 11. Error messages and Debug mode 12. Delete secondary characters 13. Media search for images, audio streams and videos 13.1 Media indexing 13.2 Not supported media content 13.3 Search for media content 13.4 Statistics for media content 14. Feed support 14.1 XML product feeds 14.2 RDF, RSD, RSS and Atom feeds 15. Result . . .
. . .
Overview 16.2 Definition and configuration 16.3 Activate / disable database 16.4 Backup & Restore of databases 16.5 Copy & and Move 16.6 Enhancing functionality of multiple database support 17. Search in categories 17.1 Hierachical structure 17.2 Parallel structure 18. User suggested sites 19. Vulnerability protection 19.1 Prevent queries . . .
. . .
templates 22.2 Embed the search engine into existing HTML code 22.3 The different style sheet files 23. JSON, XML and RSS result output [ Documentation ] 1. Settings, customizing and statistics If you want to change settings, behavior and design of Sphider-plus, you can do so by means of the Admin interface. There is a wide range of . . .
. . .
interface. There is a wide range of settings foreseen for Sphider-plus. Separated into different submenus like Sites - Add Site - Index only the new - Re-index all - Re-index only preferred URLs - Erase Re-index (available also for individual URLs) - Import/export URL list - Approve sites - Banned domains Categories - Add, edit, delete - . . .
. . .
URL list - Approve sites - Banned domains Categories - Add, edit, delete - Create new subcategory under Index - Basic indexing options - Advanced options Clean - Clean keywords not associated with any link - Clean links not associated with any site - Clean Category table not associated with any site - Clean Media links - Clear Temp table . . .
. . .
- Clear Search log - Clear 'Most Popular Page Links' log - Clear 'Most Popular Media Links' log - Clear Spider log, separate and bulk delete - Clear Thumbnail images, separate and bulk delete - Clear Text cache - Clear Media cache - Clear IDS log file - Clear flood attempts log file - Clear all entries in addurl or banned table - Truncate all . . .
. . .
flood attempts log file - Clear all entries in addurl or banned table - Truncate all tables in database Settings - General Settings - Index Log Settings - Spider Settings - Search Settings - Order of Result listing - Suggest Options - Page Indexing Weights Database - Configure up to 5 databases with unlimited number of table sets - Activate . . .
. . .
Database - Configure up to 5 databases with unlimited number of table sets - Activate separately for 'Admin', 'Search' user and 'Suggest URL user' - Backup / Restore - Copy / Move - Optimize Templates In order to enable customer's integration of Sphider-plus into existing sites, HTML templates are prepared for Search form Text result . . .
. . .
Search form Text result listing Media result listing Most popular queries etc. Three different designs are offered, which may be selected in submenu 'Settings'. If the layout does not fit the design of your site (which is normal), you may create new designs and modify the appropriate file /templates/My_template/adminstyle.css . . .
. . .
the appropriate file /templates/My_template/adminstyle.css /templates/My_template/userstyle.css Statistics output - Top keywords (Top 50 with hit counter). - All indexed thumbnails w 3e80 ith ID3 and EXIF info. - Larges pages offering link URL and file size. - Most Popular Searches for text links offering Link addr., total clicks, last . . .
. . .
Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Searches for media links offering Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Links (click counter). - Search log offering Query, Results, Queried at, Time taken, User IP, Country, Host name (Latest100) - Index log offering . . .
. . .
Country, Host name (Latest100) - Index log offering File-name, index date and delete option - sitemap log offering sitemap.xml output sitemap list offering file/page suffixes - IDS log offering IP, host, query, impact, involved tags, date and time of intrusion. - Flood attempts log offering IP, query, date and time of flood attempt. - Auto . . .
. . .
attempts log offering IP, query, date and time of flood attempt. - Auto Re-index log file - Server info offering Server software, environment, MySQL, PDF-converter, image functions, php.ini file PHP integration, PHP security info. Each item holding lists of details. All text links, media links and thumbnails are active linked. As stated in . . .
. . .
individual Re-indexing, the periodical Re-indexer could be started and aborted in the "Options" menu of each site. 2.5 Preferred Re-indexing Each new URL added to the Admin backend, could be supplied with a priority level. This level will be used by the option 'Re-index only preferred sites'. Level 1 will be interpreted as most important, while . . .
. . .
less all index results will be stored in log files in sub folder /admin/log/ The names of the log files look like db2_100524-21.47.56_1.html (log file of first thread) db2_100524-21.48.12_2.html (log file of second thread) and is build by the following items db2 - Number of database. 100524 - Date (May 24, 2010) 21.47.56 - Time when this thread . . .
. . .
spider.php -new1 php spider.php -new2 etc. The IDs will be added to the name of the corresponding log files like db2_100524-21.47.56_ID1.html (log file of first thread) db2_100524-21.48.12_ID2.html (log file of second thread) IDs could be defined by personal requirements, but the limitations for file names with respect to the OS should be taken . . .
. . .
but will not erase the content of all the other tables. So the check whether the content of a page has changed (MD5sum) is still available for a fast re-index procedure. Once prepared, multithreaded re-indexing could be invoked by starting several threads and adding individual IDs to the option parameter like php spider.php -erased1 php . . .
. . .
4.4 Ignoring parts of a page Sphider-plus includes an option to exclude parts of pages from being indexed. Thi 775e s can for example be used to prevent search result flooding when certain keywords appear on certain part in most pages (like a header, footer or a menu). Any part of a page between <! sphider_noindex > and <! . . .
. . .
<! sphider_noindex > and <! /sphider_noindex > tags is not indexed, however links in it are followed. 4.5 Ignoring parts of a page by <div id='abc'> Ignoring parts of a page by the <! sphider_noindex > tags requires direct access to the page, because the tags need to be added (edited) to the page. A more flexible method, . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example */ menu[0-5] / 4.6 Indexing only parts of a page by <div id='abc'> If enabled in Admin settings, the values as defined in the list-file /include/common/divs_use.txt will be used to index only the content between <div . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example */ table[0-5] / 4.7 Ignore HTML elements defined by <tagname> . . </tagname> This option is foreseen to cooperate with the new HTML5 elements like section, nav, aside, hgroup, article, header, footer, etc. HTML elements . . .
. . .
contain a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example */nav[0-5]/ Please keep in mind that element names placed in /include/common/elements_not.txt will be processed case-sensitive. 4.8 Index only HTML elements defined by <tagname> . . </tagname> This is the vice versa . . .
. . .
of all the duplicate content URLs http//www.example.com/product.php?item=swedish-fish&category=gummy-candy http//www.example.com/product.php?item=swedish-fish&trackingid=1234&sessionid=5678 and Sphider-plus will understand that the duplicates all refer to the canonical URL http//www.example.com/product.php?item=swedish-fish. The . . .
. . .
charset are supported by the ConvertCharset function and will be used to convert text into UTF-8 Unicode WINDOWS windows-1250 - Central Europe windows-1251 - Cyrillic windows-1252 - Latin I windows-1253 - Greek windows-1254 - Turkish windows-1255 - Hebrew windows-1256 - Arabic windows-1257 - Baltic windows-1258 - Viet Nam cp874 - Thai - this . . .
. . .
- Baltic windows-1258 - Viet Nam cp874 - Thai - this file is also for DOS DOS cp437 - Latin US cp737 - Greek cp775 - BaltRim cp850 - Latin1 cp852 - Latin2 cp855 - Cyrylic cp857 - Turkish cp860 - Portuguese cp861 - Iceland cp862 - Hebrew cp863 - Canada cp864 - Arabic cp865 - Nordic cp866 - Cyrylic Russian (this is the one, used in IE . . .
. . .
IE Cyrillic (DOS) ) cp869 - Greek2 MAC (Apple) x-mac-cyrillic x-mac-greek x-mac-icelandic x-mac-ce x-mac-roman ISO iso-8859-1 iso-8859-2 iso-8859-3 iso-8859-4 iso-8859-5 iso-8859-6 iso-8859-7 iso-8859-8 iso-8859-9 iso-8859-10 iso-8859-11 iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 . . .
. . .
iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 cp500 cp856 cp875 cp1006 cp1026 koi8-r (Cyrillic) koi8-u (Cyrillic Ukrainian) nextstep us-ascii us-ascii-quotes DSP implementation for NeXT stdenc symbol zdingbat And specially for old Polish programs mazovia This list is to be read . . .
. . .
Access denied; you need the RELOAD privilege. . . Top 10. Enable real-time output of logging data Up to version 1.5 of Sphider-plus, during index / re-index there was no printout available because - Several servers, especially on Win32, buffer the output from the script until it terminates before transmitting the results to the browser. - . . .
. . .
is seen. - Some versions of Microsoft Internet Explorer only start to display the page after they have received 256 bytes of output. As progress was not presented during index / re- index procedure, waiting for results became a pain in the neck. Selectable in Admin setting together with the update interval (1 - 10 seconds), AJAX technology was . . .
. . .
characters in front of words Warning This option should be used with special care and not be activated for non ISO-8859 charsets. Some special characters as part of the word ending might be erased by accidental. Top 13. Media search for images, audio streams and videos 13.1 Media indexing Index of media files is enabled by separated Admin . . .
. . .
and 'Found at' - Total clicks - Last clicked - Query input 'Indexed Image Thumbnails' presenting - Thumbnail 150 x 100 pixel - Image details like title, filename size of original image, link- and thumb-id - Option to delete the thumbnail In order to open the media files all tables contain active links. Media results are also stored in . . .
. . .
Settings' menu of the admin backend First option For results of XML product feeds, present the text extract of 350 characters as part of product attributes. If this option is not activated, all products of the XML product feed will be presented in search results, and the hits of the query string will be highlighted in all involved products. . . .
. . .
content of the database tables (those with the same table prefix) will be destroyed by the restore procedure. 16.5 Copy & Move This section of the Database Management will present only those databases, which are correctly configured, assigned and do have a set of installed tables as described in chapter Definition and configuration. This . . .
. . .
results. Never the less to find these x results, the complete database had to been browsed. Starting with version 2.5, an additional clean option is offered as part of the Admin backend. Main advantage of this option is a significant reduction of the search time for any query, because the content of the db could be limited to offer only x . . .
. . .
<link>http//www.abc.de/images/warp.gif</link> <title>warp.gif</title> <x_size>635</x_size> <y_size>98</y_size> </media_result> <media_result> <num>2</num> <type>audio</type> <url>http//www.abc.de/index.php</url> . . .
. . .
the following content {"query""colour","ip""1","host_name""guard_007-hoster","query_time""2016-03-18 105623 AM","consumed"0.016,"total_results"2,"num_of_results"2,"from"1,"to"2,"text_results"[{"num"1,"weight""100.0","url""http\/\/www.english.le-piaggie.info\/Info_eng.pdf","title"" Info_eng","fulltxt"" . . . . . .
. . .
placed on the ground floor, today's big kitchen with the heating fireplace for open and close mode of . . . ","page_size""5,661.6kb","domain_name""www.english.le-piaggie.info"},{"num"2,"weight""50.0","url""http\/\/www.english.le-piaggie.info\/html\/description.html","title"" Description","fulltxt"" . . . cable. Additional active . . .
. . .
D.O.C.G. as far as to the Pratomagno mountains 160 olive-trees Detached house, about 800 m to reach the . . . ","page_size""26.4 kb","domain_name""www.english.le-piaggie.info"}]} Top . . .


5a2 All required information. Introduction Release and Legal Info Installation Documentation Change Log [ Documentation Summary ] Preamble The info presented here is valid only for the latest release of Sphider-plus. At present version 4.2024a published February 12, 2024 is the actual release. 1. Settings, customizing and statistics 2. Indexing . . .
. . .
2. Indexing 2.1 Various options 2.2 Allow other hosts in same domain 2.3 Word stemming 2.4 Periodical Re-indexing 2.5 Preferred indexing 2.6 Multithreaded indexing 2.7 Create thumbnails during index procedure 2.8 Prevent indexing of known malware and pishing pages 2.9 Follow and create sitemap files 2.10 Use private sitemap instead of global . . .
. . .
Sitemap file 3. Using the indexer from command line 3.1 All options 3.2 Multithreaded indexing 4. Keeping pages, words and files from being indexed 4.1 robots.txt 4.2 Must include / must not include string list 4.3 Ignoring links 4.4 Ignoring parts of a page by <! sphider_noindex > 4.5 Ignoring parts of a page by <div id='abc'> . . .
. . .
charset' 6. Search modes 6.1 Search with wildcards * 6.2 Strict search ! 6.3 Tolerant search 6.4 Link search 6.5 Media search 6.6 Search only in one domain 6.7 Search in categories 6.8 Greek language support 6.9 Block queries 7. Chronological order for result listing 7.1 Text result listing 7.2 Media result listing 8. PDF converter 9. Clean . . .
. . .
of logging data 11. Error messages and Debug mode 12. Delete secondary characters 13. Media search for images, audio streams and videos 13.1 Media indexing 13.2 Not supported media content 13.3 Search for media content 13.4 Statistics for media content 14. Feed support 14.1 XML product feeds 14.2 RDF, RSD, RSS and Atom feeds 15. Result . . .
. . .
Overview 16.2 Definition and configuration 16.3 Activate / disable database 16.4 Backup & Restore of databases 16.5 Copy & and Move 16.6 Enhancing functionality of multiple database support 17. Search in categories 17.1 Hierachical structure 17.2 Parallel structure 18. User suggested sites 19. Vulnerability protection 19.1 Prevent queries . . .
. . .
templates 22.2 Embed the search engine into existing HTML code 22.3 The different style sheet files 23. JSON, XML and RSS result output [ Documentation ] 1. Settings, customizing and statistics If you want to change settings, behavior and design of Sphider-plus, you can do so by means of the Admin interface. There is a wide range of . . .
. . .
interface. There is a wide range of settings foreseen for Sphider-plus. Separated into different submenus like Sites - Add Site - Index only the new - Re-index all - Re-index only preferred URLs - Erase Re-index (available also for individual URLs) - Import/export URL list - Approve sites - Banned domains Categories - Add, edit, delete - . . .
. . .
URL list - Approve sites - Banned domains Categories - Add, edit, delete - Create new subcategory under Index - Basic indexing options - Advanced options Clean - Clean keywords not associated with any link - Clean links not associated with any site - Clean Category table not associated with any site - Clean Media links - Clear Temp table . . .
. . .
- Clear Search log - Clear 'Most Popular Page Links' log - Clear 'Most Popular Media Links' log - Clear Spider log, separate and bulk delete - Clear Thumbnail images, separate and bulk delete - Clear Text cache - Clear Media cache - Clear IDS log file - Clear flood attempts log file - Clear all entries in addurl or banned table - Truncate all . . .
. . .
flood attempts log file - Clear all entries in addurl or banned table - Truncate all tables in database Settings - General Settings - Index Log Settings - Spider Settings - Search Settings - Order of Result listing - Suggest Options - Page Indexing Weights Database - Configure up to 5 databases with unlimited number of table sets - Activate . . .
. . .
Database - Configure up to 5 databases with unlimited number of table sets - Activate separately for 'Admin', 'Search' user and 'Suggest URL user' - Backup / Restore - Copy / Move - Optimize Templates In order to enable customer's integration of Sphider-plus into existing sites, HTML templates are prepared for Search form Text result . . .
. . .
Search form Text result listing Media result listing Most popular queries etc. Three different designs are offered, which may be selected in submenu 'Settings'. If the layout does not fit the design of your site (which is normal), you may create new designs and modify the appropriate file /templates/My_template/adminstyle.css . . .
. . .
the appropriate file /templates/My_template/adminstyle.css /templates/My_template/userstyle.css Statistics output - Top keywords (Top 50 with hit counter). - All indexed thumbnails w 3e80 ith ID3 and EXIF info. - Larges pages offering link URL and file size. - Most Popular Searches for text links offering Link addr., total clicks, last . . .
. . .
Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Searches for media links offering Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Links (click counter). - Search log offering Query, Results, Queried at, Time taken, User IP, Country, Host name (Latest100) - Index log offering . . .
. . .
Country, Host name (Latest100) - Index log offering File-name, index date and delete option - sitemap log offering sitemap.xml output sitemap list offering file/page suffixes - IDS log offering IP, host, query, impact, involved tags, date and time of intrusion. - Flood attempts log offering IP, query, date and time of flood attempt. - Auto . . .
. . .
attempts log offering IP, query, date and time of flood attempt. - Auto Re-index log file - Server info offering Server software, environment, MySQL, PDF-converter, image functions, php.ini file PHP integration, PHP security info. Each item holding lists of details. All text links, media links and thumbnails are active linked. As stated in . . .
. . .
individual Re-indexing, the periodical Re-indexer could be started and aborted in the "Options" menu of each site. 2.5 Preferred Re-indexing Each new URL added to the Admin backend, could be supplied with a priority level. This level will be used by the option 'Re-index only preferred sites'. Level 1 will be interpreted as most important, while . . .
. . .
less all index results will be stored in log files in sub folder /admin/log/ The names of the log files look like db2_100524-21.47.56_1.html (log file of first thread) db2_100524-21.48.12_2.html (log file of second thread) and is build by the following items db2 - Number of database. 100524 - Date (May 24, 2010) 21.47.56 - Time when this thread . . .
. . .
spider.php -new1 php spider.php -new2 etc. The IDs will be added to the name of the corresponding log files like db2_100524-21.47.56_ID1.html (log file of first thread) db2_100524-21.48.12_ID2.html (log file of second thread) IDs could be defined by personal requirements, but the limitations for file names with respect to the OS should be taken . . .
. . .
but will not erase the content of all the other tables. So the check whether the content of a page has changed (MD5sum) is still available for a fast re-index procedure. Once prepared, multithreaded re-indexing could be invoked by starting several threads and adding individual IDs to the option parameter like php spider.php -erased1 php . . .
. . .
4.4 Ignoring parts of a page Sphider-plus includes an option to exclude parts of pages from being indexed. Thi 775e s can for example be used to prevent search result flooding when certain keywords appear on certain part in most pages (like a header, footer or a menu). Any part of a page between <! sphider_noindex > and <! . . .
. . .
<! sphider_noindex > and <! /sphider_noindex > tags is not indexed, however links in it are followed. 4.5 Ignoring parts of a page by <div id='abc'> Ignoring parts of a page by the <! sphider_noindex > tags requires direct access to the page, because the tags need to be added (edited) to the page. A more flexible method, . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example */ menu[0-5] / 4.6 Indexing only parts of a page by <div id='abc'> If enabled in Admin settings, the values as defined in the list-file /include/common/divs_use.txt will be used to index only the content between <div . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example */ table[0-5] / 4.7 Ignore HTML elements defined by <tagname> . . </tagname> This option is foreseen to cooperate with the new HTML5 elements like section, nav, aside, hgroup, article, header, footer, etc. HTML elements . . .
. . .
contain a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example */nav[0-5]/ Please keep in mind that element names placed in /include/common/elements_not.txt will be processed case-sensitive. 4.8 Index only HTML elements defined by <tagname> . . </tagname> This is the vice versa . . .
. . .
of all the duplicate content URLs http//www.example.com/product.php?item=swedish-fish&category=gummy-candy http//www.example.com/product.php?item=swedish-fish&trackingid=1234&sessionid=5678 and Sphider-plus will understand that the duplicates all refer to the canonical URL http//www.example.com/product.php?item=swedish-fish. The . . .
. . .
charset are supported by the ConvertCharset function and will be used to convert text into UTF-8 Unicode WINDOWS windows-1250 - Central Europe windows-1251 - Cyrillic windows-1252 - Latin I windows-1253 - Greek windows-1254 - Turkish windows-1255 - Hebrew windows-1256 - Arabic windows-1257 - Baltic windows-1258 - Viet Nam cp874 - Thai - this . . .
. . .
- Baltic windows-1258 - Viet Nam cp874 - Thai - this file is also for DOS DOS cp437 - Latin US cp737 - Greek cp775 - BaltRim cp850 - Latin1 cp852 - Latin2 cp855 - Cyrylic cp857 - Turkish cp860 - Portuguese cp861 - Iceland cp862 - Hebrew cp863 - Canada cp864 - Arabic cp865 - Nordic cp866 - Cyrylic Russian (this is the one, used in IE . . .
. . .
IE Cyrillic (DOS) ) cp869 - Greek2 MAC (Apple) x-mac-cyrillic x-mac-greek x-mac-icelandic x-mac-ce x-mac-roman ISO iso-8859-1 iso-8859-2 iso-8859-3 iso-8859-4 iso-8859-5 iso-8859-6 iso-8859-7 iso-8859-8 iso-8859-9 iso-8859-10 iso-8859-11 iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 . . .
. . .
iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 cp500 cp856 cp875 cp1006 cp1026 koi8-r (Cyrillic) koi8-u (Cyrillic Ukrainian) nextstep us-ascii us-ascii-quotes DSP implementation for NeXT stdenc symbol zdingbat And specially for old Polish programs mazovia This list is to be read . . .
. . .
Access denied; you need the RELOAD privilege. . . Top 10. Enable real-time output of logging data Up to version 1.5 of Sphider-plus, during index / re-index there was no printout available because - Several servers, especially on Win32, buffer the output from the script until it terminates before transmitting the results to the browser. - . . .
. . .
is seen. - Some versions of Microsoft Internet Explorer only start to display the page after they have received 256 bytes of output. As progress was not presented during index / re- index procedure, waiting for results became a pain in the neck. Selectable in Admin setting together with the update interval (1 - 10 seconds), AJAX technology was . . .
. . .
characters in front of words Warning This option should be used with special care and not be activated for non ISO-8859 charsets. Some special characters as part of the word ending might be erased by accidental. Top 13. Media search for images, audio streams and videos 13.1 Media indexing Index of media files is enabled by separated Admin . . .
. . .
and 'Found at' - Total clicks - Last clicked - Query input 'Indexed Image Thumbnails' presenting - Thumbnail 150 x 100 pixel - Image details like title, filename size of original image, link- and thumb-id - Option to delete the thumbnail In order to open the media files all tables contain active links. Media results are also stored in . . .
. . .
Settings' menu of the admin backend First option For results of XML product feeds, present the text extract of 350 characters as part of product attributes. If this option is not activated, all products of the XML product feed will be presented in search results, and the hits of the query string will be highlighted in all involved products. . . .
. . .
content of the database tables (those with the same table prefix) will be destroyed by the restore procedure. 16.5 Copy & Move This section of the Database Management will present only those databases, which are correctly configured, assigned and do have a set of installed tables as described in chapter Definition and configuration. This . . .
. . .
results. Never the less to find these x results, the complete database had to been browsed. Starting with version 2.5, an additional clean option is offered as part of the Admin backend. Main advantage of this option is a significant reduction of the search time for any query, because the content of the db could be limited to offer only x . . .
. . .
<link>http//www.abc.de/images/warp.gif</link> <title>warp.gif</title> <x_size>635</x_size> <y_size>98</y_size> </media_result> <media_result> <num>2</num> <type>audio</type> <url>http//www.abc.de/index.php</url> . . .
. . .
the following content {"query""colour","ip""1","host_name""guard_007-hoster","query_time""2016-03-18 105623 AM","consumed"0.016,"total_results"2,"num_of_results"2,"from"1,"to"2,"text_results"[{"num"1,"weight""100.0","url""http\/\/www.english.le-piaggie.info\/Info_eng.pdf","title"" Info_eng","fulltxt"" . . . . . .
. . .
placed on the ground floor, today's big kitchen with the heating fireplace for open and close mode of . . . ","page_size""5,661.6kb","domain_name""www.english.le-piaggie.info"},{"num"2,"weight""50.0","url""http\/\/www.english.le-piaggie.info\/html\/description.html","title"" Description","fulltxt"" . . . cable. Additional active . . .
. . .
D.O.C.G. as far as to the Pratomagno mountains 160 olive-trees Detached house, about 800 m to reach the . . . ","page_size""26.4 kb","domain_name""www.english.le-piaggie.info"}]} Top . . .


5a2 All required information. Introduction Release and Legal Info Installation Documentation Change Log [ Documentation Summary ] Preamble: The info presented here is valid only for the latest release of Sphider-plus. At present version 4.2024a published February 12, 2024 is the actual release. 1. Settings, customizing and statistics 2. Indexing . . .
. . .
2. Indexing 2.1 Various options 2.2 Allow other hosts in same domain 2.3 Word stemming 2.4 Periodical Re-indexing 2.5 Preferred indexing 2.6 Multithreaded indexing 2.7 Create thumbnails during index procedure 2.8 Prevent indexing of known malware and pishing pages 2.9 Follow and create sitemap files 2.10 Use private sitemap instead of global . . .
. . .
Sitemap file 3. Using the indexer from command line 3.1 All options 3.2 Multithreaded indexing 4. Keeping pages, words and files from being indexed 4.1 robots.txt 4.2 Must include / must not include string list 4.3 Ignoring links 4.4 Ignoring parts of a page by <! sphider_noindex > 4.5 Ignoring parts of a page by <div id='abc'> . . .
. . .
charset' 6. Search modes 6.1 Search with wildcards * 6.2 Strict search ! 6.3 Tolerant search 6.4 Link search 6.5 Media search 6.6 Search only in one domain 6.7 Search in categories 6.8 Greek language support 6.9 Block queries 7. Chronological order for result listing 7.1 Text result listing 7.2 Media result listing 8. PDF converter 9. Clean . . .
. . .
of logging data 11. Error messages and Debug mode 12. Delete secondary characters 13. Media search for images, audio streams and videos 13.1 Media indexing 13.2 Not supported media content 13.3 Search for media content 13.4 Statistics for media content 14. Feed support 14.1 XML product feeds 14.2 RDF, RSD, RSS and Atom feeds 15. Result . . .
. . .
Overview 16.2 Definition and configuration 16.3 Activate / disable database 16.4 Backup & Restore of databases 16.5 Copy & and Move 16.6 Enhancing functionality of multiple database support 17. Search in categories 17.1 Hierachical structure 17.2 Parallel structure 18. User suggested sites 19. Vulnerability protection 19.1 Prevent queries . . .
. . .
templates 22.2 Embed the search engine into existing HTML code 22.3 The different style sheet files 23. JSON, XML and RSS result output [ Documentation ] 1. Settings, customizing and statistics If you want to change settings, behavior and design of Sphider-plus, you can do so by means of the Admin interface. There is a wide range of . . .
. . .
interface. There is a wide range of settings foreseen for Sphider-plus. Separated into different submenus like: Sites: - Add Site - Index only the new - Re-index all - Re-index only preferred URLs - Erase Re-index (available also for individual URLs) - Import/export URL list - Approve sites - Banned domains Categories: - Add, edit, delete - . . .
. . .
URL list - Approve sites - Banned domains Categories: - Add, edit, delete - Create new subcategory under Index: - Basic indexing options - Advanced options Clean: - Clean keywords not associated with any link - Clean links not associated with any site - Clean Category table not associated with any site - Clean Media links - Clear Temp table . . .
. . .
- Clear Search log - Clear 'Most Popular Page Links' log - Clear 'Most Popular Media Links' log - Clear Spider log, separate and bulk delete - Clear Thumbnail images, separate and bulk delete - Clear Text cache - Clear Media cache - Clear IDS log file - Clear flood attempts log file - Clear all entries in addurl or banned table - Truncate all . . .
. . .
flood attempts log file - Clear all entries in addurl or banned table - Truncate all tables in database Settings: - General Settings - Index Log Settings - Spider Settings - Search Settings - Order of Result listing - Suggest Options - Page Indexing Weights Database: - Configure up to 5 databases with unlimited number of table sets - Activate . . .
. . .
Database: - Configure up to 5 databases with unlimited number of table sets - Activate separately for 'Admin', 'Search' user and 'Suggest URL user' - Backup / Restore - Copy / Move - Optimize Templates: In order to enable customer's integration of Sphider-plus into existing sites, HTML templates are prepared for Search form Text result . . .
. . .
Search form Text result listing Media result listing Most popular queries etc. Three different designs are offered, which may be selected in submenu 'Settings'. If the layout does not fit the design of your site (which is normal), you may create new designs and modify the appropriate file /templates/My_template/adminstyle.css . . .
. . .
the appropriate file /templates/My_template/adminstyle.css /templates/My_template/userstyle.css Statistics output: - Top keywords (Top 50 with hit counter). - All indexed thumbnails w 3e80 ith ID3 and EXIF info. - Larges pages offering link URL and file size. - Most Popular Searches for text links offering: Link addr., total clicks, last . . .
. . .
Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Searches for media links offering: Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Links (click counter). - Search log offering: Query, Results, Queried at, Time taken, User IP, Country, Host name (Latest100) - Index log offering: . . .
. . .
Country, Host name (Latest100) - Index log offering: File-name, index date and delete option - sitemap log offering: sitemap.xml output sitemap list offering file/page suffixes - IDS log offering: IP, host, query, impact, involved tags, date and time of intrusion. - Flood attempts log offering: IP, query, date and time of flood attempt. - Auto . . .
. . .
attempts log offering: IP, query, date and time of flood attempt. - Auto Re-index log file - Server info offering: Server software, environment, MySQL, PDF-converter, image functions, php.ini file PHP integration, PHP security info. Each item holding lists of details. All text links, media links and thumbnails are active linked. As stated in . . .
. . .
individual Re-indexing, the periodical Re-indexer could be started and aborted in the "Options" menu of each site. 2.5 Preferred Re-indexing Each new URL added to the Admin backend, could be supplied with a priority level. This level will be used by the option 'Re-index only preferred sites'. Level 1 will be interpreted as most important, while . . .
. . .
less all index results will be stored in log files in sub folder /admin/log/ The names of the log files look like: db2_100524-21.47.56_1.html (log file of first thread) db2_100524-21.48.12_2.html (log file of second thread) and is build by the following items: db2 - Number of database. 100524 - Date (May 24, 2010) 21.47.56 - Time when this thread . . .
. . .
spider.php -new1 php spider.php -new2 etc. The IDs will be added to the name of the corresponding log files like: db2_100524-21.47.56_ID1.html (log file of first thread) db2_100524-21.48.12_ID2.html (log file of second thread) IDs could be defined by personal requirements, but the limitations for file names with respect to the OS should be taken . . .
. . .
but will not erase the content of all the other tables. So the check whether the content of a page has changed (MD5sum) is still available for a fast re-index procedure. Once prepared, multithreaded re-indexing could be invoked by starting several threads and adding individual IDs to the option parameter like: php spider.php -erased1 php . . .
. . .
4.4 Ignoring parts of a page Sphider-plus includes an option to exclude parts of pages from being indexed. Thi 775e s can for example be used to prevent search result flooding when certain keywords appear on certain part in most pages (like a header, footer or a menu). Any part of a page between <! sphider_noindex > and <! . . .
. . .
<! sphider_noindex > and <! /sphider_noindex > tags is not indexed, however links in it are followed. 4.5 Ignoring parts of a page by <div id='abc'> Ignoring parts of a page by the <! sphider_noindex > tags requires direct access to the page, because the tags need to be added (edited) to the page. A more flexible method, . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ menu[0-5] / 4.6 Indexing only parts of a page by <div id='abc'> If enabled in Admin settings, the values as defined in the list-file /include/common/divs_use.txt will be used to index only the content between <div . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ table[0-5] / 4.7 Ignore HTML elements defined by <tagname> . . </tagname> This option is foreseen to cooperate with the new HTML5 elements like section, nav, aside, hgroup, article, header, footer, etc. HTML elements . . .
. . .
contain a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */nav[0-5]/ Please keep in mind that element names placed in /include/common/elements_not.txt will be processed case-sensitive. 4.8 Index only HTML elements defined by <tagname> . . </tagname> This is the vice versa . . .
. . .
of all the duplicate content URLs: http://www.example.com/product.php?item=swedish-fish&category=gummy-candy http://www.example.com/product.php?item=swedish-fish&trackingid=1234&sessionid=5678 and Sphider-plus will understand that the duplicates all refer to the canonical URL: http://www.example.com/product.php?item=swedish-fish. The . . .
. . .
charset are supported by the ConvertCharset function and will be used to convert text into UTF-8 Unicode: WINDOWS windows-1250 - Central Europe windows-1251 - Cyrillic windows-1252 - Latin I windows-1253 - Greek windows-1254 - Turkish windows-1255 - Hebrew windows-1256 - Arabic windows-1257 - Baltic windows-1258 - Viet Nam cp874 - Thai - this . . .
. . .
- Baltic windows-1258 - Viet Nam cp874 - Thai - this file is also for DOS DOS cp437 - Latin US cp737 - Greek cp775 - BaltRim cp850 - Latin1 cp852 - Latin2 cp855 - Cyrylic cp857 - Turkish cp860 - Portuguese cp861 - Iceland cp862 - Hebrew cp863 - Canada cp864 - Arabic cp865 - Nordic cp866 - Cyrylic Russian (this is the one, used in IE . . .
. . .
IE Cyrillic (DOS) ) cp869 - Greek2 MAC (Apple) x-mac-cyrillic x-mac-greek x-mac-icelandic x-mac-ce x-mac-roman ISO iso-8859-1 iso-8859-2 iso-8859-3 iso-8859-4 iso-8859-5 iso-8859-6 iso-8859-7 iso-8859-8 iso-8859-9 iso-8859-10 iso-8859-11 iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 . . .
. . .
iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 cp500 cp856 cp875 cp1006 cp1026 koi8-r (Cyrillic) koi8-u (Cyrillic Ukrainian) nextstep us-ascii us-ascii-quotes DSP implementation for NeXT stdenc symbol zdingbat And specially for old Polish programs: mazovia This list is to be read . . .
. . .
Access denied; you need the RELOAD privilege. . . Top 10. Enable real-time output of logging data Up to version 1.5 of Sphider-plus, during index / re-index there was no printout available because: - Several servers, especially on Win32, buffer the output from the script until it terminates before transmitting the results to the browser. - . . .
. . .
is seen. - Some versions of Microsoft Internet Explorer only start to display the page after they have received 256 bytes of output. As progress was not presented during index / re- index procedure, waiting for results became a pain in the neck. Selectable in Admin setting together with the update interval (1 - 10 seconds), AJAX technology was . . .
. . .
characters in front of words Warning: This option should be used with special care and not be activated for non ISO-8859 charsets. Some special characters as part of the word ending might be erased by accidental. Top 13. Media search for images, audio streams and videos 13.1 Media indexing Index of media files is enabled by separated Admin . . .
. . .
and 'Found at' - Total clicks - Last clicked - Query input 'Indexed Image Thumbnails' presenting: - Thumbnail 150 x 100 pixel - Image details like title, filename size of original image, link- and thumb-id - Option to delete the thumbnail In order to open the media files all tables contain active links. Media results are also stored in . . .
. . .
Settings' menu of the admin backend: First option: For results of XML product feeds, present the text extract of 350 characters as part of product attributes. If this option is not activated, all products of the XML product feed will be presented in search results, and the hits of the query string will be highlighted in all involved products. . . .
. . .
content of the database tables (those with the same table prefix) will be destroyed by the restore procedure. 16.5 Copy & Move This section of the Database Management will present only those databases, which are correctly configured, assigned and do have a set of installed tables as described in chapter Definition and configuration. This . . .
. . .
results. Never the less to find these x results, the complete database had to been browsed. Starting with version 2.5, an additional clean option is offered as part of the Admin backend. Main advantage of this option is a significant reduction of the search time for any query, because the content of the db could be limited to offer only x . . .
. . .
<link>http://www.abc.de/images/warp.gif</link> <title>warp.gif</title> <x_size>635</x_size> <y_size>98</y_size> </media_result> <media_result> <num>2</num> <type>audio</type> <url>http://www.abc.de/index.php</url> . . .
. . .
the following content: {"query":"colour","ip":"::1","host_name":"guard_007-hoster","query_time":"2016-03-18 10:56:23 AM","consumed":0.016,"total_results"2numofresults2from1to2textresults[{num1weight1000urlhttp\/\/wwwenglishle-piaggieinfo\/Infoengpdftitle"num_of_results"2numofresults2from1to2textresults[{num1weight1000urlhttp\/\/wwwenglishle-piaggieinfo\/Infoengpdftitle"from":1,"to"2numofresults2from1to2textresults[{num1weight1000urlhttp\/\/wwwenglishle-piaggieinfo\/Infoengpdftitle"text_results":[{"num":1,"weight":"100.0","url":"http:\/\/www.english.le-piaggie.info\/Info_eng.pdf","title":" Info_eng","fulltxt":" . . . . . .
. . .
placed on the ground floor, today's big kitchen with the heating fireplace for open and close mode of . . . ","page_size":"5,661.6kb","domain_name":"www.english.le-piaggie.info"},{"num"2numofresults2from1to2textresults[{num1weight1000urlhttp\/\/wwwenglishle-piaggieinfo\/Infoengpdftitle"weight":"50.0","url":"http:\/\/www.english.le-piaggie.info\/html\/description.html","title":" Description","fulltxt":" . . . cable. Additional active . . .
. . .
D.O.C.G. as far as to the Pratomagno mountains 160 olive-trees Detached house, about 800 m to reach the . . . ","page_size":"26.4 kb","domain_name":"www.english.le-piaggie.info"}]} Top . . .


5a2 All required information. Introduction Release and Legal Info Installation Documentation Change Log [ Documentation Summary ] Preamble: The info presented here is valid only for the latest release of Sphider-plus. At present version 4.2024a published February 12, 2024 is the actual release. 1. Settings, customizing and statistics 2. Indexing . . .
. . .
2. Indexing 2.1 Various options 2.2 Allow other hosts in same domain 2.3 Word stemming 2.4 Periodical Re-indexing 2.5 Preferred indexing 2.6 Multithreaded indexing 2.7 Create thumbnails during index procedure 2.8 Prevent indexing of known malware and pishing pages 2.9 Follow and create sitemap files 2.10 Use private sitemap instead of global . . .
. . .
Sitemap file 3. Using the indexer from command line 3.1 All options 3.2 Multithreaded indexing 4. Keeping pages, words and files from being indexed 4.1 robots.txt 4.2 Must include / must not include string list 4.3 Ignoring links 4.4 Ignoring parts of a page by <! sphider_noindex > 4.5 Ignoring parts of a page by <div id='abc'> . . .
. . .
charset' 6. Search modes 6.1 Search with wildcards * 6.2 Strict search ! 6.3 Tolerant search 6.4 Link search 6.5 Media search 6.6 Search only in one domain 6.7 Search in categories 6.8 Greek language support 6.9 Block queries 7. Chronological order for result listing 7.1 Text result listing 7.2 Media result listing 8. PDF converter 9. Clean . . .
. . .
of logging data 11. Error messages and Debug mode 12. Delete secondary characters 13. Media search for images, audio streams and videos 13.1 Media indexing 13.2 Not supported media content 13.3 Search for media content 13.4 Statistics for media content 14. Feed support 14.1 XML product feeds 14.2 RDF, RSD, RSS and Atom feeds 15. Result . . .
. . .
Overview 16.2 Definition and configuration 16.3 Activate / disable database 16.4 Backup & Restore of databases 16.5 Copy & and Move 16.6 Enhancing functionality of multiple database support 17. Search in categories 17.1 Hierachical structure 17.2 Parallel structure 18. User suggested sites 19. Vulnerability protection 19.1 Prevent queries . . .
. . .
templates 22.2 Embed the search engine into existing HTML code 22.3 The different style sheet files 23. JSON, XML and RSS result output [ Documentation ] 1. Settings, customizing and statistics If you want to change settings, behavior and design of Sphider-plus, you can do so by means of the Admin interface. There is a wide range of . . .
. . .
interface. There is a wide range of settings foreseen for Sphider-plus. Separated into different submenus like: Sites: - Add Site - Index only the new - Re-index all - Re-index only preferred URLs - Erase Re-index (available also for individual URLs) - Import/export URL list - Approve sites - Banned domains Categories: - Add, edit, delete - . . .
. . .
URL list - Approve sites - Banned domains Categories: - Add, edit, delete - Create new subcategory under Index: - Basic indexing options - Advanced options Clean: - Clean keywords not associated with any link - Clean links not associated with any site - Clean Category table not associated with any site - Clean Media links - Clear Temp table . . .
. . .
- Clear Search log - Clear 'Most Popular Page Links' log - Clear 'Most Popular Media Links' log - Clear Spider log, separate and bulk delete - Clear Thumbnail images, separate and bulk delete - Clear Text cache - Clear Media cache - Clear IDS log file - Clear flood attempts log file - Clear all entries in addurl or banned table - Truncate all . . .
. . .
flood attempts log file - Clear all entries in addurl or banned table - Truncate all tables in database Settings: - General Settings - Index Log Settings - Spider Settings - Search Settings - Order of Result listing - Suggest Options - Page Indexing Weights Database: - Configure up to 5 databases with unlimited number of table sets - Activate . . .
. . .
Database: - Configure up to 5 databases with unlimited number of table sets - Activate separately for 'Admin', 'Search' user and 'Suggest URL user' - Backup / Restore - Copy / Move - Optimize Templates: In order to enable customer's integration of Sphider-plus into existing sites, HTML templates are prepared for Search form Text result . . .
. . .
Search form Text result listing Media result listing Most popular queries etc. Three different designs are offered, which may be selected in submenu 'Settings'. If the layout does not fit the design of your site (which is normal), you may create new designs and modify the appropriate file /templates/My_template/adminstyle.css . . .
. . .
the appropriate file /templates/My_template/adminstyle.css /templates/My_template/userstyle.css Statistics output: - Top keywords (Top 50 with hit counter). - All indexed thumbnails w 3e80 ith ID3 and EXIF info. - Larges pages offering link URL and file size. - Most Popular Searches for text links offering: Link addr., total clicks, last . . .
. . .
Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Searches for media links offering: Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Links (click counter). - Search log offering: Query, Results, Queried at, Time taken, User IP, Country, Host name (Latest100) - Index log offering: . . .
. . .
Country, Host name (Latest100) - Index log offering: File-name, index date and delete option - sitemap log offering: sitemap.xml output sitemap list offering file/page suffixes - IDS log offering: IP, host, query, impact, involved tags, date and time of intrusion. - Flood attempts log offering: IP, query, date and time of flood attempt. - Auto . . .
. . .
attempts log offering: IP, query, date and time of flood attempt. - Auto Re-index log file - Server info offering: Server software, environment, MySQL, PDF-converter, image functions, php.ini file PHP integration, PHP security info. Each item holding lists of details. All text links, media links and thumbnails are active linked. As stated in . . .
. . .
individual Re-indexing, the periodical Re-indexer could be started and aborted in the "Options" menu of each site. 2.5 Preferred Re-indexing Each new URL added to the Admin backend, could be supplied with a priority level. This level will be used by the option 'Re-index only preferred sites'. Level 1 will be interpreted as most important, while . . .
. . .
less all index results will be stored in log files in sub folder /admin/log/ The names of the log files look like: db2_100524-21.47.56_1.html (log file of first thread) db2_100524-21.48.12_2.html (log file of second thread) and is build by the following items: db2 - Number of database. 100524 - Date (May 24, 2010) 21.47.56 - Time when this thread . . .
. . .
spider.php -new1 php spider.php -new2 etc. The IDs will be added to the name of the corresponding log files like: db2_100524-21.47.56_ID1.html (log file of first thread) db2_100524-21.48.12_ID2.html (log file of second thread) IDs could be defined by personal requirements, but the limitations for file names with respect to the OS should be taken . . .
. . .
but will not erase the content of all the other tables. So the check whether the content of a page has changed (MD5sum) is still available for a fast re-index procedure. Once prepared, multithreaded re-indexing could be invoked by starting several threads and adding individual IDs to the option parameter like: php spider.php -erased1 php . . .
. . .
4.4 Ignoring parts of a page Sphider-plus includes an option to exclude parts of pages from being indexed. Thi 775e s can for example be used to prevent search result flooding when certain keywords appear on certain part in most pages (like a header, footer or a menu). Any part of a page between <! sphider_noindex > and <! . . .
. . .
<! sphider_noindex > and <! /sphider_noindex > tags is not indexed, however links in it are followed. 4.5 Ignoring parts of a page by <div id='abc'> Ignoring parts of a page by the <! sphider_noindex > tags requires direct access to the page, because the tags need to be added (edited) to the page. A more flexible method, . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ menu[0-5] / 4.6 Indexing only parts of a page by <div id='abc'> If enabled in Admin settings, the values as defined in the list-file /include/common/divs_use.txt will be used to index only the content between <div . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ table[0-5] / 4.7 Ignore HTML elements defined by <tagname> . . </tagname> This option is foreseen to cooperate with the new HTML5 elements like section, nav, aside, hgroup, article, header, footer, etc. HTML elements . . .
. . .
contain a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */nav[0-5]/ Please keep in mind that element names placed in /include/common/elements_not.txt will be processed case-sensitive. 4.8 Index only HTML elements defined by <tagname> . . </tagname> This is the vice versa . . .
. . .
of all the duplicate content URLs: http://www.example.com/product.php?item=swedish-fish&category=gummy-candy http://www.example.com/product.php?item=swedish-fish&trackingid=1234&sessionid=5678 and Sphider-plus will understand that the duplicates all refer to the canonical URL: http://www.example.com/product.php?item=swedish-fish. The . . .
. . .
charset are supported by the ConvertCharset function and will be used to convert text into UTF-8 Unicode: WINDOWS windows-1250 - Central Europe windows-1251 - Cyrillic windows-1252 - Latin I windows-1253 - Greek windows-1254 - Turkish windows-1255 - Hebrew windows-1256 - Arabic windows-1257 - Baltic windows-1258 - Viet Nam cp874 - Thai - this . . .
. . .
- Baltic windows-1258 - Viet Nam cp874 - Thai - this file is also for DOS DOS cp437 - Latin US cp737 - Greek cp775 - BaltRim cp850 - Latin1 cp852 - Latin2 cp855 - Cyrylic cp857 - Turkish cp860 - Portuguese cp861 - Iceland cp862 - Hebrew cp863 - Canada cp864 - Arabic cp865 - Nordic cp866 - Cyrylic Russian (this is the one, used in IE . . .
. . .
IE Cyrillic (DOS) ) cp869 - Greek2 MAC (Apple) x-mac-cyrillic x-mac-greek x-mac-icelandic x-mac-ce x-mac-roman ISO iso-8859-1 iso-8859-2 iso-8859-3 iso-8859-4 iso-8859-5 iso-8859-6 iso-8859-7 iso-8859-8 iso-8859-9 iso-8859-10 iso-8859-11 iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 . . .
. . .
iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 cp500 cp856 cp875 cp1006 cp1026 koi8-r (Cyrillic) koi8-u (Cyrillic Ukrainian) nextstep us-ascii us-ascii-quotes DSP implementation for NeXT stdenc symbol zdingbat And specially for old Polish programs: mazovia This list is to be read . . .
. . .
Access denied; you need the RELOAD privilege. . . Top 10. Enable real-time output of logging data Up to version 1.5 of Sphider-plus, during index / re-index there was no printout available because: - Several servers, especially on Win32, buffer the output from the script until it terminates before transmitting the results to the browser. - . . .
. . .
is seen. - Some versions of Microsoft Internet Explorer only start to display the page after they have received 256 bytes of output. As progress was not presented during index / re- index procedure, waiting for results became a pain in the neck. Selectable in Admin setting together with the update interval (1 - 10 seconds), AJAX technology was . . .
. . .
characters in front of words Warning: This option should be used with special care and not be activated for non ISO-8859 charsets. Some special characters as part of the word ending might be erased by accidental. Top 13. Media search for images, audio streams and videos 13.1 Media indexing Index of media files is enabled by separated Admin . . .
. . .
and 'Found at' - Total clicks - Last clicked - Query input 'Indexed Image Thumbnails' presenting: - Thumbnail 150 x 100 pixel - Image details like title, filename size of original image, link- and thumb-id - Option to delete the thumbnail In order to open the media files all tables contain active links. Media results are also stored in . . .
. . .
Settings' menu of the admin backend: First option: For results of XML product feeds, present the text extract of 350 characters as part of product attributes. If this option is not activated, all products of the XML product feed will be presented in search results, and the hits of the query string will be highlighted in all involved products. . . .
. . .
content of the database tables (those with the same table prefix) will be destroyed by the restore procedure. 16.5 Copy & Move This section of the Database Management will present only those databases, which are correctly configured, assigned and do have a set of installed tables as described in chapter Definition and configuration. This . . .
. . .
results. Never the less to find these x results, the complete database had to been browsed. Starting with version 2.5, an additional clean option is offered as part of the Admin backend. Main advantage of this option is a significant reduction of the search time for any query, because the content of the db could be limited to offer only x . . .
. . .
<link>http://www.abc.de/images/warp.gif</link> <title>warp.gif</title> <x_size>635</x_size> <y_size>98</y_size> </media_result> <media_result> <num>2</num> <type>audio</type> <url>http://www.abc.de/index.php</url> . . .
. . .
the following content: {"query":"colour","ip":"::1","host_name":"guard_007-hoster","query_time":"2016-03-18 10:56:23 AM","consumed":0.016,"total_results":2,"num_of_results":2,"from":1,"to":2,"text_results":[{"num":1,"weight":"100.0","url":"http:\/\/www.english.le-piaggie.info\/Info_eng.pdf","title":" Info_eng","fulltxt":" . . . . . .
. . .
placed on the ground floor, today's big kitchen with the heating fireplace for open and close mode of . . . ","page_size":"5,661.6kb","domain_name":"www.english.le-piaggie.info"},{"num":2,"weight":"50.0","url":"http:\/\/www.english.le-piaggie.info\/html\/description.html","title":" Description","fulltxt":" . . . cable. Additional active . . .
. . .
D.O.C.G. as far as to the Pratomagno mountains 160 olive-trees Detached house, about 800 m to reach the . . . ","page_size":"26.4 kb","domain_name":"www.english.le-piaggie.info"}]} Top . . .


5a2 All required information. Introduction Release and Legal Info Installation Documentation Change Log [ Documentation Summary ] Preamble: The info presented here is valid only for the latest release of Sphider-plus. At present version 4.2024a published February 12, 2024 is the actual release. 1. Settings, customizing and statistics 2. Indexing . . .
. . .
2. Indexing 2.1 Various options 2.2 Allow other hosts in same domain 2.3 Word stemming 2.4 Periodical Re-indexing 2.5 Preferred indexing 2.6 Multithreaded indexing 2.7 Create thumbnails during index procedure 2.8 Prevent indexing of known malware and pishing pages 2.9 Follow and create sitemap files 2.10 Use private sitemap instead of global . . .
. . .
Sitemap file 3. Using the indexer from command line 3.1 All options 3.2 Multithreaded indexing 4. Keeping pages, words and files from being indexed 4.1 robots.txt 4.2 Must include / must not include string list 4.3 Ignoring links 4.4 Ignoring parts of a page by <! sphider_noindex > 4.5 Ignoring parts of a page by <div id='abc'> . . .
. . .
charset' 6. Search modes 6.1 Search with wildcards * 6.2 Strict search ! 6.3 Tolerant search 6.4 Link search 6.5 Media search 6.6 Search only in one domain 6.7 Search in categories 6.8 Greek language support 6.9 Block queries 7. Chronological order for result listing 7.1 Text result listing 7.2 Media result listing 8. PDF converter 9. Clean . . .
. . .
of logging data 11. Error messages and Debug mode 12. Delete secondary characters 13. Media search for images, audio streams and videos 13.1 Media indexing 13.2 Not supported media content 13.3 Search for media content 13.4 Statistics for media content 14. Feed support 14.1 XML product feeds 14.2 RDF, RSD, RSS and Atom feeds 15. Result . . .
. . .
Overview 16.2 Definition and configuration 16.3 Activate / disable database 16.4 Backup & Restore of databases 16.5 Copy & and Move 16.6 Enhancing functionality of multiple database support 17. Search in categories 17.1 Hierachical structure 17.2 Parallel structure 18. User suggested sites 19. Vulnerability protection 19.1 Prevent queries . . .
. . .
templates 22.2 Embed the search engine into existing HTML code 22.3 The different style sheet files 23. JSON, XML and RSS result output [ Documentation ] 1. Settings, customizing and statistics If you want to change settings, behavior and design of Sphider-plus, you can do so by means of the Admin interface. There is a wide range of . . .
. . .
interface. There is a wide range of settings foreseen for Sphider-plus. Separated into different submenus like: Sites: - Add Site - Index only the new - Re-index all - Re-index only preferred URLs - Erase Re-index (available also for individual URLs) - Import/export URL list - Approve sites - Banned domains Categories: - Add, edit, delete - . . .
. . .
URL list - Approve sites - Banned domains Categories: - Add, edit, delete - Create new subcategory under Index: - Basic indexing options - Advanced options Clean: - Clean keywords not associated with any link - Clean links not associated with any site - Clean Category table not associated with any site - Clean Media links - Clear Temp table . . .
. . .
- Clear Search log - Clear 'Most Popular Page Links' log - Clear 'Most Popular Media Links' log - Clear Spider log, separate and bulk delete - Clear Thumbnail images, separate and bulk delete - Clear Text cache - Clear Media cache - Clear IDS log file - Clear flood attempts log file - Clear all entries in addurl or banned table - Truncate all . . .
. . .
flood attempts log file - Clear all entries in addurl or banned table - Truncate all tables in database Settings: - General Settings - Index Log Settings - Spider Settings - Search Settings - Order of Result listing - Suggest Options - Page Indexing Weights Database: - Configure up to 5 databases with unlimited number of table sets - Activate . . .
. . .
Database: - Configure up to 5 databases with unlimited number of table sets - Activate separately for 'Admin', 'Search' user and 'Suggest URL user' - Backup / Restore - Copy / Move - Optimize Templates: In order to enable customer's integration of Sphider-plus into existing sites, HTML templates are prepared for Search form Text result . . .
. . .
Search form Text result listing Media result listing Most popular queries etc. Three different designs are offered, which may be selected in submenu 'Settings'. If the layout does not fit the design of your site (which is normal), you may create new designs and modify the appropriate file /templates/My_template/adminstyle.css . . .
. . .
the appropriate file /templates/My_template/adminstyle.css /templates/My_template/userstyle.css Statistics output: - Top keywords (Top 50 with hit counter). - All indexed thumbnails w 3e80 ith ID3 and EXIF info. - Larges pages offering link URL and file size. - Most Popular Searches for text links offering: Link addr., total clicks, last . . .
. . .
Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Searches for media links offering: Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Links (click counter). - Search log offering: Query, Results, Queried at, Time taken, User IP, Country, Host name (Latest100) - Index log offering: . . .
. . .
Country, Host name (Latest100) - Index log offering: File-name, index date and delete option - sitemap log offering: sitemap.xml output sitemap list offering file/page suffixes - IDS log offering: IP, host, query, impact, involved tags, date and time of intrusion. - Flood attempts log offering: IP, query, date and time of flood attempt. - Auto . . .
. . .
attempts log offering: IP, query, date and time of flood attempt. - Auto Re-index log file - Server info offering: Server software, environment, MySQL, PDF-converter, image functions, php.ini file PHP integration, PHP security info. Each item holding lists of details. All text links, media links and thumbnails are active linked. As stated in . . .
. . .
individual Re-indexing, the periodical Re-indexer could be started and aborted in the "Options" menu of each site. 2.5 Preferred Re-indexing Each new URL added to the Admin backend, could be supplied with a priority level. This level will be used by the option 'Re-index only preferred sites'. Level 1 will be interpreted as most important, while . . .
. . .
less all index results will be stored in log files in sub folder /admin/log/ The names of the log files look like: db2_100524-21.47.56_1.html (log file of first thread) db2_100524-21.48.12_2.html (log file of second thread) and is build by the following items: db2 - Number of database. 100524 - Date (May 24, 2010) 21.47.56 - Time when this thread . . .
. . .
spider.php -new1 php spider.php -new2 etc. The IDs will be added to the name of the corresponding log files like: db2_100524-21.47.56_ID1.html (log file of first thread) db2_100524-21.48.12_ID2.html (log file of second thread) IDs could be defined by personal requirements, but the limitations for file names with respect to the OS should be taken . . .
. . .
but will not erase the content of all the other tables. So the check whether the content of a page has changed (MD5sum) is still available for a fast re-index procedure. Once prepared, multithreaded re-indexing could be invoked by starting several threads and adding individual IDs to the option parameter like: php spider.php -erased1 php . . .
. . .
4.4 Ignoring parts of a page Sphider-plus includes an option to exclude parts of pages from being indexed. Thi 775e s can for example be used to prevent search result flooding when certain keywords appear on certain part in most pages (like a header, footer or a menu). Any part of a page between <! sphider_noindex > and <! . . .
. . .
<! sphider_noindex > and <! /sphider_noindex > tags is not indexed, however links in it are followed. 4.5 Ignoring parts of a page by <div id='abc'> Ignoring parts of a page by the <! sphider_noindex > tags requires direct access to the page, because the tags need to be added (edited) to the page. A more flexible method, . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ menu[0-5] / 4.6 Indexing only parts of a page by <div id='abc'> If enabled in Admin settings, the values as defined in the list-file /include/common/divs_use.txt will be used to index only the content between <div . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ table[0-5] / 4.7 Ignore HTML elements defined by <tagname> . . </tagname> This option is foreseen to cooperate with the new HTML5 elements like section, nav, aside, hgroup, article, header, footer, etc. HTML elements . . .
. . .
contain a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */nav[0-5]/ Please keep in mind that element names placed in /include/common/elements_not.txt will be processed case-sensitive. 4.8 Index only HTML elements defined by <tagname> . . </tagname> This is the vice versa . . .
. . .
of all the duplicate content URLs: http://www.example.com/product.php?item=swedish-fish&category=gummy-candy http://www.example.com/product.php?item=swedish-fish&trackingid=1234&sessionid=5678 and Sphider-plus will understand that the duplicates all refer to the canonical URL: http://www.example.com/product.php?item=swedish-fish. The . . .
. . .
charset are supported by the ConvertCharset function and will be used to convert text into UTF-8 Unicode: WINDOWS windows-1250 - Central Europe windows-1251 - Cyrillic windows-1252 - Latin I windows-1253 - Greek windows-1254 - Turkish windows-1255 - Hebrew windows-1256 - Arabic windows-1257 - Baltic windows-1258 - Viet Nam cp874 - Thai - this . . .
. . .
- Baltic windows-1258 - Viet Nam cp874 - Thai - this file is also for DOS DOS cp437 - Latin US cp737 - Greek cp775 - BaltRim cp850 - Latin1 cp852 - Latin2 cp855 - Cyrylic cp857 - Turkish cp860 - Portuguese cp861 - Iceland cp862 - Hebrew cp863 - Canada cp864 - Arabic cp865 - Nordic cp866 - Cyrylic Russian (this is the one, used in IE . . .
. . .
IE Cyrillic (DOS) ) cp869 - Greek2 MAC (Apple) x-mac-cyrillic x-mac-greek x-mac-icelandic x-mac-ce x-mac-roman ISO iso-8859-1 iso-8859-2 iso-8859-3 iso-8859-4 iso-8859-5 iso-8859-6 iso-8859-7 iso-8859-8 iso-8859-9 iso-8859-10 iso-8859-11 iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 . . .
. . .
iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 cp500 cp856 cp875 cp1006 cp1026 koi8-r (Cyrillic) koi8-u (Cyrillic Ukrainian) nextstep us-ascii us-ascii-quotes DSP implementation for NeXT stdenc symbol zdingbat And specially for old Polish programs: mazovia This list is to be read . . .
. . .
Access denied; you need the RELOAD privilege. . . Top 10. Enable real-time output of logging data Up to version 1.5 of Sphider-plus, during index / re-index there was no printout available because: - Several servers, especially on Win32, buffer the output from the script until it terminates before transmitting the results to the browser. - . . .
. . .
is seen. - Some versions of Microsoft Internet Explorer only start to display the page after they have received 256 bytes of output. As progress was not presented during index / re- index procedure, waiting for results became a pain in the neck. Selectable in Admin setting together with the update interval (1 - 10 seconds), AJAX technology was . . .
. . .
characters in front of words Warning: This option should be used with special care and not be activated for non ISO-8859 charsets. Some special characters as part of the word ending might be erased by accidental. Top 13. Media search for images, audio streams and videos 13.1 Media indexing Index of media files is enabled by separated Admin . . .
. . .
and 'Found at' - Total clicks - Last clicked - Query input 'Indexed Image Thumbnails' presenting: - Thumbnail 150 x 100 pixel - Image details like title, filename size of original image, link- and thumb-id - Option to delete the thumbnail In order to open the media files all tables contain active links. Media results are also stored in . . .
. . .
Settings' menu of the admin backend: First option: For results of XML product feeds, present the text extract of 350 characters as part of product attributes. If this option is not activated, all products of the XML product feed will be presented in search results, and the hits of the query string will be highlighted in all involved products. . . .
. . .
content of the database tables (those with the same table prefix) will be destroyed by the restore procedure. 16.5 Copy & Move This section of the Database Management will present only those databases, which are correctly configured, assigned and do have a set of installed tables as described in chapter Definition and configuration. This . . .
. . .
results. Never the less to find these x results, the complete database had to been browsed. Starting with version 2.5, an additional clean option is offered as part of the Admin backend. Main advantage of this option is a significant reduction of the search time for any query, because the content of the db could be limited to offer only x . . .
. . .
<link>http://www.abc.de/images/warp.gif</link> <title>warp.gif</title> <x_size>635</x_size> <y_size>98</y_size> </media_result> <media_result> <num>2</num> <type>audio</type> <url>http://www.abc.de/index.php</url> . . .
. . .
the following content: {"query":"colour","ip":"::1","host_name":"guard_007-hoster","query_time":"2016-03-18 10:56:23 AM","consumed":0.016,"total_results":2,"num_of_results":2,"from":1,"to":2,"text_results":[{"num":1,"weight":"100.0","url":"http:\/\/www.english.le-piaggie.info\/Info_eng.pdf","title":" Info_eng","fulltxt":" . . . . . .
. . .
placed on the ground floor, today's big kitchen with the heating fireplace for open and close mode of . . . ","page_size":"5,661.6kb","domain_name":"www.english.le-piaggie.info"},{"num":2,"weight":"50.0","url":"http:\/\/www.english.le-piaggie.info\/html\/description.html","title":" Description","fulltxt":" . . . cable. Additional active . . .
. . .
D.O.C.G. as far as to the Pratomagno mountains 160 olive-trees Detached house, about 800 m to reach the . . . ","page_size":"26.4 kb","domain_name":"www.english.le-piaggie.info"}]} Top . . .


5a2 All required information. Introduction Release and Legal Info Installation Documentation Change Log [ Documentation Summary ] Preamble: The info presented here is valid only for the latest release of Sphider-plus. At present version 4.2024a published February 12, 2024 is the actual release. 1. Settings, customizing and statistics 2. Indexing . . .
. . .
2. Indexing 2.1 Various options 2.2 Allow other hosts in same domain 2.3 Word stemming 2.4 Periodical Re-indexing 2.5 Preferred indexing 2.6 Multithreaded indexing 2.7 Create thumbnails during index procedure 2.8 Prevent indexing of known malware and pishing pages 2.9 Follow and create sitemap files 2.10 Use private sitemap instead of global . . .
. . .
Sitemap file 3. Using the indexer from command line 3.1 All options 3.2 Multithreaded indexing 4. Keeping pages, words and files from being indexed 4.1 robots.txt 4.2 Must include / must not include string list 4.3 Ignoring links 4.4 Ignoring parts of a page by <! sphider_noindex > 4.5 Ignoring parts of a page by <div id='abc'> . . .
. . .
charset' 6. Search modes 6.1 Search with wildcards * 6.2 Strict search ! 6.3 Tolerant search 6.4 Link search 6.5 Media search 6.6 Search only in one domain 6.7 Search in categories 6.8 Greek language support 6.9 Block queries 7. Chronological order for result listing 7.1 Text result listing 7.2 Media result listing 8. PDF converter 9. Clean . . .
. . .
of logging data 11. Error messages and Debug mode 12. Delete secondary characters 13. Media search for images, audio streams and videos 13.1 Media indexing 13.2 Not supported media content 13.3 Search for media content 13.4 Statistics for media content 14. Feed support 14.1 XML product feeds 14.2 RDF, RSD, RSS and Atom feeds 15. Result . . .
. . .
Overview 16.2 Definition and configuration 16.3 Activate / disable database 16.4 Backup & Restore of databases 16.5 Copy & and Move 16.6 Enhancing functionality of multiple database support 17. Search in categories 17.1 Hierachical structure 17.2 Parallel structure 18. User suggested sites 19. Vulnerability protection 19.1 Prevent queries . . .
. . .
templates 22.2 Embed the search engine into existing HTML code 22.3 The different style sheet files 23. JSON, XML and RSS result output [ Documentation ] 1. Settings, customizing and statistics If you want to change settings, behavior and design of Sphider-plus, you can do so by means of the Admin interface. There is a wide range of . . .
. . .
interface. There is a wide range of settings foreseen for Sphider-plus. Separated into different submenus like: Sites: - Add Site - Index only the new - Re-index all - Re-index only preferred URLs - Erase Re-index (available also for individual URLs) - Import/export URL list - Approve sites - Banned domains Categories: - Add, edit, delete - . . .
. . .
URL list - Approve sites - Banned domains Categories: - Add, edit, delete - Create new subcategory under Index: - Basic indexing options - Advanced options Clean: - Clean keywords not associated with any link - Clean links not associated with any site - Clean Category table not associated with any site - Clean Media links - Clear Temp table . . .
. . .
- Clear Search log - Clear 'Most Popular Page Links' log - Clear 'Most Popular Media Links' log - Clear Spider log, separate and bulk delete - Clear Thumbnail images, separate and bulk delete - Clear Text cache - Clear Media cache - Clear IDS log file - Clear flood attempts log file - Clear all entries in addurl or banned table - Truncate all . . .
. . .
flood attempts log file - Clear all entries in addurl or banned table - Truncate all tables in database Settings: - General Settings - Index Log Settings - Spider Settings - Search Settings - Order of Result listing - Suggest Options - Page Indexing Weights Database: - Configure up to 5 databases with unlimited number of table sets - Activate . . .
. . .
Database: - Configure up to 5 databases with unlimited number of table sets - Activate separately for 'Admin', 'Search' user and 'Suggest URL user' - Backup / Restore - Copy / Move - Optimize Templates: In order to enable customer's integration of Sphider-plus into existing sites, HTML templates are prepared for Search form Text result . . .
. . .
Search form Text result listing Media result listing Most popular queries etc. Three different designs are offered, which may be selected in submenu 'Settings'. If the layout does not fit the design of your site (which is normal), you may create new designs and modify the appropriate file /templates/My_template/adminstyle.css . . .
. . .
the appropriate file /templates/My_template/adminstyle.css /templates/My_template/userstyle.css Statistics output: - Top keywords (Top 50 with hit counter). - All indexed thumbnails w 3e80 ith ID3 and EXIF info. - Larges pages offering link URL and file size. - Most Popular Searches for text links offering: Link addr., total clicks, last . . .
. . .
Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Searches for media links offering: Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Links (click counter). - Search log offering: Query, Results, Queried at, Time taken, User IP, Country, Host name (Latest100) - Index log offering: . . .
. . .
Country, Host name (Latest100) - Index log offering: File-name, index date and delete option - sitemap log offering: sitemap.xml output sitemap list offering file/page suffixes - IDS log offering: IP, host, query, impact, involved tags, date and time of intrusion. - Flood attempts log offering: IP, query, date and time of flood attempt. - Auto . . .
. . .
attempts log offering: IP, query, date and time of flood attempt. - Auto Re-index log file - Server info offering: Server software, environment, MySQL, PDF-converter, image functions, php.ini file PHP integration, PHP security info. Each item holding lists of details. All text links, media links and thumbnails are active linked. As stated in . . .
. . .
individual Re-indexing, the periodical Re-indexer could be started and aborted in the "Options" menu of each site. 2.5 Preferred Re-indexing Each new URL added to the Admin backend, could be supplied with a priority level. This level will be used by the option 'Re-index only preferred sites'. Level 1 will be interpreted as most important, while . . .
. . .
less all index results will be stored in log files in sub folder /admin/log/ The names of the log files look like: db2_100524-21.47.56_1.html (log file of first thread) db2_100524-21.48.12_2.html (log file of second thread) and is build by the following items: db2 - Number of database. 100524 - Date (May 24, 2010) 21.47.56 - Time when this thread . . .
. . .
spider.php -new1 php spider.php -new2 etc. The IDs will be added to the name of the corresponding log files like: db2_100524-21.47.56_ID1.html (log file of first thread) db2_100524-21.48.12_ID2.html (log file of second thread) IDs could be defined by personal requirements, but the limitations for file names with respect to the OS should be taken . . .
. . .
but will not erase the content of all the other tables. So the check whether the content of a page has changed (MD5sum) is still available for a fast re-index procedure. Once prepared, multithreaded re-indexing could be invoked by starting several threads and adding individual IDs to the option parameter like: php spider.php -erased1 php . . .
. . .
4.4 Ignoring parts of a page Sphider-plus includes an option to exclude parts of pages from being indexed. Thi 775e s can for example be used to prevent search result flooding when certain keywords appear on certain part in most pages (like a header, footer or a menu). Any part of a page between <! sphider_noindex > and <! . . .
. . .
<! sphider_noindex > and <! /sphider_noindex > tags is not indexed, however links in it are followed. 4.5 Ignoring parts of a page by <div id='abc'> Ignoring parts of a page by the <! sphider_noindex > tags requires direct access to the page, because the tags need to be added (edited) to the page. A more flexible method, . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ menu[0-5] / 4.6 Indexing only parts of a page by <div id='abc'> If enabled in Admin settings, the values as defined in the list-file /include/common/divs_use.txt will be used to index only the content between <div . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ table[0-5] / 4.7 Ignore HTML elements defined by <tagname> . . </tagname> This option is foreseen to cooperate with the new HTML5 elements like section, nav, aside, hgroup, article, header, footer, etc. HTML elements . . .
. . .
contain a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */nav[0-5]/ Please keep in mind that element names placed in /include/common/elements_not.txt will be processed case-sensitive. 4.8 Index only HTML elements defined by <tagname> . . </tagname> This is the vice versa . . .
. . .
of all the duplicate content URLs: http://www.example.com/product.php?item=swedish-fish&category=gummy-candy http://www.example.com/product.php?item=swedish-fish&trackingid=1234&sessionid=5678 and Sphider-plus will understand that the duplicates all refer to the canonical URL: http://www.example.com/product.php?item=swedish-fish. The . . .
. . .
charset are supported by the ConvertCharset function and will be used to convert text into UTF-8 Unicode: WINDOWS windows-1250 - Central Europe windows-1251 - Cyrillic windows-1252 - Latin I windows-1253 - Greek windows-1254 - Turkish windows-1255 - Hebrew windows-1256 - Arabic windows-1257 - Baltic windows-1258 - Viet Nam cp874 - Thai - this . . .
. . .
- Baltic windows-1258 - Viet Nam cp874 - Thai - this file is also for DOS DOS cp437 - Latin US cp737 - Greek cp775 - BaltRim cp850 - Latin1 cp852 - Latin2 cp855 - Cyrylic cp857 - Turkish cp860 - Portuguese cp861 - Iceland cp862 - Hebrew cp863 - Canada cp864 - Arabic cp865 - Nordic cp866 - Cyrylic Russian (this is the one, used in IE . . .
. . .
IE Cyrillic (DOS) ) cp869 - Greek2 MAC (Apple) x-mac-cyrillic x-mac-greek x-mac-icelandic x-mac-ce x-mac-roman ISO iso-8859-1 iso-8859-2 iso-8859-3 iso-8859-4 iso-8859-5 iso-8859-6 iso-8859-7 iso-8859-8 iso-8859-9 iso-8859-10 iso-8859-11 iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 . . .
. . .
iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 cp500 cp856 cp875 cp1006 cp1026 koi8-r (Cyrillic) koi8-u (Cyrillic Ukrainian) nextstep us-ascii us-ascii-quotes DSP implementation for NeXT stdenc symbol zdingbat And specially for old Polish programs: mazovia This list is to be read . . .
. . .
Access denied; you need the RELOAD privilege. . . Top 10. Enable real-time output of logging data Up to version 1.5 of Sphider-plus, during index / re-index there was no printout available because: - Several servers, especially on Win32, buffer the output from the script until it terminates before transmitting the results to the browser. - . . .
. . .
is seen. - Some versions of Microsoft Internet Explorer only start to display the page after they have received 256 bytes of output. As progress was not presented during index / re- index procedure, waiting for results became a pain in the neck. Selectable in Admin setting together with the update interval (1 - 10 seconds), AJAX technology was . . .
. . .
characters in front of words Warning: This option should be used with special care and not be activated for non ISO-8859 charsets. Some special characters as part of the word ending might be erased by accidental. Top 13. Media search for images, audio streams and videos 13.1 Media indexing Index of media files is enabled by separated Admin . . .
. . .
and 'Found at' - Total clicks - Last clicked - Query input 'Indexed Image Thumbnails' presenting: - Thumbnail 150 x 100 pixel - Image details like title, filename size of original image, link- and thumb-id - Option to delete the thumbnail In order to open the media files all tables contain active links. Media results are also stored in . . .
. . .
Settings' menu of the admin backend: First option: For results of XML product feeds, present the text extract of 350 characters as part of product attributes. If this option is not activated, all products of the XML product feed will be presented in search results, and the hits of the query string will be highlighted in all involved products. . . .
. . .
content of the database tables (those with the same table prefix) will be destroyed by the restore procedure. 16.5 Copy & Move This section of the Database Management will present only those databases, which are correctly configured, assigned and do have a set of installed tables as described in chapter Definition and configuration. This . . .
. . .
results. Never the less to find these x results, the complete database had to been browsed. Starting with version 2.5, an additional clean option is offered as part of the Admin backend. Main advantage of this option is a significant reduction of the search time for any query, because the content of the db could be limited to offer only x . . .
. . .
<link>http://www.abc.de/images/warp.gif</link> <title>warp.gif</title> <x_size>635</x_size> <y_size>98</y_size> </media_result> <media_result> <num>2</num> <type>audio</type> <url>http://www.abc.de/index.php</url> . . .
. . .
the following content: {"query":"colour","ip":"::1","host_name":"guard_007-hoster","query_time":"2016-03-18 10:56:23 AM","consumed":0.016,"total_results":2,"num_of_results":2,"from":1,"to":2,"text_results":[{"num":1,"weight":"100.0","url":"http:\/\/www.english.le-piaggie.info\/Info_eng.pdf","title":" Info_eng","fulltxt":" . . . . . .
. . .
placed on the ground floor, today's big kitchen with the heating fireplace for open and close mode of . . . ","page_size":"5,661.6kb","domain_name":"www.english.le-piaggie.info"},{"num":2,"weight":"50.0","url":"http:\/\/www.english.le-piaggie.info\/html\/description.html","title":" Description","fulltxt":" . . . cable. Additional active . . .
. . .
D.O.C.G. as far as to the Pratomagno mountains 160 olive-trees Detached house, about 800 m to reach the . . . ","page_size":"26.4 kb","domain_name":"www.english.le-piaggie.info"}]} Top . . .


5a2 All required information. Introduction Release and Legal Info Installation Documentation Change Log [ Documentation Summary ] Preamble: The info presented here is valid only for the latest release of Sphider-plus. At present version 4.2024a published February 12, 2024 is the actual release. 1. Settings, customizing and statistics 2. Indexing . . .
. . .
2. Indexing 2.1 Various options 2.2 Allow other hosts in same domain 2.3 Word stemming 2.4 Periodical Re-indexing 2.5 Preferred indexing 2.6 Multithreaded indexing 2.7 Create thumbnails during index procedure 2.8 Prevent indexing of known malware and pishing pages 2.9 Follow and create sitemap files 2.10 Use private sitemap instead of global . . .
. . .
Sitemap file 3. Using the indexer from command line 3.1 All options 3.2 Multithreaded indexing 4. Keeping pages, words and files from being indexed 4.1 robots.txt 4.2 Must include / must not include string list 4.3 Ignoring links 4.4 Ignoring parts of a page by <! sphider_noindex > 4.5 Ignoring parts of a page by <div id='abc'> . . .
. . .
charset' 6. Search modes 6.1 Search with wildcards * 6.2 Strict search ! 6.3 Tolerant search 6.4 Link search 6.5 Media search 6.6 Search only in one domain 6.7 Search in categories 6.8 Greek language support 6.9 Block queries 7. Chronological order for result listing 7.1 Text result listing 7.2 Media result listing 8. PDF converter 9. Clean . . .
. . .
of logging data 11. Error messages and Debug mode 12. Delete secondary characters 13. Media search for images, audio streams and videos 13.1 Media indexing 13.2 Not supported media content 13.3 Search for media content 13.4 Statistics for media content 14. Feed support 14.1 XML product feeds 14.2 RDF, RSD, RSS and Atom feeds 15. Result . . .
. . .
Overview 16.2 Definition and configuration 16.3 Activate / disable database 16.4 Backup & Restore of databases 16.5 Copy & and Move 16.6 Enhancing functionality of multiple database support 17. Search in categories 17.1 Hierachical structure 17.2 Parallel structure 18. User suggested sites 19. Vulnerability protection 19.1 Prevent queries . . .
. . .
templates 22.2 Embed the search engine into existing HTML code 22.3 The different style sheet files 23. JSON, XML and RSS result output [ Documentation ] 1. Settings, customizing and statistics If you want to change settings, behavior and design of Sphider-plus, you can do so by means of the Admin interface. There is a wide range of . . .
. . .
interface. There is a wide range of settings foreseen for Sphider-plus. Separated into different submenus like: Sites: - Add Site - Index only the new - Re-index all - Re-index only preferred URLs - Erase Re-index (available also for individual URLs) - Import/export URL list - Approve sites - Banned domains Categories: - Add, edit, delete - . . .
. . .
URL list - Approve sites - Banned domains Categories: - Add, edit, delete - Create new subcategory under Index: - Basic indexing options - Advanced options Clean: - Clean keywords not associated with any link - Clean links not associated with any site - Clean Category table not associated with any site - Clean Media links - Clear Temp table . . .
. . .
- Clear Search log - Clear 'Most Popular Page Links' log - Clear 'Most Popular Media Links' log - Clear Spider log, separate and bulk delete - Clear Thumbnail images, separate and bulk delete - Clear Text cache - Clear Media cache - Clear IDS log file - Clear flood attempts log file - Clear all entries in addurl or banned table - Truncate all . . .
. . .
flood attempts log file - Clear all entries in addurl or banned table - Truncate all tables in database Settings: - General Settings - Index Log Settings - Spider Settings - Search Settings - Order of Result listing - Suggest Options - Page Indexing Weights Database: - Configure up to 5 databases with unlimited number of table sets - Activate . . .
. . .
Database: - Configure up to 5 databases with unlimited number of table sets - Activate separately for 'Admin', 'Search' user and 'Suggest URL user' - Backup / Restore - Copy / Move - Optimize Templates: In order to enable customer's integration of Sphider-plus into existing sites, HTML templates are prepared for Search form Text result . . .
. . .
Search form Text result listing Media result listing Most popular queries etc. Three different designs are offered, which may be selected in submenu 'Settings'. If the layout does not fit the design of your site (which is normal), you may create new designs and modify the appropriate file /templates/My_template/adminstyle.css . . .
. . .
the appropriate file /templates/My_template/adminstyle.css /templates/My_template/userstyle.css Statistics output: - Top keywords (Top 50 with hit counter). - All indexed thumbnails w 3e80 ith ID3 and EXIF info. - Larges pages offering link URL and file size. - Most Popular Searches for text links offering: Link addr., total clicks, last . . .
. . .
Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Searches for media links offering: Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Links (click counter). - Search log offering: Query, Results, Queried at, Time taken, User IP, Country, Host name (Latest100) - Index log offering: . . .
. . .
Country, Host name (Latest100) - Index log offering: File-name, index date and delete option - sitemap log offering: sitemap.xml output sitemap list offering file/page suffixes - IDS log offering: IP, host, query, impact, involved tags, date and time of intrusion. - Flood attempts log offering: IP, query, date and time of flood attempt. - Auto . . .
. . .
attempts log offering: IP, query, date and time of flood attempt. - Auto Re-index log file - Server info offering: Server software, environment, MySQL, PDF-converter, image functions, php.ini file PHP integration, PHP security info. Each item holding lists of details. All text links, media links and thumbnails are active linked. As stated in . . .
. . .
individual Re-indexing, the periodical Re-indexer could be started and aborted in the "Options" menu of each site. 2.5 Preferred Re-indexing Each new URL added to the Admin backend, could be supplied with a priority level. This level will be used by the option 'Re-index only preferred sites'. Level 1 will be interpreted as most important, while . . .
. . .
less all index results will be stored in log files in sub folder /admin/log/ The names of the log files look like: db2_100524-21.47.56_1.html (log file of first thread) db2_100524-21.48.12_2.html (log file of second thread) and is build by the following items: db2 - Number of database. 100524 - Date (May 24, 2010) 21.47.56 - Time when this thread . . .
. . .
spider.php -new1 php spider.php -new2 etc. The IDs will be added to the name of the corresponding log files like: db2_100524-21.47.56_ID1.html (log file of first thread) db2_100524-21.48.12_ID2.html (log file of second thread) IDs could be defined by personal requirements, but the limitations for file names with respect to the OS should be taken . . .
. . .
but will not erase the content of all the other tables. So the check whether the content of a page has changed (MD5sum) is still available for a fast re-index procedure. Once prepared, multithreaded re-indexing could be invoked by starting several threads and adding individual IDs to the option parameter like: php spider.php -erased1 php . . .
. . .
4.4 Ignoring parts of a page Sphider-plus includes an option to exclude parts of pages from being indexed. Thi 775e s can for example be used to prevent search result flooding when certain keywords appear on certain part in most pages (like a header, footer or a menu). Any part of a page between <! sphider_noindex > and <! . . .
. . .
<! sphider_noindex > and <! /sphider_noindex > tags is not indexed, however links in it are followed. 4.5 Ignoring parts of a page by <div id='abc'> Ignoring parts of a page by the <! sphider_noindex > tags requires direct access to the page, because the tags need to be added (edited) to the page. A more flexible method, . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ menu[0-5] / 4.6 Indexing only parts of a page by <div id='abc'> If enabled in Admin settings, the values as defined in the list-file /include/common/divs_use.txt will be used to index only the content between <div . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ table[0-5] / 4.7 Ignore HTML elements defined by <tagname> . . </tagname> This option is foreseen to cooperate with the new HTML5 elements like section, nav, aside, hgroup, article, header, footer, etc. HTML elements . . .
. . .
contain a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */nav[0-5]/ Please keep in mind that element names placed in /include/common/elements_not.txt will be processed case-sensitive. 4.8 Index only HTML elements defined by <tagname> . . </tagname> This is the vice versa . . .
. . .
of all the duplicate content URLs: http://www.example.com/product.php?item=swedish-fish&category=gummy-candy http://www.example.com/product.php?item=swedish-fish&trackingid=1234&sessionid=5678 and Sphider-plus will understand that the duplicates all refer to the canonical URL: http://www.example.com/product.php?item=swedish-fish. The . . .
. . .
charset are supported by the ConvertCharset function and will be used to convert text into UTF-8 Unicode: WINDOWS windows-1250 - Central Europe windows-1251 - Cyrillic windows-1252 - Latin I windows-1253 - Greek windows-1254 - Turkish windows-1255 - Hebrew windows-1256 - Arabic windows-1257 - Baltic windows-1258 - Viet Nam cp874 - Thai - this . . .
. . .
- Baltic windows-1258 - Viet Nam cp874 - Thai - this file is also for DOS DOS cp437 - Latin US cp737 - Greek cp775 - BaltRim cp850 - Latin1 cp852 - Latin2 cp855 - Cyrylic cp857 - Turkish cp860 - Portuguese cp861 - Iceland cp862 - Hebrew cp863 - Canada cp864 - Arabic cp865 - Nordic cp866 - Cyrylic Russian (this is the one, used in IE . . .
. . .
IE Cyrillic (DOS) ) cp869 - Greek2 MAC (Apple) x-mac-cyrillic x-mac-greek x-mac-icelandic x-mac-ce x-mac-roman ISO iso-8859-1 iso-8859-2 iso-8859-3 iso-8859-4 iso-8859-5 iso-8859-6 iso-8859-7 iso-8859-8 iso-8859-9 iso-8859-10 iso-8859-11 iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 . . .
. . .
iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 cp500 cp856 cp875 cp1006 cp1026 koi8-r (Cyrillic) koi8-u (Cyrillic Ukrainian) nextstep us-ascii us-ascii-quotes DSP implementation for NeXT stdenc symbol zdingbat And specially for old Polish programs: mazovia This list is to be read . . .
. . .
Access denied; you need the RELOAD privilege. . . Top 10. Enable real-time output of logging data Up to version 1.5 of Sphider-plus, during index / re-index there was no printout available because: - Several servers, especially on Win32, buffer the output from the script until it terminates before transmitting the results to the browser. - . . .
. . .
is seen. - Some versions of Microsoft Internet Explorer only start to display the page after they have received 256 bytes of output. As progress was not presented during index / re- index procedure, waiting for results became a pain in the neck. Selectable in Admin setting together with the update interval (1 - 10 seconds), AJAX technology was . . .
. . .
characters in front of words Warning: This option should be used with special care and not be activated for non ISO-8859 charsets. Some special characters as part of the word ending might be erased by accidental. Top 13. Media search for images, audio streams and videos 13.1 Media indexing Index of media files is enabled by separated Admin . . .
. . .
and 'Found at' - Total clicks - Last clicked - Query input 'Indexed Image Thumbnails' presenting: - Thumbnail 150 x 100 pixel - Image details like title, filename size of original image, link- and thumb-id - Option to delete the thumbnail In order to open the media files all tables contain active links. Media results are also stored in . . .
. . .
Settings' menu of the admin backend: First option: For results of XML product feeds, present the text extract of 350 characters as part of product attributes. If this option is not activated, all products of the XML product feed will be presented in search results, and the hits of the query string will be highlighted in all involved products. . . .
. . .
content of the database tables (those with the same table prefix) will be destroyed by the restore procedure. 16.5 Copy & Move This section of the Database Management will present only those databases, which are correctly configured, assigned and do have a set of installed tables as described in chapter Definition and configuration. This . . .
. . .
results. Never the less to find these x results, the complete database had to been browsed. Starting with version 2.5, an additional clean option is offered as part of the Admin backend. Main advantage of this option is a significant reduction of the search time for any query, because the content of the db could be limited to offer only x . . .
. . .
<link>http://www.abc.de/images/warp.gif</link> <title>warp.gif</title> <x_size>635</x_size> <y_size>98</y_size> </media_result> <media_result> <num>2</num> <type>audio</type> <url>http://www.abc.de/index.php</url> . . .
. . .
the following content: {"query":"colour","ip":"::1","host_name":"guard_007-hoster","query_time":"2016-03-18 10:56:23 AM","consumed":0.016,"total_results":2,"num_of_results":2,"from":1,"to":2,"text_results":[{"num":1,"weight":"100.0","url":"http:\/\/www.english.le-piaggie.info\/Info_eng.pdf","title":" Info_eng","fulltxt":" . . . . . .
. . .
placed on the ground floor, today's big kitchen with the heating fireplace for open and close mode of . . . ","page_size":"5,661.6kb","domain_name":"www.english.le-piaggie.info"},{"num":2,"weight":"50.0","url":"http:\/\/www.english.le-piaggie.info\/html\/description.html","title":" Description","fulltxt":" . . . cable. Additional active . . .
. . .
D.O.C.G. as far as to the Pratomagno mountains 160 olive-trees Detached house, about 800 m to reach the . . . ","page_size":"26.4 kb","domain_name":"www.english.le-piaggie.info"}]} Top . . .


5a2 All required information. Introduction Release and Legal Info Installation Documentation Change Log [ Documentation Summary ] Preamble: The info presented here is valid only for the latest release of Sphider-plus. At present version 4.2024a published February 12, 2024 is the actual release. 1. Settings, customizing and statistics 2. Indexing . . .
. . .
2. Indexing 2.1 Various options 2.2 Allow other hosts in same domain 2.3 Word stemming 2.4 Periodical Re-indexing 2.5 Preferred indexing 2.6 Multithreaded indexing 2.7 Create thumbnails during index procedure 2.8 Prevent indexing of known malware and pishing pages 2.9 Follow and create sitemap files 2.10 Use private sitemap instead of global . . .
. . .
Sitemap file 3. Using the indexer from command line 3.1 All options 3.2 Multithreaded indexing 4. Keeping pages, words and files from being indexed 4.1 robots.txt 4.2 Must include / must not include string list 4.3 Ignoring links 4.4 Ignoring parts of a page by <! sphider_noindex > 4.5 Ignoring parts of a page by <div id='abc'> . . .
. . .
charset' 6. Search modes 6.1 Search with wildcards * 6.2 Strict search ! 6.3 Tolerant search 6.4 Link search 6.5 Media search 6.6 Search only in one domain 6.7 Search in categories 6.8 Greek language support 6.9 Block queries 7. Chronological order for result listing 7.1 Text result listing 7.2 Media result listing 8. PDF converter 9. Clean . . .
. . .
of logging data 11. Error messages and Debug mode 12. Delete secondary characters 13. Media search for images, audio streams and videos 13.1 Media indexing 13.2 Not supported media content 13.3 Search for media content 13.4 Statistics for media content 14. Feed support 14.1 XML product feeds 14.2 RDF, RSD, RSS and Atom feeds 15. Result . . .
. . .
Overview 16.2 Definition and configuration 16.3 Activate / disable database 16.4 Backup & Restore of databases 16.5 Copy & and Move 16.6 Enhancing functionality of multiple database support 17. Search in categories 17.1 Hierachical structure 17.2 Parallel structure 18. User suggested sites 19. Vulnerability protection 19.1 Prevent queries . . .
. . .
templates 22.2 Embed the search engine into existing HTML code 22.3 The different style sheet files 23. JSON, XML and RSS result output [ Documentation ] 1. Settings, customizing and statistics If you want to change settings, behavior and design of Sphider-plus, you can do so by means of the Admin interface. There is a wide range of . . .
. . .
interface. There is a wide range of settings foreseen for Sphider-plus. Separated into different submenus like: Sites: - Add Site - Index only the new - Re-index all - Re-index only preferred URLs - Erase Re-index (available also for individual URLs) - Import/export URL list - Approve sites - Banned domains Categories: - Add, edit, delete - . . .
. . .
URL list - Approve sites - Banned domains Categories: - Add, edit, delete - Create new subcategory under Index: - Basic indexing options - Advanced options Clean: - Clean keywords not associated with any link - Clean links not associated with any site - Clean Category table not associated with any site - Clean Media links - Clear Temp table . . .
. . .
- Clear Search log - Clear 'Most Popular Page Links' log - Clear 'Most Popular Media Links' log - Clear Spider log, separate and bulk delete - Clear Thumbnail images, separate and bulk delete - Clear Text cache - Clear Media cache - Clear IDS log file - Clear flood attempts log file - Clear all entries in addurl or banned table - Truncate all . . .
. . .
flood attempts log file - Clear all entries in addurl or banned table - Truncate all tables in database Settings: - General Settings - Index Log Settings - Spider Settings - Search Settings - Order of Result listing - Suggest Options - Page Indexing Weights Database: - Configure up to 5 databases with unlimited number of table sets - Activate . . .
. . .
Database: - Configure up to 5 databases with unlimited number of table sets - Activate separately for 'Admin', 'Search' user and 'Suggest URL user' - Backup / Restore - Copy / Move - Optimize Templates: In order to enable customer's integration of Sphider-plus into existing sites, HTML templates are prepared for Search form Text result . . .
. . .
Search form Text result listing Media result listing Most popular queries etc. Three different designs are offered, which may be selected in submenu 'Settings'. If the layout does not fit the design of your site (which is normal), you may create new designs and modify the appropriate file /templates/My_template/adminstyle.css . . .
. . .
the appropriate file /templates/My_template/adminstyle.css /templates/My_template/userstyle.css Statistics output: - Top keywords (Top 50 with hit counter). - All indexed thumbnails w 3e80 ith ID3 and EXIF info. - Larges pages offering link URL and file size. - Most Popular Searches for text links offering: Link addr., total clicks, last . . .
. . .
Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Searches for media links offering: Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Links (click counter). - Search log offering: Query, Results, Queried at, Time taken, User IP, Country, Host name (Latest100) - Index log offering: . . .
. . .
Country, Host name (Latest100) - Index log offering: File-name, index date and delete option - sitemap log offering: sitemap.xml output sitemap list offering file/page suffixes - IDS log offering: IP, host, query, impact, involved tags, date and time of intrusion. - Flood attempts log offering: IP, query, date and time of flood attempt. - Auto . . .
. . .
attempts log offering: IP, query, date and time of flood attempt. - Auto Re-index log file - Server info offering: Server software, environment, MySQL, PDF-converter, image functions, php.ini file PHP integration, PHP security info. Each item holding lists of details. All text links, media links and thumbnails are active linked. As stated in . . .
. . .
individual Re-indexing, the periodical Re-indexer could be started and aborted in the "Options" menu of each site. 2.5 Preferred Re-indexing Each new URL added to the Admin backend, could be supplied with a priority level. This level will be used by the option 'Re-index only preferred sites'. Level 1 will be interpreted as most important, while . . .
. . .
less all index results will be stored in log files in sub folder /admin/log/ The names of the log files look like: db2_100524-21.47.56_1.html (log file of first thread) db2_100524-21.48.12_2.html (log file of second thread) and is build by the following items: db2 - Number of database. 100524 - Date (May 24, 2010) 21.47.56 - Time when this thread . . .
. . .
spider.php -new1 php spider.php -new2 etc. The IDs will be added to the name of the corresponding log files like: db2_100524-21.47.56_ID1.html (log file of first thread) db2_100524-21.48.12_ID2.html (log file of second thread) IDs could be defined by personal requirements, but the limitations for file names with respect to the OS should be taken . . .
. . .
but will not erase the content of all the other tables. So the check whether the content of a page has changed (MD5sum) is still available for a fast re-index procedure. Once prepared, multithreaded re-indexing could be invoked by starting several threads and adding individual IDs to the option parameter like: php spider.php -erased1 php . . .
. . .
4.4 Ignoring parts of a page Sphider-plus includes an option to exclude parts of pages from being indexed. Thi 775e s can for example be used to prevent search result flooding when certain keywords appear on certain part in most pages (like a header, footer or a menu). Any part of a page between <! sphider_noindex > and <! . . .
. . .
<! sphider_noindex > and <! /sphider_noindex > tags is not indexed, however links in it are followed. 4.5 Ignoring parts of a page by <div id='abc'> Ignoring parts of a page by the <! sphider_noindex > tags requires direct access to the page, because the tags need to be added (edited) to the page. A more flexible method, . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ menu[0-5] / 4.6 Indexing only parts of a page by <div id='abc'> If enabled in Admin settings, the values as defined in the list-file /include/common/divs_use.txt will be used to index only the content between <div . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ table[0-5] / 4.7 Ignore HTML elements defined by <tagname> . . </tagname> This option is foreseen to cooperate with the new HTML5 elements like section, nav, aside, hgroup, article, header, footer, etc. HTML elements . . .
. . .
contain a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */nav[0-5]/ Please keep in mind that element names placed in /include/common/elements_not.txt will be processed case-sensitive. 4.8 Index only HTML elements defined by <tagname> . . </tagname> This is the vice versa . . .
. . .
of all the duplicate content URLs: http://www.example.com/product.php?item=swedish-fish&category=gummy-candy http://www.example.com/product.php?item=swedish-fish&trackingid=1234&sessionid=5678 and Sphider-plus will understand that the duplicates all refer to the canonical URL: http://www.example.com/product.php?item=swedish-fish. The . . .
. . .
charset are supported by the ConvertCharset function and will be used to convert text into UTF-8 Unicode: WINDOWS windows-1250 - Central Europe windows-1251 - Cyrillic windows-1252 - Latin I windows-1253 - Greek windows-1254 - Turkish windows-1255 - Hebrew windows-1256 - Arabic windows-1257 - Baltic windows-1258 - Viet Nam cp874 - Thai - this . . .
. . .
- Baltic windows-1258 - Viet Nam cp874 - Thai - this file is also for DOS DOS cp437 - Latin US cp737 - Greek cp775 - BaltRim cp850 - Latin1 cp852 - Latin2 cp855 - Cyrylic cp857 - Turkish cp860 - Portuguese cp861 - Iceland cp862 - Hebrew cp863 - Canada cp864 - Arabic cp865 - Nordic cp866 - Cyrylic Russian (this is the one, used in IE . . .
. . .
IE Cyrillic (DOS) ) cp869 - Greek2 MAC (Apple) x-mac-cyrillic x-mac-greek x-mac-icelandic x-mac-ce x-mac-roman ISO iso-8859-1 iso-8859-2 iso-8859-3 iso-8859-4 iso-8859-5 iso-8859-6 iso-8859-7 iso-8859-8 iso-8859-9 iso-8859-10 iso-8859-11 iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 . . .
. . .
iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 cp500 cp856 cp875 cp1006 cp1026 koi8-r (Cyrillic) koi8-u (Cyrillic Ukrainian) nextstep us-ascii us-ascii-quotes DSP implementation for NeXT stdenc symbol zdingbat And specially for old Polish programs: mazovia This list is to be read . . .
. . .
Access denied; you need the RELOAD privilege. . . Top 10. Enable real-time output of logging data Up to version 1.5 of Sphider-plus, during index / re-index there was no printout available because: - Several servers, especially on Win32, buffer the output from the script until it terminates before transmitting the results to the browser. - . . .
. . .
is seen. - Some versions of Microsoft Internet Explorer only start to display the page after they have received 256 bytes of output. As progress was not presented during index / re- index procedure, waiting for results became a pain in the neck. Selectable in Admin setting together with the update interval (1 - 10 seconds), AJAX technology was . . .
. . .
characters in front of words Warning: This option should be used with special care and not be activated for non ISO-8859 charsets. Some special characters as part of the word ending might be erased by accidental. Top 13. Media search for images, audio streams and videos 13.1 Media indexing Index of media files is enabled by separated Admin . . .
. . .
and 'Found at' - Total clicks - Last clicked - Query input 'Indexed Image Thumbnails' presenting: - Thumbnail 150 x 100 pixel - Image details like title, filename size of original image, link- and thumb-id - Option to delete the thumbnail In order to open the media files all tables contain active links. Media results are also stored in . . .
. . .
Settings' menu of the admin backend: First option: For results of XML product feeds, present the text extract of 350 characters as part of product attributes. If this option is not activated, all products of the XML product feed will be presented in search results, and the hits of the query string will be highlighted in all involved products. . . .
. . .
content of the database tables (those with the same table prefix) will be destroyed by the restore procedure. 16.5 Copy & Move This section of the Database Management will present only those databases, which are correctly configured, assigned and do have a set of installed tables as described in chapter Definition and configuration. This . . .
. . .
results. Never the less to find these x results, the complete database had to been browsed. Starting with version 2.5, an additional clean option is offered as part of the Admin backend. Main advantage of this option is a significant reduction of the search time for any query, because the content of the db could be limited to offer only x . . .
. . .
<link>http://www.abc.de/images/warp.gif</link> <title>warp.gif</title> <x_size>635</x_size> <y_size>98</y_size> </media_result> <media_result> <num>2</num> <type>audio</type> <url>http://www.abc.de/index.php</url> . . .
. . .
the following content: {"query":"colour","ip":"::1","host_name":"guard_007-hoster","query_time":"2016-03-18 10:56:23 AM","consumed":0.016,"total_results":2,"num_of_results":2,"from":1,"to":2,"text_results":[{"num":1,"weight":"100.0","url":"http:\/\/www.english.le-piaggie.info\/Info_eng.pdf","title":" Info_eng","fulltxt":" . . . . . .
. . .
placed on the ground floor, today's big kitchen with the heating fireplace for open and close mode of . . . ","page_size":"5,661.6kb","domain_name":"www.english.le-piaggie.info"},{"num":2,"weight":"50.0","url":"http:\/\/www.english.le-piaggie.info\/html\/description.html","title":" Description","fulltxt":" . . . cable. Additional active . . .
. . .
D.O.C.G. as far as to the Pratomagno mountains 160 olive-trees Detached house, about 800 m to reach the . . . ","page_size":"26.4 kb","domain_name":"www.english.le-piaggie.info"}]} Top . . .


5a2 All required information. Introduction Release and Legal Info Installation Documentation Change Log [ Documentation Summary ] Preamble: The info presented here is valid only for the latest release of Sphider-plus. At present version 4.2024a published February 12, 2024 is the actual release. 1. Settings, customizing and statistics 2. Indexing . . .
. . .
2. Indexing 2.1 Various options 2.2 Allow other hosts in same domain 2.3 Word stemming 2.4 Periodical Re-indexing 2.5 Preferred indexing 2.6 Multithreaded indexing 2.7 Create thumbnails during index procedure 2.8 Prevent indexing of known malware and pishing pages 2.9 Follow and create sitemap files 2.10 Use private sitemap instead of global . . .
. . .
Sitemap file 3. Using the indexer from command line 3.1 All options 3.2 Multithreaded indexing 4. Keeping pages, words and files from being indexed 4.1 robots.txt 4.2 Must include / must not include string list 4.3 Ignoring links 4.4 Ignoring parts of a page by <! sphider_noindex > 4.5 Ignoring parts of a page by <div id='abc'> . . .
. . .
charset' 6. Search modes 6.1 Search with wildcards * 6.2 Strict search ! 6.3 Tolerant search 6.4 Link search 6.5 Media search 6.6 Search only in one domain 6.7 Search in categories 6.8 Greek language support 6.9 Block queries 7. Chronological order for result listing 7.1 Text result listing 7.2 Media result listing 8. PDF converter 9. Clean . . .
. . .
of logging data 11. Error messages and Debug mode 12. Delete secondary characters 13. Media search for images, audio streams and videos 13.1 Media indexing 13.2 Not supported media content 13.3 Search for media content 13.4 Statistics for media content 14. Feed support 14.1 XML product feeds 14.2 RDF, RSD, RSS and Atom feeds 15. Result . . .
. . .
Overview 16.2 Definition and configuration 16.3 Activate / disable database 16.4 Backup & Restore of databases 16.5 Copy & and Move 16.6 Enhancing functionality of multiple database support 17. Search in categories 17.1 Hierachical structure 17.2 Parallel structure 18. User suggested sites 19. Vulnerability protection 19.1 Prevent queries . . .
. . .
templates 22.2 Embed the search engine into existing HTML code 22.3 The different style sheet files 23. JSON, XML and RSS result output [ Documentation ] 1. Settings, customizing and statistics If you want to change settings, behavior and design of Sphider-plus, you can do so by means of the Admin interface. There is a wide range of . . .
. . .
interface. There is a wide range of settings foreseen for Sphider-plus. Separated into different submenus like: Sites: - Add Site - Index only the new - Re-index all - Re-index only preferred URLs - Erase Re-index (available also for individual URLs) - Import/export URL list - Approve sites - Banned domains Categories: - Add, edit, delete - . . .
. . .
URL list - Approve sites - Banned domains Categories: - Add, edit, delete - Create new subcategory under Index: - Basic indexing options - Advanced options Clean: - Clean keywords not associated with any link - Clean links not associated with any site - Clean Category table not associated with any site - Clean Media links - Clear Temp table . . .
. . .
- Clear Search log - Clear 'Most Popular Page Links' log - Clear 'Most Popular Media Links' log - Clear Spider log, separate and bulk delete - Clear Thumbnail images, separate and bulk delete - Clear Text cache - Clear Media cache - Clear IDS log file - Clear flood attempts log file - Clear all entries in addurl or banned table - Truncate all . . .
. . .
flood attempts log file - Clear all entries in addurl or banned table - Truncate all tables in database Settings: - General Settings - Index Log Settings - Spider Settings - Search Settings - Order of Result listing - Suggest Options - Page Indexing Weights Database: - Configure up to 5 databases with unlimited number of table sets - Activate . . .
. . .
Database: - Configure up to 5 databases with unlimited number of table sets - Activate separately for 'Admin', 'Search' user and 'Suggest URL user' - Backup / Restore - Copy / Move - Optimize Templates: In order to enable customer's integration of Sphider-plus into existing sites, HTML templates are prepared for Search form Text result . . .
. . .
Search form Text result listing Media result listing Most popular queries etc. Three different designs are offered, which may be selected in submenu 'Settings'. If the layout does not fit the design of your site (which is normal), you may create new designs and modify the appropriate file /templates/My_template/adminstyle.css . . .
. . .
the appropriate file /templates/My_template/adminstyle.css /templates/My_template/userstyle.css Statistics output: - Top keywords (Top 50 with hit counter). - All indexed thumbnails w 3e80 ith ID3 and EXIF info. - Larges pages offering link URL and file size. - Most Popular Searches for text links offering: Link addr., total clicks, last . . .
. . .
Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Searches for media links offering: Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Links (click counter). - Search log offering: Query, Results, Queried at, Time taken, User IP, Country, Host name (Latest100) - Index log offering: . . .
. . .
Country, Host name (Latest100) - Index log offering: File-name, index date and delete option - sitemap log offering: sitemap.xml output sitemap list offering file/page suffixes - IDS log offering: IP, host, query, impact, involved tags, date and time of intrusion. - Flood attempts log offering: IP, query, date and time of flood attempt. - Auto . . .
. . .
attempts log offering: IP, query, date and time of flood attempt. - Auto Re-index log file - Server info offering: Server software, environment, MySQL, PDF-converter, image functions, php.ini file PHP integration, PHP security info. Each item holding lists of details. All text links, media links and thumbnails are active linked. As stated in . . .
. . .
individual Re-indexing, the periodical Re-indexer could be started and aborted in the "Options" menu of each site. 2.5 Preferred Re-indexing Each new URL added to the Admin backend, could be supplied with a priority level. This level will be used by the option 'Re-index only preferred sites'. Level 1 will be interpreted as most important, while . . .
. . .
less all index results will be stored in log files in sub folder /admin/log/ The names of the log files look like: db2_100524-21.47.56_1.html (log file of first thread) db2_100524-21.48.12_2.html (log file of second thread) and is build by the following items: db2 - Number of database. 100524 - Date (May 24, 2010) 21.47.56 - Time when this thread . . .
. . .
spider.php -new1 php spider.php -new2 etc. The IDs will be added to the name of the corresponding log files like: db2_100524-21.47.56_ID1.html (log file of first thread) db2_100524-21.48.12_ID2.html (log file of second thread) IDs could be defined by personal requirements, but the limitations for file names with respect to the OS should be taken . . .
. . .
but will not erase the content of all the other tables. So the check whether the content of a page has changed (MD5sum) is still available for a fast re-index procedure. Once prepared, multithreaded re-indexing could be invoked by starting several threads and adding individual IDs to the option parameter like: php spider.php -erased1 php . . .
. . .
4.4 Ignoring parts of a page Sphider-plus includes an option to exclude parts of pages from being indexed. Thi 775e s can for example be used to prevent search result flooding when certain keywords appear on certain part in most pages (like a header, footer or a menu). Any part of a page between <! sphider_noindex > and <! . . .
. . .
<! sphider_noindex > and <! /sphider_noindex > tags is not indexed, however links in it are followed. 4.5 Ignoring parts of a page by <div id='abc'> Ignoring parts of a page by the <! sphider_noindex > tags requires direct access to the page, because the tags need to be added (edited) to the page. A more flexible method, . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ menu[0-5] / 4.6 Indexing only parts of a page by <div id='abc'> If enabled in Admin settings, the values as defined in the list-file /include/common/divs_use.txt will be used to index only the content between <div . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ table[0-5] / 4.7 Ignore HTML elements defined by <tagname> . . </tagname> This option is foreseen to cooperate with the new HTML5 elements like section, nav, aside, hgroup, article, header, footer, etc. HTML elements . . .
. . .
contain a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */nav[0-5]/ Please keep in mind that element names placed in /include/common/elements_not.txt will be processed case-sensitive. 4.8 Index only HTML elements defined by <tagname> . . </tagname> This is the vice versa . . .
. . .
of all the duplicate content URLs: http://www.example.com/product.php?item=swedish-fish&category=gummy-candy http://www.example.com/product.php?item=swedish-fish&trackingid=1234&sessionid=5678 and Sphider-plus will understand that the duplicates all refer to the canonical URL: http://www.example.com/product.php?item=swedish-fish. The . . .
. . .
charset are supported by the ConvertCharset function and will be used to convert text into UTF-8 Unicode: WINDOWS windows-1250 - Central Europe windows-1251 - Cyrillic windows-1252 - Latin I windows-1253 - Greek windows-1254 - Turkish windows-1255 - Hebrew windows-1256 - Arabic windows-1257 - Baltic windows-1258 - Viet Nam cp874 - Thai - this . . .
. . .
- Baltic windows-1258 - Viet Nam cp874 - Thai - this file is also for DOS DOS cp437 - Latin US cp737 - Greek cp775 - BaltRim cp850 - Latin1 cp852 - Latin2 cp855 - Cyrylic cp857 - Turkish cp860 - Portuguese cp861 - Iceland cp862 - Hebrew cp863 - Canada cp864 - Arabic cp865 - Nordic cp866 - Cyrylic Russian (this is the one, used in IE . . .
. . .
IE Cyrillic (DOS) ) cp869 - Greek2 MAC (Apple) x-mac-cyrillic x-mac-greek x-mac-icelandic x-mac-ce x-mac-roman ISO iso-8859-1 iso-8859-2 iso-8859-3 iso-8859-4 iso-8859-5 iso-8859-6 iso-8859-7 iso-8859-8 iso-8859-9 iso-8859-10 iso-8859-11 iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 . . .
. . .
iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 cp500 cp856 cp875 cp1006 cp1026 koi8-r (Cyrillic) koi8-u (Cyrillic Ukrainian) nextstep us-ascii us-ascii-quotes DSP implementation for NeXT stdenc symbol zdingbat And specially for old Polish programs: mazovia This list is to be read . . .
. . .
Access denied; you need the RELOAD privilege. . . Top 10. Enable real-time output of logging data Up to version 1.5 of Sphider-plus, during index / re-index there was no printout available because: - Several servers, especially on Win32, buffer the output from the script until it terminates before transmitting the results to the browser. - . . .
. . .
is seen. - Some versions of Microsoft Internet Explorer only start to display the page after they have received 256 bytes of output. As progress was not presented during index / re- index procedure, waiting for results became a pain in the neck. Selectable in Admin setting together with the update interval (1 - 10 seconds), AJAX technology was . . .
. . .
characters in front of words Warning: This option should be used with special care and not be activated for non ISO-8859 charsets. Some special characters as part of the word ending might be erased by accidental. Top 13. Media search for images, audio streams and videos 13.1 Media indexing Index of media files is enabled by separated Admin . . .
. . .
and 'Found at' - Total clicks - Last clicked - Query input 'Indexed Image Thumbnails' presenting: - Thumbnail 150 x 100 pixel - Image details like title, filename size of original image, link- and thumb-id - Option to delete the thumbnail In order to open the media files all tables contain active links. Media results are also stored in . . .
. . .
Settings' menu of the admin backend: First option: For results of XML product feeds, present the text extract of 350 characters as part of product attributes. If this option is not activated, all products of the XML product feed will be presented in search results, and the hits of the query string will be highlighted in all involved products. . . .
. . .
content of the database tables (those with the same table prefix) will be destroyed by the restore procedure. 16.5 Copy & Move This section of the Database Management will present only those databases, which are correctly configured, assigned and do have a set of installed tables as described in chapter Definition and configuration. This . . .
. . .
results. Never the less to find these x results, the complete database had to been browsed. Starting with version 2.5, an additional clean option is offered as part of the Admin backend. Main advantage of this option is a significant reduction of the search time for any query, because the content of the db could be limited to offer only x . . .
. . .
<link>http://www.abc.de/images/warp.gif</link> <title>warp.gif</title> <x_size>635</x_size> <y_size>98</y_size> </media_result> <media_result> <num>2</num> <type>audio</type> <url>http://www.abc.de/index.php</url> . . .
. . .
the following content: {"query":"colour","ip":"::1","host_name":"guard_007-hoster","query_time":"2016-03-18 10:56:23 AM","consumed":0.016,"total_results":2,"num_of_results":2,"from":1,"to":2,"text_results":[{"num":1,"weight":"100.0","url":"http:\/\/www.english.le-piaggie.info\/Info_eng.pdf","title":" Info_eng","fulltxt":" . . . . . .
. . .
placed on the ground floor, today's big kitchen with the heating fireplace for open and close mode of . . . ","page_size":"5,661.6kb","domain_name":"www.english.le-piaggie.info"},{"num":2,"weight":"50.0","url":"http:\/\/www.english.le-piaggie.info\/html\/description.html","title":" Description","fulltxt":" . . . cable. Additional active . . .
. . .
D.O.C.G. as far as to the Pratomagno mountains 160 olive-trees Detached house, about 800 m to reach the . . . ","page_size":"26.4 kb","domain_name":"www.english.le-piaggie.info"}]} Top . . .


5a2 All required information. Introduction Release and Legal Info Installation Documentation Change Log [ Documentation Summary ] Preamble: The info presented here is valid only for the latest release of Sphider-plus. At present version 4.2024a published February 12, 2024 is the actual release. 1. Settings, customizing and statistics 2. Indexing . . .
. . .
2. Indexing 2.1 Various options 2.2 Allow other hosts in same domain 2.3 Word stemming 2.4 Periodical Re-indexing 2.5 Preferred indexing 2.6 Multithreaded indexing 2.7 Create thumbnails during index procedure 2.8 Prevent indexing of known malware and pishing pages 2.9 Follow and create sitemap files 2.10 Use private sitemap instead of global . . .
. . .
Sitemap file 3. Using the indexer from command line 3.1 All options 3.2 Multithreaded indexing 4. Keeping pages, words and files from being indexed 4.1 robots.txt 4.2 Must include / must not include string list 4.3 Ignoring links 4.4 Ignoring parts of a page by <! sphider_noindex > 4.5 Ignoring parts of a page by <div id='abc'> . . .
. . .
charset' 6. Search modes 6.1 Search with wildcards * 6.2 Strict search ! 6.3 Tolerant search 6.4 Link search 6.5 Media search 6.6 Search only in one domain 6.7 Search in categories 6.8 Greek language support 6.9 Block queries 7. Chronological order for result listing 7.1 Text result listing 7.2 Media result listing 8. PDF converter 9. Clean . . .
. . .
of logging data 11. Error messages and Debug mode 12. Delete secondary characters 13. Media search for images, audio streams and videos 13.1 Media indexing 13.2 Not supported media content 13.3 Search for media content 13.4 Statistics for media content 14. Feed support 14.1 XML product feeds 14.2 RDF, RSD, RSS and Atom feeds 15. Result . . .
. . .
Overview 16.2 Definition and configuration 16.3 Activate / disable database 16.4 Backup & Restore of databases 16.5 Copy & and Move 16.6 Enhancing functionality of multiple database support 17. Search in categories 17.1 Hierachical structure 17.2 Parallel structure 18. User suggested sites 19. Vulnerability protection 19.1 Prevent queries . . .
. . .
templates 22.2 Embed the search engine into existing HTML code 22.3 The different style sheet files 23. JSON, XML and RSS result output [ Documentation ] 1. Settings, customizing and statistics If you want to change settings, behavior and design of Sphider-plus, you can do so by means of the Admin interface. There is a wide range of . . .
. . .
interface. There is a wide range of settings foreseen for Sphider-plus. Separated into different submenus like: Sites: - Add Site - Index only the new - Re-index all - Re-index only preferred URLs - Erase Re-index (available also for individual URLs) - Import/export URL list - Approve sites - Banned domains Categories: - Add, edit, delete - . . .
. . .
URL list - Approve sites - Banned domains Categories: - Add, edit, delete - Create new subcategory under Index: - Basic indexing options - Advanced options Clean: - Clean keywords not associated with any link - Clean links not associated with any site - Clean Category table not associated with any site - Clean Media links - Clear Temp table . . .
. . .
- Clear Search log - Clear 'Most Popular Page Links' log - Clear 'Most Popular Media Links' log - Clear Spider log, separate and bulk delete - Clear Thumbnail images, separate and bulk delete - Clear Text cache - Clear Media cache - Clear IDS log file - Clear flood attempts log file - Clear all entries in addurl or banned table - Truncate all . . .
. . .
flood attempts log file - Clear all entries in addurl or banned table - Truncate all tables in database Settings: - General Settings - Index Log Settings - Spider Settings - Search Settings - Order of Result listing - Suggest Options - Page Indexing Weights Database: - Configure up to 5 databases with unlimited number of table sets - Activate . . .
. . .
Database: - Configure up to 5 databases with unlimited number of table sets - Activate separately for 'Admin', 'Search' user and 'Suggest URL user' - Backup / Restore - Copy / Move - Optimize Templates: In order to enable customer's integration of Sphider-plus into existing sites, HTML templates are prepared for Search form Text result . . .
. . .
Search form Text result listing Media result listing Most popular queries etc. Three different designs are offered, which may be selected in submenu 'Settings'. If the layout does not fit the design of your site (which is normal), you may create new designs and modify the appropriate file /templates/My_template/adminstyle.css . . .
. . .
the appropriate file /templates/My_template/adminstyle.css /templates/My_template/userstyle.css Statistics output: - Top keywords (Top 50 with hit counter). - All indexed thumbnails w 3e80 ith ID3 and EXIF info. - Larges pages offering link URL and file size. - Most Popular Searches for text links offering: Link addr., total clicks, last . . .
. . .
Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Searches for media links offering: Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Links (click counter). - Search log offering: Query, Results, Queried at, Time taken, User IP, Country, Host name (Latest100) - Index log offering: . . .
. . .
Country, Host name (Latest100) - Index log offering: File-name, index date and delete option - sitemap log offering: sitemap.xml output sitemap list offering file/page suffixes - IDS log offering: IP, host, query, impact, involved tags, date and time of intrusion. - Flood attempts log offering: IP, query, date and time of flood attempt. - Auto . . .
. . .
attempts log offering: IP, query, date and time of flood attempt. - Auto Re-index log file - Server info offering: Server software, environment, MySQL, PDF-converter, image functions, php.ini file PHP integration, PHP security info. Each item holding lists of details. All text links, media links and thumbnails are active linked. As stated in . . .
. . .
individual Re-indexing, the periodical Re-indexer could be started and aborted in the "Options" menu of each site. 2.5 Preferred Re-indexing Each new URL added to the Admin backend, could be supplied with a priority level. This level will be used by the option 'Re-index only preferred sites'. Level 1 will be interpreted as most important, while . . .
. . .
less all index results will be stored in log files in sub folder /admin/log/ The names of the log files look like: db2_100524-21.47.56_1.html (log file of first thread) db2_100524-21.48.12_2.html (log file of second thread) and is build by the following items: db2 - Number of database. 100524 - Date (May 24, 2010) 21.47.56 - Time when this thread . . .
. . .
spider.php -new1 php spider.php -new2 etc. The IDs will be added to the name of the corresponding log files like: db2_100524-21.47.56_ID1.html (log file of first thread) db2_100524-21.48.12_ID2.html (log file of second thread) IDs could be defined by personal requirements, but the limitations for file names with respect to the OS should be taken . . .
. . .
but will not erase the content of all the other tables. So the check whether the content of a page has changed (MD5sum) is still available for a fast re-index procedure. Once prepared, multithreaded re-indexing could be invoked by starting several threads and adding individual IDs to the option parameter like: php spider.php -erased1 php . . .
. . .
4.4 Ignoring parts of a page Sphider-plus includes an option to exclude parts of pages from being indexed. Thi 775e s can for example be used to prevent search result flooding when certain keywords appear on certain part in most pages (like a header, footer or a menu). Any part of a page between <! sphider_noindex > and <! . . .
. . .
<! sphider_noindex > and <! /sphider_noindex > tags is not indexed, however links in it are followed. 4.5 Ignoring parts of a page by <div id='abc'> Ignoring parts of a page by the <! sphider_noindex > tags requires direct access to the page, because the tags need to be added (edited) to the page. A more flexible method, . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ menu[0-5] / 4.6 Indexing only parts of a page by <div id='abc'> If enabled in Admin settings, the values as defined in the list-file /include/common/divs_use.txt will be used to index only the content between <div . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ table[0-5] / 4.7 Ignore HTML elements defined by <tagname> . . </tagname> This option is foreseen to cooperate with the new HTML5 elements like section, nav, aside, hgroup, article, header, footer, etc. HTML elements . . .
. . .
contain a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */nav[0-5]/ Please keep in mind that element names placed in /include/common/elements_not.txt will be processed case-sensitive. 4.8 Index only HTML elements defined by <tagname> . . </tagname> This is the vice versa . . .
. . .
of all the duplicate content URLs: http://www.example.com/product.php?item=swedish-fish&category=gummy-candy http://www.example.com/product.php?item=swedish-fish&trackingid=1234&sessionid=5678 and Sphider-plus will understand that the duplicates all refer to the canonical URL: http://www.example.com/product.php?item=swedish-fish. The . . .
. . .
charset are supported by the ConvertCharset function and will be used to convert text into UTF-8 Unicode: WINDOWS windows-1250 - Central Europe windows-1251 - Cyrillic windows-1252 - Latin I windows-1253 - Greek windows-1254 - Turkish windows-1255 - Hebrew windows-1256 - Arabic windows-1257 - Baltic windows-1258 - Viet Nam cp874 - Thai - this . . .
. . .
- Baltic windows-1258 - Viet Nam cp874 - Thai - this file is also for DOS DOS cp437 - Latin US cp737 - Greek cp775 - BaltRim cp850 - Latin1 cp852 - Latin2 cp855 - Cyrylic cp857 - Turkish cp860 - Portuguese cp861 - Iceland cp862 - Hebrew cp863 - Canada cp864 - Arabic cp865 - Nordic cp866 - Cyrylic Russian (this is the one, used in IE . . .
. . .
IE Cyrillic (DOS) ) cp869 - Greek2 MAC (Apple) x-mac-cyrillic x-mac-greek x-mac-icelandic x-mac-ce x-mac-roman ISO iso-8859-1 iso-8859-2 iso-8859-3 iso-8859-4 iso-8859-5 iso-8859-6 iso-8859-7 iso-8859-8 iso-8859-9 iso-8859-10 iso-8859-11 iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 . . .
. . .
iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 cp500 cp856 cp875 cp1006 cp1026 koi8-r (Cyrillic) koi8-u (Cyrillic Ukrainian) nextstep us-ascii us-ascii-quotes DSP implementation for NeXT stdenc symbol zdingbat And specially for old Polish programs: mazovia This list is to be read . . .
. . .
Access denied; you need the RELOAD privilege. . . Top 10. Enable real-time output of logging data Up to version 1.5 of Sphider-plus, during index / re-index there was no printout available because: - Several servers, especially on Win32, buffer the output from the script until it terminates before transmitting the results to the browser. - . . .
. . .
is seen. - Some versions of Microsoft Internet Explorer only start to display the page after they have received 256 bytes of output. As progress was not presented during index / re- index procedure, waiting for results became a pain in the neck. Selectable in Admin setting together with the update interval (1 - 10 seconds), AJAX technology was . . .
. . .
characters in front of words Warning: This option should be used with special care and not be activated for non ISO-8859 charsets. Some special characters as part of the word ending might be erased by accidental. Top 13. Media search for images, audio streams and videos 13.1 Media indexing Index of media files is enabled by separated Admin . . .
. . .
and 'Found at' - Total clicks - Last clicked - Query input 'Indexed Image Thumbnails' presenting: - Thumbnail 150 x 100 pixel - Image details like title, filename size of original image, link- and thumb-id - Option to delete the thumbnail In order to open the media files all tables contain active links. Media results are also stored in . . .
. . .
Settings' menu of the admin backend: First option: For results of XML product feeds, present the text extract of 350 characters as part of product attributes. If this option is not activated, all products of the XML product feed will be presented in search results, and the hits of the query string will be highlighted in all involved products. . . .
. . .
content of the database tables (those with the same table prefix) will be destroyed by the restore procedure. 16.5 Copy & Move This section of the Database Management will present only those databases, which are correctly configured, assigned and do have a set of installed tables as described in chapter Definition and configuration. This . . .
. . .
results. Never the less to find these x results, the complete database had to been browsed. Starting with version 2.5, an additional clean option is offered as part of the Admin backend. Main advantage of this option is a significant reduction of the search time for any query, because the content of the db could be limited to offer only x . . .
. . .
<link>http://www.abc.de/images/warp.gif</link> <title>warp.gif</title> <x_size>635</x_size> <y_size>98</y_size> </media_result> <media_result> <num>2</num> <type>audio</type> <url>http://www.abc.de/index.php</url> . . .
. . .
the following content: {"query":"colour","ip":"::1","host_name":"guard_007-hoster","query_time":"2016-03-18 10:56:23 AM","consumed":0.016,"total_results":2,"num_of_results":2,"from":1,"to":2,"text_results":[{"num":1,"weight":"100.0","url":"http:\/\/www.english.le-piaggie.info\/Info_eng.pdf","title":" Info_eng","fulltxt":" . . . . . .
. . .
placed on the ground floor, today's big kitchen with the heating fireplace for open and close mode of . . . ","page_size":"5,661.6kb","domain_name":"www.english.le-piaggie.info"},{"num":2,"weight":"50.0","url":"http:\/\/www.english.le-piaggie.info\/html\/description.html","title":" Description","fulltxt":" . . . cable. Additional active . . .
. . .
D.O.C.G. as far as to the Pratomagno mountains 160 olive-trees Detached house, about 800 m to reach the . . . ","page_size":"26.4 kb","domain_name":"www.english.le-piaggie.info"}]} Top . . .


5a2 All required information. Introduction Release and Legal Info Installation Documentation Change Log [ Documentation Summary ] Preamble: The info presented here is valid only for the latest release of Sphider-plus. At present version 4.2024a published February 12, 2024 is the actual release. 1. Settings, customizing and statistics 2. Indexing . . .
. . .
2. Indexing 2.1 Various options 2.2 Allow other hosts in same domain 2.3 Word stemming 2.4 Periodical Re-indexing 2.5 Preferred indexing 2.6 Multithreaded indexing 2.7 Create thumbnails during index procedure 2.8 Prevent indexing of known malware and pishing pages 2.9 Follow and create sitemap files 2.10 Use private sitemap instead of global . . .
. . .
Sitemap file 3. Using the indexer from command line 3.1 All options 3.2 Multithreaded indexing 4. Keeping pages, words and files from being indexed 4.1 robots.txt 4.2 Must include / must not include string list 4.3 Ignoring links 4.4 Ignoring parts of a page by <! sphider_noindex > 4.5 Ignoring parts of a page by <div id='abc'> . . .
. . .
charset' 6. Search modes 6.1 Search with wildcards * 6.2 Strict search ! 6.3 Tolerant search 6.4 Link search 6.5 Media search 6.6 Search only in one domain 6.7 Search in categories 6.8 Greek language support 6.9 Block queries 7. Chronological order for result listing 7.1 Text result listing 7.2 Media result listing 8. PDF converter 9. Clean . . .
. . .
of logging data 11. Error messages and Debug mode 12. Delete secondary characters 13. Media search for images, audio streams and videos 13.1 Media indexing 13.2 Not supported media content 13.3 Search for media content 13.4 Statistics for media content 14. Feed support 14.1 XML product feeds 14.2 RDF, RSD, RSS and Atom feeds 15. Result . . .
. . .
Overview 16.2 Definition and configuration 16.3 Activate / disable database 16.4 Backup & Restore of databases 16.5 Copy & and Move 16.6 Enhancing functionality of multiple database support 17. Search in categories 17.1 Hierachical structure 17.2 Parallel structure 18. User suggested sites 19. Vulnerability protection 19.1 Prevent queries . . .
. . .
templates 22.2 Embed the search engine into existing HTML code 22.3 The different style sheet files 23. JSON, XML and RSS result output [ Documentation ] 1. Settings, customizing and statistics If you want to change settings, behavior and design of Sphider-plus, you can do so by means of the Admin interface. There is a wide range of . . .
. . .
interface. There is a wide range of settings foreseen for Sphider-plus. Separated into different submenus like: Sites: - Add Site - Index only the new - Re-index all - Re-index only preferred URLs - Erase Re-index (available also for individual URLs) - Import/export URL list - Approve sites - Banned domains Categories: - Add, edit, delete - . . .
. . .
URL list - Approve sites - Banned domains Categories: - Add, edit, delete - Create new subcategory under Index: - Basic indexing options - Advanced options Clean: - Clean keywords not associated with any link - Clean links not associated with any site - Clean Category table not associated with any site - Clean Media links - Clear Temp table . . .
. . .
- Clear Search log - Clear 'Most Popular Page Links' log - Clear 'Most Popular Media Links' log - Clear Spider log, separate and bulk delete - Clear Thumbnail images, separate and bulk delete - Clear Text cache - Clear Media cache - Clear IDS log file - Clear flood attempts log file - Clear all entries in addurl or banned table - Truncate all . . .
. . .
flood attempts log file - Clear all entries in addurl or banned table - Truncate all tables in database Settings: - General Settings - Index Log Settings - Spider Settings - Search Settings - Order of Result listing - Suggest Options - Page Indexing Weights Database: - Configure up to 5 databases with unlimited number of table sets - Activate . . .
. . .
Database: - Configure up to 5 databases with unlimited number of table sets - Activate separately for 'Admin', 'Search' user and 'Suggest URL user' - Backup / Restore - Copy / Move - Optimize Templates: In order to enable customer's integration of Sphider-plus into existing sites, HTML templates are prepared for Search form Text result . . .
. . .
Search form Text result listing Media result listing Most popular queries etc. Three different designs are offered, which may be selected in submenu 'Settings'. If the layout does not fit the design of your site (which is normal), you may create new designs and modify the appropriate file /templates/My_template/adminstyle.css . . .
. . .
the appropriate file /templates/My_template/adminstyle.css /templates/My_template/userstyle.css Statistics output: - Top keywords (Top 50 with hit counter). - All indexed thumbnails w 3e80 ith ID3 and EXIF info. - Larges pages offering link URL and file size. - Most Popular Searches for text links offering: Link addr., total clicks, last . . .
. . .
Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Searches for media links offering: Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Links (click counter). - Search log offering: Query, Results, Queried at, Time taken, User IP, Country, Host name (Latest100) - Index log offering: . . .
. . .
Country, Host name (Latest100) - Index log offering: File-name, index date and delete option - sitemap log offering: sitemap.xml output sitemap list offering file/page suffixes - IDS log offering: IP, host, query, impact, involved tags, date and time of intrusion. - Flood attempts log offering: IP, query, date and time of flood attempt. - Auto . . .
. . .
attempts log offering: IP, query, date and time of flood attempt. - Auto Re-index log file - Server info offering: Server software, environment, MySQL, PDF-converter, image functions, php.ini file PHP integration, PHP security info. Each item holding lists of details. All text links, media links and thumbnails are active linked. As stated in . . .
. . .
individual Re-indexing, the periodical Re-indexer could be started and aborted in the "Options" menu of each site. 2.5 Preferred Re-indexing Each new URL added to the Admin backend, could be supplied with a priority level. This level will be used by the option 'Re-index only preferred sites'. Level 1 will be interpreted as most important, while . . .
. . .
less all index results will be stored in log files in sub folder /admin/log/ The names of the log files look like: db2_100524-21.47.56_1.html (log file of first thread) db2_100524-21.48.12_2.html (log file of second thread) and is build by the following items: db2 - Number of database. 100524 - Date (May 24, 2010) 21.47.56 - Time when this thread . . .
. . .
spider.php -new1 php spider.php -new2 etc. The IDs will be added to the name of the corresponding log files like: db2_100524-21.47.56_ID1.html (log file of first thread) db2_100524-21.48.12_ID2.html (log file of second thread) IDs could be defined by personal requirements, but the limitations for file names with respect to the OS should be taken . . .
. . .
but will not erase the content of all the other tables. So the check whether the content of a page has changed (MD5sum) is still available for a fast re-index procedure. Once prepared, multithreaded re-indexing could be invoked by starting several threads and adding individual IDs to the option parameter like: php spider.php -erased1 php . . .
. . .
4.4 Ignoring parts of a page Sphider-plus includes an option to exclude parts of pages from being indexed. Thi 775e s can for example be used to prevent search result flooding when certain keywords appear on certain part in most pages (like a header, footer or a menu). Any part of a page between <! sphider_noindex > and <! . . .
. . .
<! sphider_noindex > and <! /sphider_noindex > tags is not indexed, however links in it are followed. 4.5 Ignoring parts of a page by <div id='abc'> Ignoring parts of a page by the <! sphider_noindex > tags requires direct access to the page, because the tags need to be added (edited) to the page. A more flexible method, . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ menu[0-5] / 4.6 Indexing only parts of a page by <div id='abc'> If enabled in Admin settings, the values as defined in the list-file /include/common/divs_use.txt will be used to index only the content between <div . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ table[0-5] / 4.7 Ignore HTML elements defined by <tagname> . . </tagname> This option is foreseen to cooperate with the new HTML5 elements like section, nav, aside, hgroup, article, header, footer, etc. HTML elements . . .
. . .
contain a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */nav[0-5]/ Please keep in mind that element names placed in /include/common/elements_not.txt will be processed case-sensitive. 4.8 Index only HTML elements defined by <tagname> . . </tagname> This is the vice versa . . .
. . .
of all the duplicate content URLs: http://www.example.com/product.php?item=swedish-fish&category=gummy-candy http://www.example.com/product.php?item=swedish-fish&trackingid=1234&sessionid=5678 and Sphider-plus will understand that the duplicates all refer to the canonical URL: http://www.example.com/product.php?item=swedish-fish. The . . .
. . .
charset are supported by the ConvertCharset function and will be used to convert text into UTF-8 Unicode: WINDOWS windows-1250 - Central Europe windows-1251 - Cyrillic windows-1252 - Latin I windows-1253 - Greek windows-1254 - Turkish windows-1255 - Hebrew windows-1256 - Arabic windows-1257 - Baltic windows-1258 - Viet Nam cp874 - Thai - this . . .
. . .
- Baltic windows-1258 - Viet Nam cp874 - Thai - this file is also for DOS DOS cp437 - Latin US cp737 - Greek cp775 - BaltRim cp850 - Latin1 cp852 - Latin2 cp855 - Cyrylic cp857 - Turkish cp860 - Portuguese cp861 - Iceland cp862 - Hebrew cp863 - Canada cp864 - Arabic cp865 - Nordic cp866 - Cyrylic Russian (this is the one, used in IE . . .
. . .
IE Cyrillic (DOS) ) cp869 - Greek2 MAC (Apple) x-mac-cyrillic x-mac-greek x-mac-icelandic x-mac-ce x-mac-roman ISO iso-8859-1 iso-8859-2 iso-8859-3 iso-8859-4 iso-8859-5 iso-8859-6 iso-8859-7 iso-8859-8 iso-8859-9 iso-8859-10 iso-8859-11 iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 . . .
. . .
iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 cp500 cp856 cp875 cp1006 cp1026 koi8-r (Cyrillic) koi8-u (Cyrillic Ukrainian) nextstep us-ascii us-ascii-quotes DSP implementation for NeXT stdenc symbol zdingbat And specially for old Polish programs: mazovia This list is to be read . . .
. . .
Access denied; you need the RELOAD privilege. . . Top 10. Enable real-time output of logging data Up to version 1.5 of Sphider-plus, during index / re-index there was no printout available because: - Several servers, especially on Win32, buffer the output from the script until it terminates before transmitting the results to the browser. - . . .
. . .
is seen. - Some versions of Microsoft Internet Explorer only start to display the page after they have received 256 bytes of output. As progress was not presented during index / re- index procedure, waiting for results became a pain in the neck. Selectable in Admin setting together with the update interval (1 - 10 seconds), AJAX technology was . . .
. . .
characters in front of words Warning: This option should be used with special care and not be activated for non ISO-8859 charsets. Some special characters as part of the word ending might be erased by accidental. Top 13. Media search for images, audio streams and videos 13.1 Media indexing Index of media files is enabled by separated Admin . . .
. . .
and 'Found at' - Total clicks - Last clicked - Query input 'Indexed Image Thumbnails' presenting: - Thumbnail 150 x 100 pixel - Image details like title, filename size of original image, link- and thumb-id - Option to delete the thumbnail In order to open the media files all tables contain active links. Media results are also stored in . . .
. . .
Settings' menu of the admin backend: First option: For results of XML product feeds, present the text extract of 350 characters as part of product attributes. If this option is not activated, all products of the XML product feed will be presented in search results, and the hits of the query string will be highlighted in all involved products. . . .
. . .
content of the database tables (those with the same table prefix) will be destroyed by the restore procedure. 16.5 Copy & Move This section of the Database Management will present only those databases, which are correctly configured, assigned and do have a set of installed tables as described in chapter Definition and configuration. This . . .
. . .
results. Never the less to find these x results, the complete database had to been browsed. Starting with version 2.5, an additional clean option is offered as part of the Admin backend. Main advantage of this option is a significant reduction of the search time for any query, because the content of the db could be limited to offer only x . . .
. . .
<link>http://www.abc.de/images/warp.gif</link> <title>warp.gif</title> <x_size>635</x_size> <y_size>98</y_size> </media_result> <media_result> <num>2</num> <type>audio</type> <url>http://www.abc.de/index.php</url> . . .
. . .
the following content: {"query":"colour","ip":"::1","host_name":"guard_007-hoster","query_time":"2016-03-18 10:56:23 AM","consumed":0.016,"total_results":2,"num_of_results":2,"from":1,"to":2,"text_results":[{"num":1,"weight":"100.0","url":"http:\/\/www.english.le-piaggie.info\/Info_eng.pdf","title":" Info_eng","fulltxt":" . . . . . .
. . .
placed on the ground floor, today's big kitchen with the heating fireplace for open and close mode of . . . ","page_size":"5,661.6kb","domain_name":"www.english.le-piaggie.info"},{"num":2,"weight":"50.0","url":"http:\/\/www.english.le-piaggie.info\/html\/description.html","title":" Description","fulltxt":" . . . cable. Additional active . . .
. . .
D.O.C.G. as far as to the Pratomagno mountains 160 olive-trees Detached house, about 800 m to reach the . . . ","page_size":"26.4 kb","domain_name":"www.english.le-piaggie.info"}]} Top . . .


5a2 All required information. Introduction Release and Legal Info Installation Documentation Change Log [ Documentation Summary ] Preamble: The info presented here is valid only for the latest release of Sphider-plus. At present version 4.2024a published February 12, 2024 is the actual release. 1. Settings, customizing and statistics 2. Indexing . . .
. . .
2. Indexing 2.1 Various options 2.2 Allow other hosts in same domain 2.3 Word stemming 2.4 Periodical Re-indexing 2.5 Preferred indexing 2.6 Multithreaded indexing 2.7 Create thumbnails during index procedure 2.8 Prevent indexing of known malware and pishing pages 2.9 Follow and create sitemap files 2.10 Use private sitemap instead of global . . .
. . .
Sitemap file 3. Using the indexer from command line 3.1 All options 3.2 Multithreaded indexing 4. Keeping pages, words and files from being indexed 4.1 robots.txt 4.2 Must include / must not include string list 4.3 Ignoring links 4.4 Ignoring parts of a page by <! sphider_noindex > 4.5 Ignoring parts of a page by <div id='abc'> . . .
. . .
charset' 6. Search modes 6.1 Search with wildcards * 6.2 Strict search ! 6.3 Tolerant search 6.4 Link search 6.5 Media search 6.6 Search only in one domain 6.7 Search in categories 6.8 Greek language support 6.9 Block queries 7. Chronological order for result listing 7.1 Text result listing 7.2 Media result listing 8. PDF converter 9. Clean . . .
. . .
of logging data 11. Error messages and Debug mode 12. Delete secondary characters 13. Media search for images, audio streams and videos 13.1 Media indexing 13.2 Not supported media content 13.3 Search for media content 13.4 Statistics for media content 14. Feed support 14.1 XML product feeds 14.2 RDF, RSD, RSS and Atom feeds 15. Result . . .
. . .
Overview 16.2 Definition and configuration 16.3 Activate / disable database 16.4 Backup & Restore of databases 16.5 Copy & and Move 16.6 Enhancing functionality of multiple database support 17. Search in categories 17.1 Hierachical structure 17.2 Parallel structure 18. User suggested sites 19. Vulnerability protection 19.1 Prevent queries . . .
. . .
templates 22.2 Embed the search engine into existing HTML code 22.3 The different style sheet files 23. JSON, XML and RSS result output [ Documentation ] 1. Settings, customizing and statistics If you want to change settings, behavior and design of Sphider-plus, you can do so by means of the Admin interface. There is a wide range of . . .
. . .
interface. There is a wide range of settings foreseen for Sphider-plus. Separated into different submenus like: Sites: - Add Site - Index only the new - Re-index all - Re-index only preferred URLs - Erase Re-index (available also for individual URLs) - Import/export URL list - Approve sites - Banned domains Categories: - Add, edit, delete - . . .
. . .
URL list - Approve sites - Banned domains Categories: - Add, edit, delete - Create new subcategory under Index: - Basic indexing options - Advanced options Clean: - Clean keywords not associated with any link - Clean links not associated with any site - Clean Category table not associated with any site - Clean Media links - Clear Temp table . . .
. . .
- Clear Search log - Clear 'Most Popular Page Links' log - Clear 'Most Popular Media Links' log - Clear Spider log, separate and bulk delete - Clear Thumbnail images, separate and bulk delete - Clear Text cache - Clear Media cache - Clear IDS log file - Clear flood attempts log file - Clear all entries in addurl or banned table - Truncate all . . .
. . .
flood attempts log file - Clear all entries in addurl or banned table - Truncate all tables in database Settings: - General Settings - Index Log Settings - Spider Settings - Search Settings - Order of Result listing - Suggest Options - Page Indexing Weights Database: - Configure up to 5 databases with unlimited number of table sets - Activate . . .
. . .
Database: - Configure up to 5 databases with unlimited number of table sets - Activate separately for 'Admin', 'Search' user and 'Suggest URL user' - Backup / Restore - Copy / Move - Optimize Templates: In order to enable customer's integration of Sphider-plus into existing sites, HTML templates are prepared for Search form Text result . . .
. . .
Search form Text result listing Media result listing Most popular queries etc. Three different designs are offered, which may be selected in submenu 'Settings'. If the layout does not fit the design of your site (which is normal), you may create new designs and modify the appropriate file /templates/My_template/adminstyle.css . . .
. . .
the appropriate file /templates/My_template/adminstyle.css /templates/My_template/userstyle.css Statistics output: - Top keywords (Top 50 with hit counter). - All indexed thumbnails w 3e80 ith ID3 and EXIF info. - Larges pages offering link URL and file size. - Most Popular Searches for text links offering: Link addr., total clicks, last . . .
. . .
Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Searches for media links offering: Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Links (click counter). - Search log offering: Query, Results, Queried at, Time taken, User IP, Country, Host name (Latest100) - Index log offering: . . .
. . .
Country, Host name (Latest100) - Index log offering: File-name, index date and delete option - sitemap log offering: sitemap.xml output sitemap list offering file/page suffixes - IDS log offering: IP, host, query, impact, involved tags, date and time of intrusion. - Flood attempts log offering: IP, query, date and time of flood attempt. - Auto . . .
. . .
attempts log offering: IP, query, date and time of flood attempt. - Auto Re-index log file - Server info offering: Server software, environment, MySQL, PDF-converter, image functions, php.ini file PHP integration, PHP security info. Each item holding lists of details. All text links, media links and thumbnails are active linked. As stated in . . .
. . .
individual Re-indexing, the periodical Re-indexer could be started and aborted in the "Options" menu of each site. 2.5 Preferred Re-indexing Each new URL added to the Admin backend, could be supplied with a priority level. This level will be used by the option 'Re-index only preferred sites'. Level 1 will be interpreted as most important, while . . .
. . .
less all index results will be stored in log files in sub folder /admin/log/ The names of the log files look like: db2_100524-21.47.56_1.html (log file of first thread) db2_100524-21.48.12_2.html (log file of second thread) and is build by the following items: db2 - Number of database. 100524 - Date (May 24, 2010) 21.47.56 - Time when this thread . . .
. . .
spider.php -new1 php spider.php -new2 etc. The IDs will be added to the name of the corresponding log files like: db2_100524-21.47.56_ID1.html (log file of first thread) db2_100524-21.48.12_ID2.html (log file of second thread) IDs could be defined by personal requirements, but the limitations for file names with respect to the OS should be taken . . .
. . .
but will not erase the content of all the other tables. So the check whether the content of a page has changed (MD5sum) is still available for a fast re-index procedure. Once prepared, multithreaded re-indexing could be invoked by starting several threads and adding individual IDs to the option parameter like: php spider.php -erased1 php . . .
. . .
4.4 Ignoring parts of a page Sphider-plus includes an option to exclude parts of pages from being indexed. Thi 775e s can for example be used to prevent search result flooding when certain keywords appear on certain part in most pages (like a header, footer or a menu). Any part of a page between <! sphider_noindex > and <! . . .
. . .
<! sphider_noindex > and <! /sphider_noindex > tags is not indexed, however links in it are followed. 4.5 Ignoring parts of a page by <div id='abc'> Ignoring parts of a page by the <! sphider_noindex > tags requires direct access to the page, because the tags need to be added (edited) to the page. A more flexible method, . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ menu[0-5] / 4.6 Indexing only parts of a page by <div id='abc'> If enabled in Admin settings, the values as defined in the list-file /include/common/divs_use.txt will be used to index only the content between <div . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ table[0-5] / 4.7 Ignore HTML elements defined by <tagname> . . </tagname> This option is foreseen to cooperate with the new HTML5 elements like section, nav, aside, hgroup, article, header, footer, etc. HTML elements . . .
. . .
contain a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */nav[0-5]/ Please keep in mind that element names placed in /include/common/elements_not.txt will be processed case-sensitive. 4.8 Index only HTML elements defined by <tagname> . . </tagname> This is the vice versa . . .
. . .
of all the duplicate content URLs: http://www.example.com/product.php?item=swedish-fish&category=gummy-candy http://www.example.com/product.php?item=swedish-fish&trackingid=1234&sessionid=5678 and Sphider-plus will understand that the duplicates all refer to the canonical URL: http://www.example.com/product.php?item=swedish-fish. The . . .
. . .
charset are supported by the ConvertCharset function and will be used to convert text into UTF-8 Unicode: WINDOWS windows-1250 - Central Europe windows-1251 - Cyrillic windows-1252 - Latin I windows-1253 - Greek windows-1254 - Turkish windows-1255 - Hebrew windows-1256 - Arabic windows-1257 - Baltic windows-1258 - Viet Nam cp874 - Thai - this . . .
. . .
- Baltic windows-1258 - Viet Nam cp874 - Thai - this file is also for DOS DOS cp437 - Latin US cp737 - Greek cp775 - BaltRim cp850 - Latin1 cp852 - Latin2 cp855 - Cyrylic cp857 - Turkish cp860 - Portuguese cp861 - Iceland cp862 - Hebrew cp863 - Canada cp864 - Arabic cp865 - Nordic cp866 - Cyrylic Russian (this is the one, used in IE . . .
. . .
IE Cyrillic (DOS) ) cp869 - Greek2 MAC (Apple) x-mac-cyrillic x-mac-greek x-mac-icelandic x-mac-ce x-mac-roman ISO iso-8859-1 iso-8859-2 iso-8859-3 iso-8859-4 iso-8859-5 iso-8859-6 iso-8859-7 iso-8859-8 iso-8859-9 iso-8859-10 iso-8859-11 iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 . . .
. . .
iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 cp500 cp856 cp875 cp1006 cp1026 koi8-r (Cyrillic) koi8-u (Cyrillic Ukrainian) nextstep us-ascii us-ascii-quotes DSP implementation for NeXT stdenc symbol zdingbat And specially for old Polish programs: mazovia This list is to be read . . .
. . .
Access denied; you need the RELOAD privilege. . . Top 10. Enable real-time output of logging data Up to version 1.5 of Sphider-plus, during index / re-index there was no printout available because: - Several servers, especially on Win32, buffer the output from the script until it terminates before transmitting the results to the browser. - . . .
. . .
is seen. - Some versions of Microsoft Internet Explorer only start to display the page after they have received 256 bytes of output. As progress was not presented during index / re- index procedure, waiting for results became a pain in the neck. Selectable in Admin setting together with the update interval (1 - 10 seconds), AJAX technology was . . .
. . .
characters in front of words Warning: This option should be used with special care and not be activated for non ISO-8859 charsets. Some special characters as part of the word ending might be erased by accidental. Top 13. Media search for images, audio streams and videos 13.1 Media indexing Index of media files is enabled by separated Admin . . .
. . .
and 'Found at' - Total clicks - Last clicked - Query input 'Indexed Image Thumbnails' presenting: - Thumbnail 150 x 100 pixel - Image details like title, filename size of original image, link- and thumb-id - Option to delete the thumbnail In order to open the media files all tables contain active links. Media results are also stored in . . .
. . .
Settings' menu of the admin backend: First option: For results of XML product feeds, present the text extract of 350 characters as part of product attributes. If this option is not activated, all products of the XML product feed will be presented in search results, and the hits of the query string will be highlighted in all involved products. . . .
. . .
content of the database tables (those with the same table prefix) will be destroyed by the restore procedure. 16.5 Copy & Move This section of the Database Management will present only those databases, which are correctly configured, assigned and do have a set of installed tables as described in chapter Definition and configuration. This . . .
. . .
results. Never the less to find these x results, the complete database had to been browsed. Starting with version 2.5, an additional clean option is offered as part of the Admin backend. Main advantage of this option is a significant reduction of the search time for any query, because the content of the db could be limited to offer only x . . .
. . .
<link>http://www.abc.de/images/warp.gif</link> <title>warp.gif</title> <x_size>635</x_size> <y_size>98</y_size> </media_result> <media_result> <num>2</num> <type>audio</type> <url>http://www.abc.de/index.php</url> . . .
. . .
the following content: {"query":"colour","ip":"::1","host_name":"guard_007-hoster","query_time":"2016-03-18 10:56:23 AM","consumed":0.016,"total_results":2,"num_of_results":2,"from":1,"to":2,"text_results":[{"num":1,"weight":"100.0","url":"http:\/\/www.english.le-piaggie.info\/Info_eng.pdf","title":" Info_eng","fulltxt":" . . . . . .
. . .
placed on the ground floor, today's big kitchen with the heating fireplace for open and close mode of . . . ","page_size":"5,661.6kb","domain_name":"www.english.le-piaggie.info"},{"num":2,"weight":"50.0","url":"http:\/\/www.english.le-piaggie.info\/html\/description.html","title":" Description","fulltxt":" . . . cable. Additional active . . .
. . .
D.O.C.G. as far as to the Pratomagno mountains 160 olive-trees Detached house, about 800 m to reach the . . . ","page_size":"26.4 kb","domain_name":"www.english.le-piaggie.info"}]} Top . . .


5a2 All required information. Introduction Release and Legal Info Installation Documentation Change Log [ Documentation Summary ] Preamble: The info presented here is valid only for the latest release of Sphider-plus. At present version 4.2024a published February 12, 2024 is the actual release. 1. Settings, customizing and statistics 2. Indexing . . .
. . .
2. Indexing 2.1 Various options 2.2 Allow other hosts in same domain 2.3 Word stemming 2.4 Periodical Re-indexing 2.5 Preferred indexing 2.6 Multithreaded indexing 2.7 Create thumbnails during index procedure 2.8 Prevent indexing of known malware and pishing pages 2.9 Follow and create sitemap files 2.10 Use private sitemap instead of global . . .
. . .
Sitemap file 3. Using the indexer from command line 3.1 All options 3.2 Multithreaded indexing 4. Keeping pages, words and files from being indexed 4.1 robots.txt 4.2 Must include / must not include string list 4.3 Ignoring links 4.4 Ignoring parts of a page by <! sphider_noindex > 4.5 Ignoring parts of a page by <div id='abc'> . . .
. . .
charset' 6. Search modes 6.1 Search with wildcards * 6.2 Strict search ! 6.3 Tolerant search 6.4 Link search 6.5 Media search 6.6 Search only in one domain 6.7 Search in categories 6.8 Greek language support 6.9 Block queries 7. Chronological order for result listing 7.1 Text result listing 7.2 Media result listing 8. PDF converter 9. Clean . . .
. . .
of logging data 11. Error messages and Debug mode 12. Delete secondary characters 13. Media search for images, audio streams and videos 13.1 Media indexing 13.2 Not supported media content 13.3 Search for media content 13.4 Statistics for media content 14. Feed support 14.1 XML product feeds 14.2 RDF, RSD, RSS and Atom feeds 15. Result . . .
. . .
Overview 16.2 Definition and configuration 16.3 Activate / disable database 16.4 Backup & Restore of databases 16.5 Copy & and Move 16.6 Enhancing functionality of multiple database support 17. Search in categories 17.1 Hierachical structure 17.2 Parallel structure 18. User suggested sites 19. Vulnerability protection 19.1 Prevent queries . . .
. . .
templates 22.2 Embed the search engine into existing HTML code 22.3 The different style sheet files 23. JSON, XML and RSS result output [ Documentation ] 1. Settings, customizing and statistics If you want to change settings, behavior and design of Sphider-plus, you can do so by means of the Admin interface. There is a wide range of . . .
. . .
interface. There is a wide range of settings foreseen for Sphider-plus. Separated into different submenus like: Sites: - Add Site - Index only the new - Re-index all - Re-index only preferred URLs - Erase Re-index (available also for individual URLs) - Import/export URL list - Approve sites - Banned domains Categories: - Add, edit, delete - . . .
. . .
URL list - Approve sites - Banned domains Categories: - Add, edit, delete - Create new subcategory under Index: - Basic indexing options - Advanced options Clean: - Clean keywords not associated with any link - Clean links not associated with any site - Clean Category table not associated with any site - Clean Media links - Clear Temp table . . .
. . .
- Clear Search log - Clear 'Most Popular Page Links' log - Clear 'Most Popular Media Links' log - Clear Spider log, separate and bulk delete - Clear Thumbnail images, separate and bulk delete - Clear Text cache - Clear Media cache - Clear IDS log file - Clear flood attempts log file - Clear all entries in addurl or banned table - Truncate all . . .
. . .
flood attempts log file - Clear all entries in addurl or banned table - Truncate all tables in database Settings: - General Settings - Index Log Settings - Spider Settings - Search Settings - Order of Result listing - Suggest Options - Page Indexing Weights Database: - Configure up to 5 databases with unlimited number of table sets - Activate . . .
. . .
Database: - Configure up to 5 databases with unlimited number of table sets - Activate separately for 'Admin', 'Search' user and 'Suggest URL user' - Backup / Restore - Copy / Move - Optimize Templates: In order to enable customer's integration of Sphider-plus into existing sites, HTML templates are prepared for Search form Text result . . .
. . .
Search form Text result listing Media result listing Most popular queries etc. Three different designs are offered, which may be selected in submenu 'Settings'. If the layout does not fit the design of your site (which is normal), you may create new designs and modify the appropriate file /templates/My_template/adminstyle.css . . .
. . .
the appropriate file /templates/My_template/adminstyle.css /templates/My_template/userstyle.css Statistics output: - Top keywords (Top 50 with hit counter). - All indexed thumbnails w 3e80 ith ID3 and EXIF info. - Larges pages offering link URL and file size. - Most Popular Searches for text links offering: Link addr., total clicks, last . . .
. . .
Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Searches for media links offering: Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Links (click counter). - Search log offering: Query, Results, Queried at, Time taken, User IP, Country, Host name (Latest100) - Index log offering: . . .
. . .
Country, Host name (Latest100) - Index log offering: File-name, index date and delete option - sitemap log offering: sitemap.xml output sitemap list offering file/page suffixes - IDS log offering: IP, host, query, impact, involved tags, date and time of intrusion. - Flood attempts log offering: IP, query, date and time of flood attempt. - Auto . . .
. . .
attempts log offering: IP, query, date and time of flood attempt. - Auto Re-index log file - Server info offering: Server software, environment, MySQL, PDF-converter, image functions, php.ini file PHP integration, PHP security info. Each item holding lists of details. All text links, media links and thumbnails are active linked. As stated in . . .
. . .
individual Re-indexing, the periodical Re-indexer could be started and aborted in the "Options" menu of each site. 2.5 Preferred Re-indexing Each new URL added to the Admin backend, could be supplied with a priority level. This level will be used by the option 'Re-index only preferred sites'. Level 1 will be interpreted as most important, while . . .
. . .
less all index results will be stored in log files in sub folder /admin/log/ The names of the log files look like: db2100524-2147561html-21.47.56_1.html (log file of first thread) db2100524-2147561html-21.48.12_2.html (log file of second thread) and is build by the following items: db2 - Number of database. 100524 - Date (May 24, 2010) 21.47.56 - Time when this thread . . .
. . .
spider.php -new1 php spider.php -new2 etc. The IDs will be added to the name of the corresponding log files like: db2100524-2147561html-21.47.56_ID1.html (log file of first thread) db2100524-2147561html-21.48.12_ID2.html (log file of second thread) IDs could be defined by personal requirements, but the limitations for file names with respect to the OS should be taken . . .
. . .
but will not erase the content of all the other tables. So the check whether the content of a page has changed (MD5sum) is still available for a fast re-index procedure. Once prepared, multithreaded re-indexing could be invoked by starting several threads and adding individual IDs to the option parameter like: php spider.php -erased1 php . . .
. . .
4.4 Ignoring parts of a page Sphider-plus includes an option to exclude parts of pages from being indexed. Thi 775e s can for example be used to prevent search result flooding when certain keywords appear on certain part in most pages (like a header, footer or a menu). Any part of a page between <! sphider_noindex > and <! . . .
. . .
<! sphider_noindex > and <! /sphider_noindex > tags is not indexed, however links in it are followed. 4.5 Ignoring parts of a page by <div id='abc'> Ignoring parts of a page by the <! sphider_noindex > tags requires direct access to the page, because the tags need to be added (edited) to the page. A more flexible method, . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ menu[0-5] / 4.6 Indexing only parts of a page by <div id='abc'> If enabled in Admin settings, the values as defined in the list-file /include/common/divs_use.txt will be used to index only the content between <div . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ table[0-5] / 4.7 Ignore HTML elements defined by <tagname> . . </tagname> This option is foreseen to cooperate with the new HTML5 elements like section, nav, aside, hgroup, article, header, footer, etc. HTML elements . . .
. . .
contain a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */nav[0-5]/ Please keep in mind that element names placed in /include/common/elements_not.txt will be processed case-sensitive. 4.8 Index only HTML elements defined by <tagname> . . </tagname> This is the vice versa . . .
. . .
of all the duplicate content URLs: http://www.example.com/product.php?item=swedish-fish&category=gummy-candy http://www.example.com/product.php?item=swedish-fish&trackingid=1234&sessionid=5678 and Sphider-plus will understand that the duplicates all refer to the canonical URL: http://www.example.com/product.php?item=swedish-fish. The . . .
. . .
charset are supported by the ConvertCharset function and will be used to convert text into UTF-8 Unicode: WINDOWS windows-1250 - Central Europe windows-1251 - Cyrillic windows-1252 - Latin I windows-1253 - Greek windows-1254 - Turkish windows-1255 - Hebrew windows-1256 - Arabic windows-1257 - Baltic windows-1258 - Viet Nam cp874 - Thai - this . . .
. . .
- Baltic windows-1258 - Viet Nam cp874 - Thai - this file is also for DOS DOS cp437 - Latin US cp737 - Greek cp775 - BaltRim cp850 - Latin1 cp852 - Latin2 cp855 - Cyrylic cp857 - Turkish cp860 - Portuguese cp861 - Iceland cp862 - Hebrew cp863 - Canada cp864 - Arabic cp865 - Nordic cp866 - Cyrylic Russian (this is the one, used in IE . . .
. . .
IE Cyrillic (DOS) ) cp869 - Greek2 MAC (Apple) x-mac-cyrillic x-mac-greek x-mac-icelandic x-mac-ce x-mac-roman ISO iso-8859-1 iso-8859-2 iso-8859-3 iso-8859-4 iso-8859-5 iso-8859-6 iso-8859-7 iso-8859-8 iso-8859-9 iso-8859-10 iso-8859-11 iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 . . .
. . .
iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 cp500 cp856 cp875 cp1006 cp1026 koi8-r (Cyrillic) koi8-u (Cyrillic Ukrainian) nextstep us-ascii us-ascii-quotes DSP implementation for NeXT stdenc symbol zdingbat And specially for old Polish programs: mazovia This list is to be read . . .
. . .
Access denied; you need the RELOAD privilege. . . Top 10. Enable real-time output of logging data Up to version 1.5 of Sphider-plus, during index / re-index there was no printout available because: - Several servers, especially on Win32, buffer the output from the script until it terminates before transmitting the results to the browser. - . . .
. . .
is seen. - Some versions of Microsoft Internet Explorer only start to display the page after they have received 256 bytes of output. As progress was not presented during index / re- index procedure, waiting for results became a pain in the neck. Selectable in Admin setting together with the update interval (1 - 10 seconds), AJAX technology was . . .
. . .
characters in front of words Warning: This option should be used with special care and not be activated for non ISO-8859 charsets. Some special characters as part of the word ending might be erased by accidental. Top 13. Media search for images, audio streams and videos 13.1 Media indexing Index of media files is enabled by separated Admin . . .
. . .
and 'Found at' - Total clicks - Last clicked - Query input 'Indexed Image Thumbnails' presenting: - Thumbnail 150 x 100 pixel - Image details like title, filename size of original image, link- and thumb-id - Option to delete the thumbnail In order to open the media files all tables contain active links. Media results are also stored in . . .
. . .
Settings' menu of the admin backend: First option: For results of XML product feeds, present the text extract of 350 characters as part of product attributes. If this option is not activated, all products of the XML product feed will be presented in search results, and the hits of the query string will be highlighted in all involved products. . . .
. . .
content of the database tables (those with the same table prefix) will be destroyed by the restore procedure. 16.5 Copy & Move This section of the Database Management will present only those databases, which are correctly configured, assigned and do have a set of installed tables as described in chapter Definition and configuration. This . . .
. . .
results. Never the less to find these x results, the complete database had to been browsed. Starting with version 2.5, an additional clean option is offered as part of the Admin backend. Main advantage of this option is a significant reduction of the search time for any query, because the content of the db could be limited to offer only x . . .
. . .
<link>http://www.abc.de/images/warp.gif</link> <title>warp.gif</title> <x_size>635</x_size> <y_size>98</y_size> </media_result> <media_result> <num>2</num> <type>audio</type> <url>http://www.abc.de/index.php</url> . . .
. . .
the following content: {"query":"colour","ip":"::1","host_name":"guard_007-hoster","query_time":"2016-03-18 10:56:23 AM","consumed":0.016,"total_results":2,"num_of_results":2,"from":1,"to":2,"text_results":[{"num":1,"weight":"100.0","url":"http:\/\/www.english.le-piaggie.info\/Info_eng.pdf","title":" Info_eng","fulltxt":" . . . . . .
. . .
placed on the ground floor, today's big kitchen with the heating fireplace for open and close mode of . . . ","page_size":"5,661.6kb","domain_name":"www.english.le-piaggie.info"},{"num":2,"weight":"50.0","url":"http:\/\/www.english.le-piaggie.info\/html\/description.html","title":" Description","fulltxt":" . . . cable. Additional active . . .
. . .
D.O.C.G. as far as to the Pratomagno mountains 160 olive-trees Detached house, about 800 m to reach the . . . ","page_size":"26.4 kb","domain_name":"www.english.le-piaggie.info"}]} Top . . .


5a2 All required information. Introduction Release and Legal Info Installation Documentation Change Log [ Documentation Summary ] Preamble: The info presented here is valid only for the latest release of Sphider-plus. At present version 4.2024a published February 12, 2024 is the actual release. 1. Settings, customizing and statistics 2. Indexing . . .
. . .
2. Indexing 2.1 Various options 2.2 Allow other hosts in same domain 2.3 Word stemming 2.4 Periodical Re-indexing 2.5 Preferred indexing 2.6 Multithreaded indexing 2.7 Create thumbnails during index procedure 2.8 Prevent indexing of known malware and pishing pages 2.9 Follow and create sitemap files 2.10 Use private sitemap instead of global . . .
. . .
Sitemap file 3. Using the indexer from command line 3.1 All options 3.2 Multithreaded indexing 4. Keeping pages, words and files from being indexed 4.1 robots.txt 4.2 Must include / must not include string list 4.3 Ignoring links 4.4 Ignoring parts of a page by <! sphider_noindex > 4.5 Ignoring parts of a page by <div id='abc'> . . .
. . .
charset' 6. Search modes 6.1 Search with wildcards * 6.2 Strict search ! 6.3 Tolerant search 6.4 Link search 6.5 Media search 6.6 Search only in one domain 6.7 Search in categories 6.8 Greek language support 6.9 Block queries 7. Chronological order for result listing 7.1 Text result listing 7.2 Media result listing 8. PDF converter 9. Clean . . .
. . .
of logging data 11. Error messages and Debug mode 12. Delete secondary characters 13. Media search for images, audio streams and videos 13.1 Media indexing 13.2 Not supported media content 13.3 Search for media content 13.4 Statistics for media content 14. Feed support 14.1 XML product feeds 14.2 RDF, RSD, RSS and Atom feeds 15. Result . . .
. . .
Overview 16.2 Definition and configuration 16.3 Activate / disable database 16.4 Backup & Restore of databases 16.5 Copy & and Move 16.6 Enhancing functionality of multiple database support 17. Search in categories 17.1 Hierachical structure 17.2 Parallel structure 18. User suggested sites 19. Vulnerability protection 19.1 Prevent queries . . .
. . .
templates 22.2 Embed the search engine into existing HTML code 22.3 The different style sheet files 23. JSON, XML and RSS result output [ Documentation ] 1. Settings, customizing and statistics If you want to change settings, behavior and design of Sphider-plus, you can do so by means of the Admin interface. There is a wide range of . . .
. . .
interface. There is a wide range of settings foreseen for Sphider-plus. Separated into different submenus like: Sites: - Add Site - Index only the new - Re-index all - Re-index only preferred URLs - Erase Re-index (available also for individual URLs) - Import/export URL list - Approve sites - Banned domains Categories: - Add, edit, delete - . . .
. . .
URL list - Approve sites - Banned domains Categories: - Add, edit, delete - Create new subcategory under Index: - Basic indexing options - Advanced options Clean: - Clean keywords not associated with any link - Clean links not associated with any site - Clean Category table not associated with any site - Clean Media links - Clear Temp table . . .
. . .
- Clear Search log - Clear 'Most Popular Page Links' log - Clear 'Most Popular Media Links' log - Clear Spider log, separate and bulk delete - Clear Thumbnail images, separate and bulk delete - Clear Text cache - Clear Media cache - Clear IDS log file - Clear flood attempts log file - Clear all entries in addurl or banned table - Truncate all . . .
. . .
flood attempts log file - Clear all entries in addurl or banned table - Truncate all tables in database Settings: - General Settings - Index Log Settings - Spider Settings - Search Settings - Order of Result listing - Suggest Options - Page Indexing Weights Database: - Configure up to 5 databases with unlimited number of table sets - Activate . . .
. . .
Database: - Configure up to 5 databases with unlimited number of table sets - Activate separately for 'Admin', 'Search' user and 'Suggest URL user' - Backup / Restore - Copy / Move - Optimize Templates: In order to enable customer's integration of Sphider-plus into existing sites, HTML templates are prepared for Search form Text result . . .
. . .
Search form Text result listing Media result listing Most popular queries etc. Three different designs are offered, which may be selected in submenu 'Settings'. If the layout does not fit the design of your site (which is normal), you may create new designs and modify the appropriate file /templates/My_template/adminstyle.css . . .
. . .
the appropriate file /templates/My_template/adminstyle.css /templates/My_template/userstyle.css Statistics output: - Top keywords (Top 50 with hit counter). - All indexed thumbnails w 3e80 ith ID3 and EXIF info. - Larges pages offering link URL and file size. - Most Popular Searches for text links offering: Link addr., total clicks, last . . .
. . .
Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Searches for media links offering: Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Links (click counter). - Search log offering: Query, Results, Queried at, Time taken, User IP, Country, Host name (Latest100) - Index log offering: . . .
. . .
Country, Host name (Latest100) - Index log offering: File-name, index date and delete option - sitemap log offering: sitemap.xml output sitemap list offering file/page suffixes - IDS log offering: IP, host, query, impact, involved tags, date and time of intrusion. - Flood attempts log offering: IP, query, date and time of flood attempt. - Auto . . .
. . .
attempts log offering: IP, query, date and time of flood attempt. - Auto Re-index log file - Server info offering: Server software, environment, MySQL, PDF-converter, image functions, php.ini file PHP integration, PHP security info. Each item holding lists of details. All text links, media links and thumbnails are active linked. As stated in . . .
. . .
individual Re-indexing, the periodical Re-indexer could be started and aborted in the "Options" menu of each site. 2.5 Preferred Re-indexing Each new URL added to the Admin backend, could be supplied with a priority level. This level will be used by the option 'Re-index only preferred sites'. Level 1 will be interpreted as most important, while . . .
. . .
less all index results will be stored in log files in sub folder /admin/log/ The names of the log files look like: db2100524-2147561html (log file of first thread) db2_100524-21.48.12_2.html (log file of second thread) and is build by the following items: db2 - Number of database. 100524 - Date (May 24, 2010) 21.47.56 - Time when this thread . . .
. . .
spider.php -new1 php spider.php -new2 etc. The IDs will be added to the name of the corresponding log files like: db2_100524-21.47.56_ID1.html (log file of first thread) db2_100524-21.48.12_ID2.html (log file of second thread) IDs could be defined by personal requirements, but the limitations for file names with respect to the OS should be taken . . .
. . .
but will not erase the content of all the other tables. So the check whether the content of a page has changed (MD5sum) is still available for a fast re-index procedure. Once prepared, multithreaded re-indexing could be invoked by starting several threads and adding individual IDs to the option parameter like: php spider.php -erased1 php . . .
. . .
4.4 Ignoring parts of a page Sphider-plus includes an option to exclude parts of pages from being indexed. Thi 775e s can for example be used to prevent search result flooding when certain keywords appear on certain part in most pages (like a header, footer or a menu). Any part of a page between <! sphider_noindex > and <! . . .
. . .
<! sphider_noindex > and <! /sphider_noindex > tags is not indexed, however links in it are followed. 4.5 Ignoring parts of a page by <div id='abc'> Ignoring parts of a page by the <! sphider_noindex > tags requires direct access to the page, because the tags need to be added (edited) to the page. A more flexible method, . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ menu[0-5] / 4.6 Indexing only parts of a page by <div id='abc'> If enabled in Admin settings, the values as defined in the list-file /include/common/divs_use.txt will be used to index only the content between <div . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ table[0-5] / 4.7 Ignore HTML elements defined by <tagname> . . </tagname> This option is foreseen to cooperate with the new HTML5 elements like section, nav, aside, hgroup, article, header, footer, etc. HTML elements . . .
. . .
contain a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */nav[0-5]/ Please keep in mind that element names placed in /include/common/elements_not.txt will be processed case-sensitive. 4.8 Index only HTML elements defined by <tagname> . . </tagname> This is the vice versa . . .
. . .
of all the duplicate content URLs: http://www.example.com/product.php?item=swedish-fish&category=gummy-candy http://www.example.com/product.php?item=swedish-fish&trackingid=1234&sessionid=5678 and Sphider-plus will understand that the duplicates all refer to the canonical URL: http://www.example.com/product.php?item=swedish-fish. The . . .
. . .
charset are supported by the ConvertCharset function and will be used to convert text into UTF-8 Unicode: WINDOWS windows-1250 - Central Europe windows-1251 - Cyrillic windows-1252 - Latin I windows-1253 - Greek windows-1254 - Turkish windows-1255 - Hebrew windows-1256 - Arabic windows-1257 - Baltic windows-1258 - Viet Nam cp874 - Thai - this . . .
. . .
- Baltic windows-1258 - Viet Nam cp874 - Thai - this file is also for DOS DOS cp437 - Latin US cp737 - Greek cp775 - BaltRim cp850 - Latin1 cp852 - Latin2 cp855 - Cyrylic cp857 - Turkish cp860 - Portuguese cp861 - Iceland cp862 - Hebrew cp863 - Canada cp864 - Arabic cp865 - Nordic cp866 - Cyrylic Russian (this is the one, used in IE . . .
. . .
IE Cyrillic (DOS) ) cp869 - Greek2 MAC (Apple) x-mac-cyrillic x-mac-greek x-mac-icelandic x-mac-ce x-mac-roman ISO iso-8859-1 iso-8859-2 iso-8859-3 iso-8859-4 iso-8859-5 iso-8859-6 iso-8859-7 iso-8859-8 iso-8859-9 iso-8859-10 iso-8859-11 iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 . . .
. . .
iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 cp500 cp856 cp875 cp1006 cp1026 koi8-r (Cyrillic) koi8-u (Cyrillic Ukrainian) nextstep us-ascii us-ascii-quotes DSP implementation for NeXT stdenc symbol zdingbat And specially for old Polish programs: mazovia This list is to be read . . .
. . .
Access denied; you need the RELOAD privilege. . . Top 10. Enable real-time output of logging data Up to version 1.5 of Sphider-plus, during index / re-index there was no printout available because: - Several servers, especially on Win32, buffer the output from the script until it terminates before transmitting the results to the browser. - . . .
. . .
is seen. - Some versions of Microsoft Internet Explorer only start to display the page after they have received 256 bytes of output. As progress was not presented during index / re- index procedure, waiting for results became a pain in the neck. Selectable in Admin setting together with the update interval (1 - 10 seconds), AJAX technology was . . .
. . .
characters in front of words Warning: This option should be used with special care and not be activated for non ISO-8859 charsets. Some special characters as part of the word ending might be erased by accidental. Top 13. Media search for images, audio streams and videos 13.1 Media indexing Index of media files is enabled by separated Admin . . .
. . .
and 'Found at' - Total clicks - Last clicked - Query input 'Indexed Image Thumbnails' presenting: - Thumbnail 150 x 100 pixel - Image details like title, filename size of original image, link- and thumb-id - Option to delete the thumbnail In order to open the media files all tables contain active links. Media results are also stored in . . .
. . .
Settings' menu of the admin backend: First option: For results of XML product feeds, present the text extract of 350 characters as part of product attributes. If this option is not activated, all products of the XML product feed will be presented in search results, and the hits of the query string will be highlighted in all involved products. . . .
. . .
content of the database tables (those with the same table prefix) will be destroyed by the restore procedure. 16.5 Copy & Move This section of the Database Management will present only those databases, which are correctly configured, assigned and do have a set of installed tables as described in chapter Definition and configuration. This . . .
. . .
results. Never the less to find these x results, the complete database had to been browsed. Starting with version 2.5, an additional clean option is offered as part of the Admin backend. Main advantage of this option is a significant reduction of the search time for any query, because the content of the db could be limited to offer only x . . .
. . .
<link>http://www.abc.de/images/warp.gif</link> <title>warp.gif</title> <x_size>635</x_size> <y_size>98</y_size> </media_result> <media_result> <num>2</num> <type>audio</type> <url>http://www.abc.de/index.php</url> . . .
. . .
the following content: {"query":"colour","ip":"::1","host_name":"guard_007-hoster","query_time":"2016-03-18 10:56:23 AM","consumed":0.016,"total_results":2,"num_of_results":2,"from":1,"to":2,"text_results":[{"num":1,"weight":"100.0","url":"http:\/\/www.english.le-piaggie.info\/Info_eng.pdf","title":" Info_eng","fulltxt":" . . . . . .
. . .
placed on the ground floor, today's big kitchen with the heating fireplace for open and close mode of . . . ","page_size":"5,661.6kb","domain_name":"www.english.le-piaggie.info"},{"num":2,"weight":"50.0","url":"http:\/\/www.english.le-piaggie.info\/html\/description.html","title":" Description","fulltxt":" . . . cable. Additional active . . .
. . .
D.O.C.G. as far as to the Pratomagno mountains 160 olive-trees Detached house, about 800 m to reach the . . . ","page_size":"26.4 kb","domain_name":"www.english.le-piaggie.info"}]} Top . . .


5a2 All required information. Introduction Release and Legal Info Installation Documentation Change Log [ Documentation Summary ] Preamble: The info presented here is valid only for the latest release of Sphider-plus. At present version 4.2024a published February 12, 2024 is the actual release. 1. Settings, customizing and statistics 2. Indexing . . .
. . .
2. Indexing 2.1 Various options 2.2 Allow other hosts in same domain 2.3 Word stemming 2.4 Periodical Re-indexing 2.5 Preferred indexing 2.6 Multithreaded indexing 2.7 Create thumbnails during index procedure 2.8 Prevent indexing of known malware and pishing pages 2.9 Follow and create sitemap files 2.10 Use private sitemap instead of global . . .
. . .
Sitemap file 3. Using the indexer from command line 3.1 All options 3.2 Multithreaded indexing 4. Keeping pages, words and files from being indexed 4.1 robots.txt 4.2 Must include / must not include string list 4.3 Ignoring links 4.4 Ignoring parts of a page by <! sphider_noindex > 4.5 Ignoring parts of a page by <div id='abc'> . . .
. . .
charset' 6. Search modes 6.1 Search with wildcards * 6.2 Strict search ! 6.3 Tolerant search 6.4 Link search 6.5 Media search 6.6 Search only in one domain 6.7 Search in categories 6.8 Greek language support 6.9 Block queries 7. Chronological order for result listing 7.1 Text result listing 7.2 Media result listing 8. PDF converter 9. Clean . . .
. . .
of logging data 11. Error messages and Debug mode 12. Delete secondary characters 13. Media search for images, audio streams and videos 13.1 Media indexing 13.2 Not supported media content 13.3 Search for media content 13.4 Statistics for media content 14. Feed support 14.1 XML product feeds 14.2 RDF, RSD, RSS and Atom feeds 15. Result . . .
. . .
Overview 16.2 Definition and configuration 16.3 Activate / disable database 16.4 Backup & Restore of databases 16.5 Copy & and Move 16.6 Enhancing functionality of multiple database support 17. Search in categories 17.1 Hierachical structure 17.2 Parallel structure 18. User suggested sites 19. Vulnerability protection 19.1 Prevent queries . . .
. . .
templates 22.2 Embed the search engine into existing HTML code 22.3 The different style sheet files 23. JSON, XML and RSS result output [ Documentation ] 1. Settings, customizing and statistics If you want to change settings, behavior and design of Sphider-plus, you can do so by means of the Admin interface. There is a wide range of . . .
. . .
interface. There is a wide range of settings foreseen for Sphider-plus. Separated into different submenus like: Sites: - Add Site - Index only the new - Re-index all - Re-index only preferred URLs - Erase Re-index (available also for individual URLs) - Import/export URL list - Approve sites - Banned domains Categories: - Add, edit, delete - . . .
. . .
URL list - Approve sites - Banned domains Categories: - Add, edit, delete - Create new subcategory under Index: - Basic indexing options - Advanced options Clean: - Clean keywords not associated with any link - Clean links not associated with any site - Clean Category table not associated with any site - Clean Media links - Clear Temp table . . .
. . .
- Clear Search log - Clear 'Most Popular Page Links' log - Clear 'Most Popular Media Links' log - Clear Spider log, separate and bulk delete - Clear Thumbnail images, separate and bulk delete - Clear Text cache - Clear Media cache - Clear IDS log file - Clear flood attempts log file - Clear all entries in addurl or banned table - Truncate all . . .
. . .
flood attempts log file - Clear all entries in addurl or banned table - Truncate all tables in database Settings: - General Settings - Index Log Settings - Spider Settings - Search Settings - Order of Result listing - Suggest Options - Page Indexing Weights Database: - Configure up to 5 databases with unlimited number of table sets - Activate . . .
. . .
Database: - Configure up to 5 databases with unlimited number of table sets - Activate separately for 'Admin', 'Search' user and 'Suggest URL user' - Backup / Restore - Copy / Move - Optimize Templates: In order to enable customer's integration of Sphider-plus into existing sites, HTML templates are prepared for Search form Text result . . .
. . .
Search form Text result listing Media result listing Most popular queries etc. Three different designs are offered, which may be selected in submenu 'Settings'. If the layout does not fit the design of your site (which is normal), you may create new designs and modify the appropriate file /templates/My_template/adminstyle.css . . .
. . .
the appropriate file /templates/My_template/adminstyle.css /templates/My_template/userstyle.css Statistics output: - Top keywords (Top 50 with hit counter). - All indexed thumbnails w 3e80 ith ID3 and EXIF info. - Larges pages offering link URL and file size. - Most Popular Searches for text links offering: Link addr., total clicks, last . . .
. . .
Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Searches for media links offering: Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Links (click counter). - Search log offering: Query, Results, Queried at, Time taken, User IP, Country, Host name (Latest100) - Index log offering: . . .
. . .
Country, Host name (Latest100) - Index log offering: File-name, index date and delete option - sitemap log offering: sitemap.xml output sitemap list offering file/page suffixes - IDS log offering: IP, host, query, impact, involved tags, date and time of intrusion. - Flood attempts log offering: IP, query, date and time of flood attempt. - Auto . . .
. . .
attempts log offering: IP, query, date and time of flood attempt. - Auto Re-index log file - Server info offering: Server software, environment, MySQL, PDF-converter, image functions, php.ini file PHP integration, PHP security info. Each item holding lists of details. All text links, media links and thumbnails are active linked. As stated in . . .
. . .
individual Re-indexing, the periodical Re-indexer could be started and aborted in the "Options" menu of each site. 2.5 Preferred Re-indexing Each new URL added to the Admin backend, could be supplied with a priority level. This level will be used by the option 'Re-index only preferred sites'. Level 1 will be interpreted as most important, while . . .
. . .
less all index results will be stored in log files in sub folder /admin/log/ The names of the log files look like: db2_100524-21.47.56_1.html (log file of first thread) db2_100524-21.48.12_2.html (log file of second thread) and is build by the following items: db2 - Number of database. 100524 - Date (May 24, 2010) 21.47.56 - Time when this thread . . .
. . .
spider.php -new1 php spider.php -new2 etc. The IDs will be added to the name of the corresponding log files like: db2100524-214756ID1html (log file of first thread) db2_100524-21.48.12_ID2.html (log file of second thread) IDs could be defined by personal requirements, but the limitations for file names with respect to the OS should be taken . . .
. . .
but will not erase the content of all the other tables. So the check whether the content of a page has changed (MD5sum) is still available for a fast re-index procedure. Once prepared, multithreaded re-indexing could be invoked by starting several threads and adding individual IDs to the option parameter like: php spider.php -erased1 php . . .
. . .
4.4 Ignoring parts of a page Sphider-plus includes an option to exclude parts of pages from being indexed. Thi 775e s can for example be used to prevent search result flooding when certain keywords appear on certain part in most pages (like a header, footer or a menu). Any part of a page between <! sphider_noindex > and <! . . .
. . .
<! sphider_noindex > and <! /sphider_noindex > tags is not indexed, however links in it are followed. 4.5 Ignoring parts of a page by <div id='abc'> Ignoring parts of a page by the <! sphider_noindex > tags requires direct access to the page, because the tags need to be added (edited) to the page. A more flexible method, . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ menu[0-5] / 4.6 Indexing only parts of a page by <div id='abc'> If enabled in Admin settings, the values as defined in the list-file /include/common/divs_use.txt will be used to index only the content between <div . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ table[0-5] / 4.7 Ignore HTML elements defined by <tagname> . . </tagname> This option is foreseen to cooperate with the new HTML5 elements like section, nav, aside, hgroup, article, header, footer, etc. HTML elements . . .
. . .
contain a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */nav[0-5]/ Please keep in mind that element names placed in /include/common/elements_not.txt will be processed case-sensitive. 4.8 Index only HTML elements defined by <tagname> . . </tagname> This is the vice versa . . .
. . .
of all the duplicate content URLs: http://www.example.com/product.php?item=swedish-fish&category=gummy-candy http://www.example.com/product.php?item=swedish-fish&trackingid=1234&sessionid=5678 and Sphider-plus will understand that the duplicates all refer to the canonical URL: http://www.example.com/product.php?item=swedish-fish. The . . .
. . .
charset are supported by the ConvertCharset function and will be used to convert text into UTF-8 Unicode: WINDOWS windows-1250 - Central Europe windows-1251 - Cyrillic windows-1252 - Latin I windows-1253 - Greek windows-1254 - Turkish windows-1255 - Hebrew windows-1256 - Arabic windows-1257 - Baltic windows-1258 - Viet Nam cp874 - Thai - this . . .
. . .
- Baltic windows-1258 - Viet Nam cp874 - Thai - this file is also for DOS DOS cp437 - Latin US cp737 - Greek cp775 - BaltRim cp850 - Latin1 cp852 - Latin2 cp855 - Cyrylic cp857 - Turkish cp860 - Portuguese cp861 - Iceland cp862 - Hebrew cp863 - Canada cp864 - Arabic cp865 - Nordic cp866 - Cyrylic Russian (this is the one, used in IE . . .
. . .
IE Cyrillic (DOS) ) cp869 - Greek2 MAC (Apple) x-mac-cyrillic x-mac-greek x-mac-icelandic x-mac-ce x-mac-roman ISO iso-8859-1 iso-8859-2 iso-8859-3 iso-8859-4 iso-8859-5 iso-8859-6 iso-8859-7 iso-8859-8 iso-8859-9 iso-8859-10 iso-8859-11 iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 . . .
. . .
iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 cp500 cp856 cp875 cp1006 cp1026 koi8-r (Cyrillic) koi8-u (Cyrillic Ukrainian) nextstep us-ascii us-ascii-quotes DSP implementation for NeXT stdenc symbol zdingbat And specially for old Polish programs: mazovia This list is to be read . . .
. . .
Access denied; you need the RELOAD privilege. . . Top 10. Enable real-time output of logging data Up to version 1.5 of Sphider-plus, during index / re-index there was no printout available because: - Several servers, especially on Win32, buffer the output from the script until it terminates before transmitting the results to the browser. - . . .
. . .
is seen. - Some versions of Microsoft Internet Explorer only start to display the page after they have received 256 bytes of output. As progress was not presented during index / re- index procedure, waiting for results became a pain in the neck. Selectable in Admin setting together with the update interval (1 - 10 seconds), AJAX technology was . . .
. . .
characters in front of words Warning: This option should be used with special care and not be activated for non ISO-8859 charsets. Some special characters as part of the word ending might be erased by accidental. Top 13. Media search for images, audio streams and videos 13.1 Media indexing Index of media files is enabled by separated Admin . . .
. . .
and 'Found at' - Total clicks - Last clicked - Query input 'Indexed Image Thumbnails' presenting: - Thumbnail 150 x 100 pixel - Image details like title, filename size of original image, link- and thumb-id - Option to delete the thumbnail In order to open the media files all tables contain active links. Media results are also stored in . . .
. . .
Settings' menu of the admin backend: First option: For results of XML product feeds, present the text extract of 350 characters as part of product attributes. If this option is not activated, all products of the XML product feed will be presented in search results, and the hits of the query string will be highlighted in all involved products. . . .
. . .
content of the database tables (those with the same table prefix) will be destroyed by the restore procedure. 16.5 Copy & Move This section of the Database Management will present only those databases, which are correctly configured, assigned and do have a set of installed tables as described in chapter Definition and configuration. This . . .
. . .
results. Never the less to find these x results, the complete database had to been browsed. Starting with version 2.5, an additional clean option is offered as part of the Admin backend. Main advantage of this option is a significant reduction of the search time for any query, because the content of the db could be limited to offer only x . . .
. . .
<link>http://www.abc.de/images/warp.gif</link> <title>warp.gif</title> <x_size>635</x_size> <y_size>98</y_size> </media_result> <media_result> <num>2</num> <type>audio</type> <url>http://www.abc.de/index.php</url> . . .
. . .
the following content: {"query":"colour","ip":"::1","host_name":"guard_007-hoster","query_time":"2016-03-18 10:56:23 AM","consumed":0.016,"total_results":2,"num_of_results":2,"from":1,"to":2,"text_results":[{"num":1,"weight":"100.0","url":"http:\/\/www.english.le-piaggie.info\/Info_eng.pdf","title":" Info_eng","fulltxt":" . . . . . .
. . .
placed on the ground floor, today's big kitchen with the heating fireplace for open and close mode of . . . ","page_size":"5,661.6kb","domain_name":"www.english.le-piaggie.info"},{"num":2,"weight":"50.0","url":"http:\/\/www.english.le-piaggie.info\/html\/description.html","title":" Description","fulltxt":" . . . cable. Additional active . . .
. . .
D.O.C.G. as far as to the Pratomagno mountains 160 olive-trees Detached house, about 800 m to reach the . . . ","page_size":"26.4 kb","domain_name":"www.english.le-piaggie.info"}]} Top . . .


5a2 All required information. Introduction Release and Legal Info Installation Documentation Change Log [ Documentation Summary ] Preamble: The info presented here is valid only for the latest release of Sphider-plus. At present version 4.2024a published February 12, 2024 is the actual release. 1. Settings, customizing and statistics 2. Indexing . . .
. . .
2. Indexing 2.1 Various options 2.2 Allow other hosts in same domain 2.3 Word stemming 2.4 Periodical Re-indexing 2.5 Preferred indexing 2.6 Multithreaded indexing 2.7 Create thumbnails during index procedure 2.8 Prevent indexing of known malware and pishing pages 2.9 Follow and create sitemap files 2.10 Use private sitemap instead of global . . .
. . .
Sitemap file 3. Using the indexer from command line 3.1 All options 3.2 Multithreaded indexing 4. Keeping pages, words and files from being indexed 4.1 robots.txt 4.2 Must include / must not include string list 4.3 Ignoring links 4.4 Ignoring parts of a page by <! sphider_noindex > 4.5 Ignoring parts of a page by <div id='abc'> . . .
. . .
charset' 6. Search modes 6.1 Search with wildcards * 6.2 Strict search ! 6.3 Tolerant search 6.4 Link search 6.5 Media search 6.6 Search only in one domain 6.7 Search in categories 6.8 Greek language support 6.9 Block queries 7. Chronological order for result listing 7.1 Text result listing 7.2 Media result listing 8. PDF converter 9. Clean . . .
. . .
of logging data 11. Error messages and Debug mode 12. Delete secondary characters 13. Media search for images, audio streams and videos 13.1 Media indexing 13.2 Not supported media content 13.3 Search for media content 13.4 Statistics for media content 14. Feed support 14.1 XML product feeds 14.2 RDF, RSD, RSS and Atom feeds 15. Result . . .
. . .
Overview 16.2 Definition and configuration 16.3 Activate / disable database 16.4 Backup & Restore of databases 16.5 Copy & and Move 16.6 Enhancing functionality of multiple database support 17. Search in categories 17.1 Hierachical structure 17.2 Parallel structure 18. User suggested sites 19. Vulnerability protection 19.1 Prevent queries . . .
. . .
templates 22.2 Embed the search engine into existing HTML code 22.3 The different style sheet files 23. JSON, XML and RSS result output [ Documentation ] 1. Settings, customizing and statistics If you want to change settings, behavior and design of Sphider-plus, you can do so by means of the Admin interface. There is a wide range of . . .
. . .
interface. There is a wide range of settings foreseen for Sphider-plus. Separated into different submenus like: Sites: - Add Site - Index only the new - Re-index all - Re-index only preferred URLs - Erase Re-index (available also for individual URLs) - Import/export URL list - Approve sites - Banned domains Categories: - Add, edit, delete - . . .
. . .
URL list - Approve sites - Banned domains Categories: - Add, edit, delete - Create new subcategory under Index: - Basic indexing options - Advanced options Clean: - Clean keywords not associated with any link - Clean links not associated with any site - Clean Category table not associated with any site - Clean Media links - Clear Temp table . . .
. . .
- Clear Search log - Clear 'Most Popular Page Links' log - Clear 'Most Popular Media Links' log - Clear Spider log, separate and bulk delete - Clear Thumbnail images, separate and bulk delete - Clear Text cache - Clear Media cache - Clear IDS log file - Clear flood attempts log file - Clear all entries in addurl or banned table - Truncate all . . .
. . .
flood attempts log file - Clear all entries in addurl or banned table - Truncate all tables in database Settings: - General Settings - Index Log Settings - Spider Settings - Search Settings - Order of Result listing - Suggest Options - Page Indexing Weights Database: - Configure up to 5 databases with unlimited number of table sets - Activate . . .
. . .
Database: - Configure up to 5 databases with unlimited number of table sets - Activate separately for 'Admin', 'Search' user and 'Suggest URL user' - Backup / Restore - Copy / Move - Optimize Templates: In order to enable customer's integration of Sphider-plus into existing sites, HTML templates are prepared for Search form Text result . . .
. . .
Search form Text result listing Media result listing Most popular queries etc. Three different designs are offered, which may be selected in submenu 'Settings'. If the layout does not fit the design of your site (which is normal), you may create new designs and modify the appropriate file /templates/My_template/adminstyle.css . . .
. . .
the appropriate file /templates/My_template/adminstyle.css /templates/My_template/userstyle.css Statistics output: - Top keywords (Top 50 with hit counter). - All indexed thumbnails w 3e80 ith ID3 and EXIF info. - Larges pages offering link URL and file size. - Most Popular Searches for text links offering: Link addr., total clicks, last . . .
. . .
Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Searches for media links offering: Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Links (click counter). - Search log offering: Query, Results, Queried at, Time taken, User IP, Country, Host name (Latest100) - Index log offering: . . .
. . .
Country, Host name (Latest100) - Index log offering: File-name, index date and delete option - sitemap log offering: sitemap.xml output sitemap list offering file/page suffixes - IDS log offering: IP, host, query, impact, involved tags, date and time of intrusion. - Flood attempts log offering: IP, query, date and time of flood attempt. - Auto . . .
. . .
attempts log offering: IP, query, date and time of flood attempt. - Auto Re-index log file - Server info offering: Server software, environment, MySQL, PDF-converter, image functions, php.ini file PHP integration, PHP security info. Each item holding lists of details. All text links, media links and thumbnails are active linked. As stated in . . .
. . .
individual Re-indexing, the periodical Re-indexer could be started and aborted in the "Options" menu of each site. 2.5 Preferred Re-indexing Each new URL added to the Admin backend, could be supplied with a priority level. This level will be used by the option 'Re-index only preferred sites'. Level 1 will be interpreted as most important, while . . .
. . .
less all index results will be stored in log files in sub folder /admin/log/ The names of the log files look like: db2_100524-21.47.56_1.html (log file of first thread) db2100524-2148122html (log file of second thread) and is build by the following items: db2 - Number of database. 100524 - Date (May 24, 2010) 21.47.56 - Time when this thread . . .
. . .
spider.php -new1 php spider.php -new2 etc. The IDs will be added to the name of the corresponding log files like: db2_100524-21.47.56_ID1.html (log file of first thread) db2_100524-21.48.12_ID2.html (log file of second thread) IDs could be defined by personal requirements, but the limitations for file names with respect to the OS should be taken . . .
. . .
but will not erase the content of all the other tables. So the check whether the content of a page has changed (MD5sum) is still available for a fast re-index procedure. Once prepared, multithreaded re-indexing could be invoked by starting several threads and adding individual IDs to the option parameter like: php spider.php -erased1 php . . .
. . .
4.4 Ignoring parts of a page Sphider-plus includes an option to exclude parts of pages from being indexed. Thi 775e s can for example be used to prevent search result flooding when certain keywords appear on certain part in most pages (like a header, footer or a menu). Any part of a page between <! sphider_noindex > and <! . . .
. . .
<! sphider_noindex > and <! /sphider_noindex > tags is not indexed, however links in it are followed. 4.5 Ignoring parts of a page by <div id='abc'> Ignoring parts of a page by the <! sphider_noindex > tags requires direct access to the page, because the tags need to be added (edited) to the page. A more flexible method, . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ menu[0-5] / 4.6 Indexing only parts of a page by <div id='abc'> If enabled in Admin settings, the values as defined in the list-file /include/common/divs_use.txt will be used to index only the content between <div . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ table[0-5] / 4.7 Ignore HTML elements defined by <tagname> . . </tagname> This option is foreseen to cooperate with the new HTML5 elements like section, nav, aside, hgroup, article, header, footer, etc. HTML elements . . .
. . .
contain a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */nav[0-5]/ Please keep in mind that element names placed in /include/common/elements_not.txt will be processed case-sensitive. 4.8 Index only HTML elements defined by <tagname> . . </tagname> This is the vice versa . . .
. . .
of all the duplicate content URLs: http://www.example.com/product.php?item=swedish-fish&category=gummy-candy http://www.example.com/product.php?item=swedish-fish&trackingid=1234&sessionid=5678 and Sphider-plus will understand that the duplicates all refer to the canonical URL: http://www.example.com/product.php?item=swedish-fish. The . . .
. . .
charset are supported by the ConvertCharset function and will be used to convert text into UTF-8 Unicode: WINDOWS windows-1250 - Central Europe windows-1251 - Cyrillic windows-1252 - Latin I windows-1253 - Greek windows-1254 - Turkish windows-1255 - Hebrew windows-1256 - Arabic windows-1257 - Baltic windows-1258 - Viet Nam cp874 - Thai - this . . .
. . .
- Baltic windows-1258 - Viet Nam cp874 - Thai - this file is also for DOS DOS cp437 - Latin US cp737 - Greek cp775 - BaltRim cp850 - Latin1 cp852 - Latin2 cp855 - Cyrylic cp857 - Turkish cp860 - Portuguese cp861 - Iceland cp862 - Hebrew cp863 - Canada cp864 - Arabic cp865 - Nordic cp866 - Cyrylic Russian (this is the one, used in IE . . .
. . .
IE Cyrillic (DOS) ) cp869 - Greek2 MAC (Apple) x-mac-cyrillic x-mac-greek x-mac-icelandic x-mac-ce x-mac-roman ISO iso-8859-1 iso-8859-2 iso-8859-3 iso-8859-4 iso-8859-5 iso-8859-6 iso-8859-7 iso-8859-8 iso-8859-9 iso-8859-10 iso-8859-11 iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 . . .
. . .
iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 cp500 cp856 cp875 cp1006 cp1026 koi8-r (Cyrillic) koi8-u (Cyrillic Ukrainian) nextstep us-ascii us-ascii-quotes DSP implementation for NeXT stdenc symbol zdingbat And specially for old Polish programs: mazovia This list is to be read . . .
. . .
Access denied; you need the RELOAD privilege. . . Top 10. Enable real-time output of logging data Up to version 1.5 of Sphider-plus, during index / re-index there was no printout available because: - Several servers, especially on Win32, buffer the output from the script until it terminates before transmitting the results to the browser. - . . .
. . .
is seen. - Some versions of Microsoft Internet Explorer only start to display the page after they have received 256 bytes of output. As progress was not presented during index / re- index procedure, waiting for results became a pain in the neck. Selectable in Admin setting together with the update interval (1 - 10 seconds), AJAX technology was . . .
. . .
characters in front of words Warning: This option should be used with special care and not be activated for non ISO-8859 charsets. Some special characters as part of the word ending might be erased by accidental. Top 13. Media search for images, audio streams and videos 13.1 Media indexing Index of media files is enabled by separated Admin . . .
. . .
and 'Found at' - Total clicks - Last clicked - Query input 'Indexed Image Thumbnails' presenting: - Thumbnail 150 x 100 pixel - Image details like title, filename size of original image, link- and thumb-id - Option to delete the thumbnail In order to open the media files all tables contain active links. Media results are also stored in . . .
. . .
Settings' menu of the admin backend: First option: For results of XML product feeds, present the text extract of 350 characters as part of product attributes. If this option is not activated, all products of the XML product feed will be presented in search results, and the hits of the query string will be highlighted in all involved products. . . .
. . .
content of the database tables (those with the same table prefix) will be destroyed by the restore procedure. 16.5 Copy & Move This section of the Database Management will present only those databases, which are correctly configured, assigned and do have a set of installed tables as described in chapter Definition and configuration. This . . .
. . .
results. Never the less to find these x results, the complete database had to been browsed. Starting with version 2.5, an additional clean option is offered as part of the Admin backend. Main advantage of this option is a significant reduction of the search time for any query, because the content of the db could be limited to offer only x . . .
. . .
<link>http://www.abc.de/images/warp.gif</link> <title>warp.gif</title> <x_size>635</x_size> <y_size>98</y_size> </media_result> <media_result> <num>2</num> <type>audio</type> <url>http://www.abc.de/index.php</url> . . .
. . .
the following content: {"query":"colour","ip":"::1","host_name":"guard_007-hoster","query_time":"2016-03-18 10:56:23 AM","consumed":0.016,"total_results":2,"num_of_results":2,"from":1,"to":2,"text_results":[{"num":1,"weight":"100.0","url":"http:\/\/www.english.le-piaggie.info\/Info_eng.pdf","title":" Info_eng","fulltxt":" . . . . . .
. . .
placed on the ground floor, today's big kitchen with the heating fireplace for open and close mode of . . . ","page_size":"5,661.6kb","domain_name":"www.english.le-piaggie.info"},{"num":2,"weight":"50.0","url":"http:\/\/www.english.le-piaggie.info\/html\/description.html","title":" Description","fulltxt":" . . . cable. Additional active . . .
. . .
D.O.C.G. as far as to the Pratomagno mountains 160 olive-trees Detached house, about 800 m to reach the . . . ","page_size":"26.4 kb","domain_name":"www.english.le-piaggie.info"}]} Top . . .


5a2 All required information. Introduction Release and Legal Info Installation Documentation Change Log [ Documentation Summary ] Preamble: The info presented here is valid only for the latest release of Sphider-plus. At present version 4.2024a published February 12, 2024 is the actual release. 1. Settings, customizing and statistics 2. Indexing . . .
. . .
2. Indexing 2.1 Various options 2.2 Allow other hosts in same domain 2.3 Word stemming 2.4 Periodical Re-indexing 2.5 Preferred indexing 2.6 Multithreaded indexing 2.7 Create thumbnails during index procedure 2.8 Prevent indexing of known malware and pishing pages 2.9 Follow and create sitemap files 2.10 Use private sitemap instead of global . . .
. . .
Sitemap file 3. Using the indexer from command line 3.1 All options 3.2 Multithreaded indexing 4. Keeping pages, words and files from being indexed 4.1 robots.txt 4.2 Must include / must not include string list 4.3 Ignoring links 4.4 Ignoring parts of a page by <! sphider_noindex > 4.5 Ignoring parts of a page by <div id='abc'> . . .
. . .
charset' 6. Search modes 6.1 Search with wildcards * 6.2 Strict search ! 6.3 Tolerant search 6.4 Link search 6.5 Media search 6.6 Search only in one domain 6.7 Search in categories 6.8 Greek language support 6.9 Block queries 7. Chronological order for result listing 7.1 Text result listing 7.2 Media result listing 8. PDF converter 9. Clean . . .
. . .
of logging data 11. Error messages and Debug mode 12. Delete secondary characters 13. Media search for images, audio streams and videos 13.1 Media indexing 13.2 Not supported media content 13.3 Search for media content 13.4 Statistics for media content 14. Feed support 14.1 XML product feeds 14.2 RDF, RSD, RSS and Atom feeds 15. Result . . .
. . .
Overview 16.2 Definition and configuration 16.3 Activate / disable database 16.4 Backup & Restore of databases 16.5 Copy & and Move 16.6 Enhancing functionality of multiple database support 17. Search in categories 17.1 Hierachical structure 17.2 Parallel structure 18. User suggested sites 19. Vulnerability protection 19.1 Prevent queries . . .
. . .
templates 22.2 Embed the search engine into existing HTML code 22.3 The different style sheet files 23. JSON, XML and RSS result output [ Documentation ] 1. Settings, customizing and statistics If you want to change settings, behavior and design of Sphider-plus, you can do so by means of the Admin interface. There is a wide range of . . .
. . .
interface. There is a wide range of settings foreseen for Sphider-plus. Separated into different submenus like: Sites: - Add Site - Index only the new - Re-index all - Re-index only preferred URLs - Erase Re-index (available also for individual URLs) - Import/export URL list - Approve sites - Banned domains Categories: - Add, edit, delete - . . .
. . .
URL list - Approve sites - Banned domains Categories: - Add, edit, delete - Create new subcategory under Index: - Basic indexing options - Advanced options Clean: - Clean keywords not associated with any link - Clean links not associated with any site - Clean Category table not associated with any site - Clean Media links - Clear Temp table . . .
. . .
- Clear Search log - Clear 'Most Popular Page Links' log - Clear 'Most Popular Media Links' log - Clear Spider log, separate and bulk delete - Clear Thumbnail images, separate and bulk delete - Clear Text cache - Clear Media cache - Clear IDS log file - Clear flood attempts log file - Clear all entries in addurl or banned table - Truncate all . . .
. . .
flood attempts log file - Clear all entries in addurl or banned table - Truncate all tables in database Settings: - General Settings - Index Log Settings - Spider Settings - Search Settings - Order of Result listing - Suggest Options - Page Indexing Weights Database: - Configure up to 5 databases with unlimited number of table sets - Activate . . .
. . .
Database: - Configure up to 5 databases with unlimited number of table sets - Activate separately for 'Admin', 'Search' user and 'Suggest URL user' - Backup / Restore - Copy / Move - Optimize Templates: In order to enable customer's integration of Sphider-plus into existing sites, HTML templates are prepared for Search form Text result . . .
. . .
Search form Text result listing Media result listing Most popular queries etc. Three different designs are offered, which may be selected in submenu 'Settings'. If the layout does not fit the design of your site (which is normal), you may create new designs and modify the appropriate file /templates/My_template/adminstyle.css . . .
. . .
the appropriate file /templates/My_template/adminstyle.css /templates/My_template/userstyle.css Statistics output: - Top keywords (Top 50 with hit counter). - All indexed thumbnails w 3e80 ith ID3 and EXIF info. - Larges pages offering link URL and file size. - Most Popular Searches for text links offering: Link addr., total clicks, last . . .
. . .
Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Searches for media links offering: Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Links (click counter). - Search log offering: Query, Results, Queried at, Time taken, User IP, Country, Host name (Latest100) - Index log offering: . . .
. . .
Country, Host name (Latest100) - Index log offering: File-name, index date and delete option - sitemap log offering: sitemap.xml output sitemap list offering file/page suffixes - IDS log offering: IP, host, query, impact, involved tags, date and time of intrusion. - Flood attempts log offering: IP, query, date and time of flood attempt. - Auto . . .
. . .
attempts log offering: IP, query, date and time of flood attempt. - Auto Re-index log file - Server info offering: Server software, environment, MySQL, PDF-converter, image functions, php.ini file PHP integration, PHP security info. Each item holding lists of details. All text links, media links and thumbnails are active linked. As stated in . . .
. . .
individual Re-indexing, the periodical Re-indexer could be started and aborted in the "Options" menu of each site. 2.5 Preferred Re-indexing Each new URL added to the Admin backend, could be supplied with a priority level. This level will be used by the option 'Re-index only preferred sites'. Level 1 will be interpreted as most important, while . . .
. . .
less all index results will be stored in log files in sub folder /admin/log/ The names of the log files look like: db2_100524-21.47.56_1.html (log file of first thread) db2_100524-21.48.12_2.html (log file of second thread) and is build by the following items: db2 - Number of database. 100524 - Date (May 24, 2010) 21.47.56 - Time when this thread . . .
. . .
spider.php -new1 php spider.php -new2 etc. The IDs will be added to the name of the corresponding log files like: db2_100524-21.47.56_ID1.html (log file of first thread) db2100524-214812ID2html (log file of second thread) IDs could be defined by personal requirements, but the limitations for file names with respect to the OS should be taken . . .
. . .
but will not erase the content of all the other tables. So the check whether the content of a page has changed (MD5sum) is still available for a fast re-index procedure. Once prepared, multithreaded re-indexing could be invoked by starting several threads and adding individual IDs to the option parameter like: php spider.php -erased1 php . . .
. . .
4.4 Ignoring parts of a page Sphider-plus includes an option to exclude parts of pages from being indexed. Thi 775e s can for example be used to prevent search result flooding when certain keywords appear on certain part in most pages (like a header, footer or a menu). Any part of a page between <! sphider_noindex > and <! . . .
. . .
<! sphider_noindex > and <! /sphider_noindex > tags is not indexed, however links in it are followed. 4.5 Ignoring parts of a page by <div id='abc'> Ignoring parts of a page by the <! sphider_noindex > tags requires direct access to the page, because the tags need to be added (edited) to the page. A more flexible method, . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ menu[0-5] / 4.6 Indexing only parts of a page by <div id='abc'> If enabled in Admin settings, the values as defined in the list-file /include/common/divs_use.txt will be used to index only the content between <div . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ table[0-5] / 4.7 Ignore HTML elements defined by <tagname> . . </tagname> This option is foreseen to cooperate with the new HTML5 elements like section, nav, aside, hgroup, article, header, footer, etc. HTML elements . . .
. . .
contain a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */nav[0-5]/ Please keep in mind that element names placed in /include/common/elements_not.txt will be processed case-sensitive. 4.8 Index only HTML elements defined by <tagname> . . </tagname> This is the vice versa . . .
. . .
of all the duplicate content URLs: http://www.example.com/product.php?item=swedish-fish&category=gummy-candy http://www.example.com/product.php?item=swedish-fish&trackingid=1234&sessionid=5678 and Sphider-plus will understand that the duplicates all refer to the canonical URL: http://www.example.com/product.php?item=swedish-fish. The . . .
. . .
charset are supported by the ConvertCharset function and will be used to convert text into UTF-8 Unicode: WINDOWS windows-1250 - Central Europe windows-1251 - Cyrillic windows-1252 - Latin I windows-1253 - Greek windows-1254 - Turkish windows-1255 - Hebrew windows-1256 - Arabic windows-1257 - Baltic windows-1258 - Viet Nam cp874 - Thai - this . . .
. . .
- Baltic windows-1258 - Viet Nam cp874 - Thai - this file is also for DOS DOS cp437 - Latin US cp737 - Greek cp775 - BaltRim cp850 - Latin1 cp852 - Latin2 cp855 - Cyrylic cp857 - Turkish cp860 - Portuguese cp861 - Iceland cp862 - Hebrew cp863 - Canada cp864 - Arabic cp865 - Nordic cp866 - Cyrylic Russian (this is the one, used in IE . . .
. . .
IE Cyrillic (DOS) ) cp869 - Greek2 MAC (Apple) x-mac-cyrillic x-mac-greek x-mac-icelandic x-mac-ce x-mac-roman ISO iso-8859-1 iso-8859-2 iso-8859-3 iso-8859-4 iso-8859-5 iso-8859-6 iso-8859-7 iso-8859-8 iso-8859-9 iso-8859-10 iso-8859-11 iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 . . .
. . .
iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 cp500 cp856 cp875 cp1006 cp1026 koi8-r (Cyrillic) koi8-u (Cyrillic Ukrainian) nextstep us-ascii us-ascii-quotes DSP implementation for NeXT stdenc symbol zdingbat And specially for old Polish programs: mazovia This list is to be read . . .
. . .
Access denied; you need the RELOAD privilege. . . Top 10. Enable real-time output of logging data Up to version 1.5 of Sphider-plus, during index / re-index there was no printout available because: - Several servers, especially on Win32, buffer the output from the script until it terminates before transmitting the results to the browser. - . . .
. . .
is seen. - Some versions of Microsoft Internet Explorer only start to display the page after they have received 256 bytes of output. As progress was not presented during index / re- index procedure, waiting for results became a pain in the neck. Selectable in Admin setting together with the update interval (1 - 10 seconds), AJAX technology was . . .
. . .
characters in front of words Warning: This option should be used with special care and not be activated for non ISO-8859 charsets. Some special characters as part of the word ending might be erased by accidental. Top 13. Media search for images, audio streams and videos 13.1 Media indexing Index of media files is enabled by separated Admin . . .
. . .
and 'Found at' - Total clicks - Last clicked - Query input 'Indexed Image Thumbnails' presenting: - Thumbnail 150 x 100 pixel - Image details like title, filename size of original image, link- and thumb-id - Option to delete the thumbnail In order to open the media files all tables contain active links. Media results are also stored in . . .
. . .
Settings' menu of the admin backend: First option: For results of XML product feeds, present the text extract of 350 characters as part of product attributes. If this option is not activated, all products of the XML product feed will be presented in search results, and the hits of the query string will be highlighted in all involved products. . . .
. . .
content of the database tables (those with the same table prefix) will be destroyed by the restore procedure. 16.5 Copy & Move This section of the Database Management will present only those databases, which are correctly configured, assigned and do have a set of installed tables as described in chapter Definition and configuration. This . . .
. . .
results. Never the less to find these x results, the complete database had to been browsed. Starting with version 2.5, an additional clean option is offered as part of the Admin backend. Main advantage of this option is a significant reduction of the search time for any query, because the content of the db could be limited to offer only x . . .
. . .
<link>http://www.abc.de/images/warp.gif</link> <title>warp.gif</title> <x_size>635</x_size> <y_size>98</y_size> </media_result> <media_result> <num>2</num> <type>audio</type> <url>http://www.abc.de/index.php</url> . . .
. . .
the following content: {"query":"colour","ip":"::1","host_name":"guard_007-hoster","query_time":"2016-03-18 10:56:23 AM","consumed":0.016,"total_results":2,"num_of_results":2,"from":1,"to":2,"text_results":[{"num":1,"weight":"100.0","url":"http:\/\/www.english.le-piaggie.info\/Info_eng.pdf","title":" Info_eng","fulltxt":" . . . . . .
. . .
placed on the ground floor, today's big kitchen with the heating fireplace for open and close mode of . . . ","page_size":"5,661.6kb","domain_name":"www.english.le-piaggie.info"},{"num":2,"weight":"50.0","url":"http:\/\/www.english.le-piaggie.info\/html\/description.html","title":" Description","fulltxt":" . . . cable. Additional active . . .
. . .
D.O.C.G. as far as to the Pratomagno mountains 160 olive-trees Detached house, about 800 m to reach the . . . ","page_size":"26.4 kb","domain_name":"www.english.le-piaggie.info"}]} Top . . .


5a2 All required information. Introduction Release and Legal Info Installation Documentation Change Log [ Documentation Summary ] Preamble: The info presented here is valid only for the latest release of Sphider-plus. At present version 4.2024a published February 12, 2024 is the actual release. 1. Settings, customizing and statistics 2. Indexing . . .
. . .
2. Indexing 2.1 Various options 2.2 Allow other hosts in same domain 2.3 Word stemming 2.4 Periodical Re-indexing 2.5 Preferred indexing 2.6 Multithreaded indexing 2.7 Create thumbnails during index procedure 2.8 Prevent indexing of known malware and pishing pages 2.9 Follow and create sitemap files 2.10 Use private sitemap instead of global . . .
. . .
Sitemap file 3. Using the indexer from command line 3.1 All options 3.2 Multithreaded indexing 4. Keeping pages, words and files from being indexed 4.1 robots.txt 4.2 Must include / must not include string list 4.3 Ignoring links 4.4 Ignoring parts of a page by <! sphider_noindex > 4.5 Ignoring parts of a page by <div id='abc'> . . .
. . .
charset' 6. Search modes 6.1 Search with wildcards * 6.2 Strict search ! 6.3 Tolerant search 6.4 Link search 6.5 Media search 6.6 Search only in one domain 6.7 Search in categories 6.8 Greek language support 6.9 Block queries 7. Chronological order for result listing 7.1 Text result listing 7.2 Media result listing 8. PDF converter 9. Clean . . .
. . .
of logging data 11. Error messages and Debug mode 12. Delete secondary characters 13. Media search for images, audio streams and videos 13.1 Media indexing 13.2 Not supported media content 13.3 Search for media content 13.4 Statistics for media content 14. Feed support 14.1 XML product feeds 14.2 RDF, RSD, RSS and Atom feeds 15. Result . . .
. . .
Overview 16.2 Definition and configuration 16.3 Activate / disable database 16.4 Backup & Restore of databases 16.5 Copy & and Move 16.6 Enhancing functionality of multiple database support 17. Search in categories 17.1 Hierachical structure 17.2 Parallel structure 18. User suggested sites 19. Vulnerability protection 19.1 Prevent queries . . .
. . .
templates 22.2 Embed the search engine into existing HTML code 22.3 The different style sheet files 23. JSON, XML and RSS result output [ Documentation ] 1. Settings, customizing and statistics If you want to change settings, behavior and design of Sphider-plus, you can do so by means of the Admin interface. There is a wide range of . . .
. . .
interface. There is a wide range of settings foreseen for Sphider-plus. Separated into different submenus like: Sites: - Add Site - Index only the new - Re-index all - Re-index only preferred URLs - Erase Re-index (available also for individual URLs) - Import/export URL list - Approve sites - Banned domains Categories: - Add, edit, delete - . . .
. . .
URL list - Approve sites - Banned domains Categories: - Add, edit, delete - Create new subcategory under Index: - Basic indexing options - Advanced options Clean: - Clean keywords not associated with any link - Clean links not associated with any site - Clean Category table not associated with any site - Clean Media links - Clear Temp table . . .
. . .
- Clear Search log - Clear 'Most Popular Page Links' log - Clear 'Most Popular Media Links' log - Clear Spider log, separate and bulk delete - Clear Thumbnail images, separate and bulk delete - Clear Text cache - Clear Media cache - Clear IDS log file - Clear flood attempts log file - Clear all entries in addurl or banned table - Truncate all . . .
. . .
flood attempts log file - Clear all entries in addurl or banned table - Truncate all tables in database Settings: - General Settings - Index Log Settings - Spider Settings - Search Settings - Order of Result listing - Suggest Options - Page Indexing Weights Database: - Configure up to 5 databases with unlimited number of table sets - Activate . . .
. . .
Database: - Configure up to 5 databases with unlimited number of table sets - Activate separately for 'Admin', 'Search' user and 'Suggest URL user' - Backup / Restore - Copy / Move - Optimize Templates: In order to enable customer's integration of Sphider-plus into existing sites, HTML templates are prepared for Search form Text result . . .
. . .
Search form Text result listing Media result listing Most popular queries etc. Three different designs are offered, which may be selected in submenu 'Settings'. If the layout does not fit the design of your site (which is normal), you may create new designs and modify the appropriate file /templates/My_template/adminstyle.css . . .
. . .
the appropriate file /templates/My_template/adminstyle.css /templates/My_template/userstyle.css Statistics output: - Top keywords (Top 50 with hit counter). - All indexed thumbnails w 3e80 ith ID3 and EXIF info. - Larges pages offering link URL and file size. - Most Popular Searches for text links offering: Link addr., total clicks, last . . .
. . .
Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Searches for media links offering: Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Links (click counter). - Search log offering: Query, Results, Queried at, Time taken, User IP, Country, Host name (Latest100) - Index log offering: . . .
. . .
Country, Host name (Latest100) - Index log offering: File-name, index date and delete option - sitemap log offering: sitemap.xml output sitemap list offering file/page suffixes - IDS log offering: IP, host, query, impact, involved tags, date and time of intrusion. - Flood attempts log offering: IP, query, date and time of flood attempt. - Auto . . .
. . .
attempts log offering: IP, query, date and time of flood attempt. - Auto Re-index log file - Server info offering: Server software, environment, MySQL, PDF-converter, image functions, php.ini file PHP integration, PHP security info. Each item holding lists of details. All text links, media links and thumbnails are active linked. As stated in . . .
. . .
individual Re-indexing, the periodical Re-indexer could be started and aborted in the "Options" menu of each site. 2.5 Preferred Re-indexing Each new URL added to the Admin backend, could be supplied with a priority level. This level will be used by the option 'Re-index only preferred sites'. Level 1 will be interpreted as most important, while . . .
. . .
less all index results will be stored in log files in sub folder /admin/log/ The names of the log files look like: db2_100524-21.47.56_1.html (log file of first thread) db2_100524-21.48.12_2.html (log file of second thread) and is build by the following items: db2 - Number of database. 100524 - Date (May 24, 2010) 21.47.56 - Time when this thread . . .
. . .
spider.php -new1 php spider.php -new2 etc. The IDs will be added to the name of the corresponding log files like: db2_100524-21.47.56_ID1.html (log file of first thread) db2_100524-21.48.12_ID2.html (log file of second thread) IDs could be defined by personal requirements, but the limitations for file names with respect to the OS should be taken . . .
. . .
but will not erase the content of all the other tables. So the check whether the content of a page has changed (MD5sum) is still available for a fast re-index procedure. Once prepared, multithreaded re-indexing could be invoked by starting several threads and adding individual IDs to the option parameter like: php spider.php -erased1 php . . .
. . .
4.4 Ignoring parts of a page Sphider-plus includes an option to exclude parts of pages from being indexed. Thi 775e s can for example be used to prevent search result flooding when certain keywords appear on certain part in most pages (like a header, footer or a menu). Any part of a page between <! sphider_noindex > and <! . . .
. . .
<! sphider_noindex > and <! /sphider_noindex > tags is not indexed, however links in it are followed. 4.5 Ignoring parts of a page by <div id='abc'> Ignoring parts of a page by the <! sphider_noindex > tags requires direct access to the page, because the tags need to be added (edited) to the page. A more flexible method, . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ menu[0-5] / 4.6 Indexing only parts of a page by <div id='abc'> If enabled in Admin settings, the values as defined in the list-file /include/common/divs_use.txt will be used to index only the content between <div . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ table[0-5] / 4.7 Ignore HTML elements defined by <tagname> . . </tagname> This option is foreseen to cooperate with the new HTML5 elements like section, nav, aside, hgroup, article, header, footer, etc. HTML elements . . .
. . .
contain a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */nav[0-5]/ Please keep in mind that element names placed in /include/common/elements_not.txt will be processed case-sensitive. 4.8 Index only HTML elements defined by <tagname> . . </tagname> This is the vice versa . . .
. . .
of all the duplicate content URLs: http://www.example.com/product.php?item=swedish-fish&category=gummy-candy http://www.example.com/product.php?item=swedish-fish&trackingid=1234&sessionid=5678 and Sphider-plus will understand that the duplicates all refer to the canonical URL: http://www.example.com/product.php?item=swedish-fish. The . . .
. . .
charset are supported by the ConvertCharset function and will be used to convert text into UTF-8 Unicode: WINDOWS windows-1250 - Central Europe windows-1251 - Cyrillic windows-1252 - Latin I windows-1253 - Greek windows-1254 - Turkish windows-1255 - Hebrew windows-1256 - Arabic windows-1257 - Baltic windows-1258 - Viet Nam cp874 - Thai - this . . .
. . .
- Baltic windows-1258 - Viet Nam cp874 - Thai - this file is also for DOS DOS cp437 - Latin US cp737 - Greek cp775 - BaltRim cp850 - Latin1 cp852 - Latin2 cp855 - Cyrylic cp857 - Turkish cp860 - Portuguese cp861 - Iceland cp862 - Hebrew cp863 - Canada cp864 - Arabic cp865 - Nordic cp866 - Cyrylic Russian (this is the one, used in IE . . .
. . .
IE Cyrillic (DOS) ) cp869 - Greek2 MAC (Apple) x-mac-cyrillic x-mac-greek x-mac-icelandic x-mac-ce x-mac-roman ISO iso-8859-1 iso-8859-2 iso-8859-3 iso-8859-4 iso-8859-5 iso-8859-6 iso-8859-7 iso-8859-8 iso-8859-9 iso-8859-10 iso-8859-11 iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 . . .
. . .
iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 cp500 cp856 cp875 cp1006 cp1026 koi8-r (Cyrillic) koi8-u (Cyrillic Ukrainian) nextstep us-ascii us-ascii-quotes DSP implementation for NeXT stdenc symbol zdingbat And specially for old Polish programs: mazovia This list is to be read . . .
. . .
Access denied; you need the RELOAD privilege. . . Top 10. Enable real-time output of logging data Up to version 1.5 of Sphider-plus, during index / re-index there was no printout available because: - Several servers, especially on Win32, buffer the output from the script until it terminates before transmitting the results to the browser. - . . .
. . .
is seen. - Some versions of Microsoft Internet Explorer only start to display the page after they have received 256 bytes of output. As progress was not presented during index / re- index procedure, waiting for results became a pain in the neck. Selectable in Admin setting together with the update interval (1 - 10 seconds), AJAX technology was . . .
. . .
characters in front of words Warning: This option should be used with special care and not be activated for non ISO-8859 charsets. Some special characters as part of the word ending might be erased by accidental. Top 13. Media search for images, audio streams and videos 13.1 Media indexing Index of media files is enabled by separated Admin . . .
. . .
and 'Found at' - Total clicks - Last clicked - Query input 'Indexed Image Thumbnails' presenting: - Thumbnail 150 x 100 pixel - Image details like title, filename size of original image, link- and thumb-id - Option to delete the thumbnail In order to open the media files all tables contain active links. Media results are also stored in . . .
. . .
Settings' menu of the admin backend: First option: For results of XML product feeds, present the text extract of 350 characters as part of product attributes. If this option is not activated, all products of the XML product feed will be presented in search results, and the hits of the query string will be highlighted in all involved products. . . .
. . .
content of the database tables (those with the same table prefix) will be destroyed by the restore procedure. 16.5 Copy & Move This section of the Database Management will present only those databases, which are correctly configured, assigned and do have a set of installed tables as described in chapter Definition and configuration. This . . .
. . .
results. Never the less to find these x results, the complete database had to been browsed. Starting with version 2.5, an additional clean option is offered as part of the Admin backend. Main advantage of this option is a significant reduction of the search time for any query, because the content of the db could be limited to offer only x . . .
. . .
<link>http://www.abc.de/images/warp.gif</link> <title>warp.gif</title> <x_size>635</x_size> <y_size>98</y_size> </media_result> <media_result> <num>2</num> <type>audio</type> <url>http://www.abc.de/index.php</url> . . .
. . .
the following content: {"query":"colour","ip":"::1","host_name":"guard_007-hoster","query_time":"2016-03-18 10:56:23 AM","consumed":0.016,"total_results":2,"num_of_results":2,"from":1,"to":2,"text_results":[{"num":1,"weight":"100.0","url":"http:\/\/www.english.le-piaggie.info\/Info_eng.pdf","title":" Info_eng","fulltxt":" . . . . . .
. . .
placed on the ground floor, today's big kitchen with the heating fireplace for open and close mode of . . . ","page_size":"5,661.6kb","domainnamewwwenglishle-piaggieinfo}{num2weight500urlhttp\/\/wwwenglishle-piaggieinfo\/html\/descriptionhtmltitle":"www.english.le-piaggie.info"},{"num":2,"weight":"50.0","url":"http:\/\/www.english.le-piaggie.info\/html\/description.html","title":" Description","fulltxt":" . . . cable. Additional active . . .
. . .
D.O.C.G. as far as to the Pratomagno mountains 160 olive-trees Detached house, about 800 m to reach the . . . ","page_size":"26.4 kb","domainnamewwwenglishle-piaggieinfo}{num2weight500urlhttp\/\/wwwenglishle-piaggieinfo\/html\/descriptionhtmltitle":"www.english.le-piaggie.info"}]} Top . . .


5a2 All required information. Introduction Release and Legal Info Installation Documentation Change Log [ Documentation Summary ] Preamble: The info presented here is valid only for the latest release of Sphider-plus. At present version 4.2024a published February 12, 2024 is the actual release. 1. Settings, customizing and statistics 2. Indexing . . .
. . .
2. Indexing 2.1 Various options 2.2 Allow other hosts in same domain 2.3 Word stemming 2.4 Periodical Re-indexing 2.5 Preferred indexing 2.6 Multithreaded indexing 2.7 Create thumbnails during index procedure 2.8 Prevent indexing of known malware and pishing pages 2.9 Follow and create sitemap files 2.10 Use private sitemap instead of global . . .
. . .
Sitemap file 3. Using the indexer from command line 3.1 All options 3.2 Multithreaded indexing 4. Keeping pages, words and files from being indexed 4.1 robots.txt 4.2 Must include / must not include string list 4.3 Ignoring links 4.4 Ignoring parts of a page by <! sphider_noindex > 4.5 Ignoring parts of a page by <div id='abc'> . . .
. . .
charset' 6. Search modes 6.1 Search with wildcards * 6.2 Strict search ! 6.3 Tolerant search 6.4 Link search 6.5 Media search 6.6 Search only in one domain 6.7 Search in categories 6.8 Greek language support 6.9 Block queries 7. Chronological order for result listing 7.1 Text result listing 7.2 Media result listing 8. PDF converter 9. Clean . . .
. . .
of logging data 11. Error messages and Debug mode 12. Delete secondary characters 13. Media search for images, audio streams and videos 13.1 Media indexing 13.2 Not supported media content 13.3 Search for media content 13.4 Statistics for media content 14. Feed support 14.1 XML product feeds 14.2 RDF, RSD, RSS and Atom feeds 15. Result . . .
. . .
Overview 16.2 Definition and configuration 16.3 Activate / disable database 16.4 Backup & Restore of databases 16.5 Copy & and Move 16.6 Enhancing functionality of multiple database support 17. Search in categories 17.1 Hierachical structure 17.2 Parallel structure 18. User suggested sites 19. Vulnerability protection 19.1 Prevent queries . . .
. . .
templates 22.2 Embed the search engine into existing HTML code 22.3 The different style sheet files 23. JSON, XML and RSS result output [ Documentation ] 1. Settings, customizing and statistics If you want to change settings, behavior and design of Sphider-plus, you can do so by means of the Admin interface. There is a wide range of . . .
. . .
interface. There is a wide range of settings foreseen for Sphider-plus. Separated into different submenus like: Sites: - Add Site - Index only the new - Re-index all - Re-index only preferred URLs - Erase Re-index (available also for individual URLs) - Import/export URL list - Approve sites - Banned domains Categories: - Add, edit, delete - . . .
. . .
URL list - Approve sites - Banned domains Categories: - Add, edit, delete - Create new subcategory under Index: - Basic indexing options - Advanced options Clean: - Clean keywords not associated with any link - Clean links not associated with any site - Clean Category table not associated with any site - Clean Media links - Clear Temp table . . .
. . .
- Clear Search log - Clear 'Most Popular Page Links' log - Clear 'Most Popular Media Links' log - Clear Spider log, separate and bulk delete - Clear Thumbnail images, separate and bulk delete - Clear Text cache - Clear Media cache - Clear IDS log file - Clear flood attempts log file - Clear all entries in addurl or banned table - Truncate all . . .
. . .
flood attempts log file - Clear all entries in addurl or banned table - Truncate all tables in database Settings: - General Settings - Index Log Settings - Spider Settings - Search Settings - Order of Result listing - Suggest Options - Page Indexing Weights Database: - Configure up to 5 databases with unlimited number of table sets - Activate . . .
. . .
Database: - Configure up to 5 databases with unlimited number of table sets - Activate separately for 'Admin', 'Search' user and 'Suggest URL user' - Backup / Restore - Copy / Move - Optimize Templates: In order to enable customer's integration of Sphider-plus into existing sites, HTML templates are prepared for Search form Text result . . .
. . .
Search form Text result listing Media result listing Most popular queries etc. Three different designs are offered, which may be selected in submenu 'Settings'. If the layout does not fit the design of your site (which is normal), you may create new designs and modify the appropriate file /templates/My_template/adminstyle.css . . .
. . .
the appropriate file /templates/My_template/adminstyle.css /templates/My_template/userstyle.css Statistics output: - Top keywords (Top 50 with hit counter). - All indexed thumbnails w 3e80 ith ID3 and EXIF info. - Larges pages offering link URL and file size. - Most Popular Searches for text links offering: Link addr., total clicks, last . . .
. . .
Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Searches for media links offering: Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Links (click counter). - Search log offering: Query, Results, Queried at, Time taken, User IP, Country, Host name (Latest100) - Index log offering: . . .
. . .
Country, Host name (Latest100) - Index log offering: File-name, index date and delete option - sitemap log offering: sitemap.xml output sitemap list offering file/page suffixes - IDS log offering: IP, host, query, impact, involved tags, date and time of intrusion. - Flood attempts log offering: IP, query, date and time of flood attempt. - Auto . . .
. . .
attempts log offering: IP, query, date and time of flood attempt. - Auto Re-index log file - Server info offering: Server software, environment, MySQL, PDF-converter, image functions, php.ini file PHP integration, PHP security info. Each item holding lists of details. All text links, media links and thumbnails are active linked. As stated in . . .
. . .
individual Re-indexing, the periodical Re-indexer could be started and aborted in the "Options" menu of each site. 2.5 Preferred Re-indexing Each new URL added to the Admin backend, could be supplied with a priority level. This level will be used by the option 'Re-index only preferred sites'. Level 1 will be interpreted as most important, while . . .
. . .
less all index results will be stored in log files in sub folder /admin/log/ The names of the log files look like: db2_100524-21.47.56_1.html (log file of first thread) db2_100524-21.48.12_2.html (log file of second thread) and is build by the following items: db2 - Number of database. 100524 - Date (May 24, 2010) 21.47.56 - Time when this thread . . .
. . .
spider.php -new1 php spider.php -new2 etc. The IDs will be added to the name of the corresponding log files like: db2_100524-21.47.56_ID1.html (log file of first thread) db2_100524-21.48.12_ID2.html (log file of second thread) IDs could be defined by personal requirements, but the limitations for file names with respect to the OS should be taken . . .
. . .
but will not erase the content of all the other tables. So the check whether the content of a page has changed (MD5sum) is still available for a fast re-index procedure. Once prepared, multithreaded re-indexing could be invoked by starting several threads and adding individual IDs to the option parameter like: php spider.php -erased1 php . . .
. . .
4.4 Ignoring parts of a page Sphider-plus includes an option to exclude parts of pages from being indexed. Thi 775e s can for example be used to prevent search result flooding when certain keywords appear on certain part in most pages (like a header, footer or a menu). Any part of a page between <! sphider_noindex > and <! . . .
. . .
<! sphider_noindex > and <! /sphider_noindex > tags is not indexed, however links in it are followed. 4.5 Ignoring parts of a page by <div id='abc'> Ignoring parts of a page by the <! sphider_noindex > tags requires direct access to the page, because the tags need to be added (edited) to the page. A more flexible method, . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ menu[0-5] / 4.6 Indexing only parts of a page by <div id='abc'> If enabled in Admin settings, the values as defined in the list-file /include/common/divs_use.txt will be used to index only the content between <div . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ table[0-5] / 4.7 Ignore HTML elements defined by <tagname> . . </tagname> This option is foreseen to cooperate with the new HTML5 elements like section, nav, aside, hgroup, article, header, footer, etc. HTML elements . . .
. . .
contain a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */nav[0-5]/ Please keep in mind that element names placed in /include/common/elements_not.txt will be processed case-sensitive. 4.8 Index only HTML elements defined by <tagname> . . </tagname> This is the vice versa . . .
. . .
of all the duplicate content URLs: http://www.example.com/product.php?item=swedish-fish&category=gummy-candy http://www.example.com/product.php?item=swedish-fish&trackingid=1234&sessionid=5678 and Sphider-plus will understand that the duplicates all refer to the canonical URL: http://www.example.com/product.php?item=swedish-fish. The . . .
. . .
charset are supported by the ConvertCharset function and will be used to convert text into UTF-8 Unicode: WINDOWS windows-1250 - Central Europe windows-1251 - Cyrillic windows-1252 - Latin I windows-1253 - Greek windows-1254 - Turkish windows-1255 - Hebrew windows-1256 - Arabic windows-1257 - Baltic windows-1258 - Viet Nam cp874 - Thai - this . . .
. . .
- Baltic windows-1258 - Viet Nam cp874 - Thai - this file is also for DOS DOS cp437 - Latin US cp737 - Greek cp775 - BaltRim cp850 - Latin1 cp852 - Latin2 cp855 - Cyrylic cp857 - Turkish cp860 - Portuguese cp861 - Iceland cp862 - Hebrew cp863 - Canada cp864 - Arabic cp865 - Nordic cp866 - Cyrylic Russian (this is the one, used in IE . . .
. . .
IE Cyrillic (DOS) ) cp869 - Greek2 MAC (Apple) x-mac-cyrillic x-mac-greek x-mac-icelandic x-mac-ce x-mac-roman ISO iso-8859-1 iso-8859-2 iso-8859-3 iso-8859-4 iso-8859-5 iso-8859-6 iso-8859-7 iso-8859-8 iso-8859-9 iso-8859-10 iso-8859-11 iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 . . .
. . .
iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 cp500 cp856 cp875 cp1006 cp1026 koi8-r (Cyrillic) koi8-u (Cyrillic Ukrainian) nextstep us-ascii us-ascii-quotes DSP implementation for NeXT stdenc symbol zdingbat And specially for old Polish programs: mazovia This list is to be read . . .
. . .
Access denied; you need the RELOAD privilege. . . Top 10. Enable real-time output of logging data Up to version 1.5 of Sphider-plus, during index / re-index there was no printout available because: - Several servers, especially on Win32, buffer the output from the script until it terminates before transmitting the results to the browser. - . . .
. . .
is seen. - Some versions of Microsoft Internet Explorer only start to display the page after they have received 256 bytes of output. As progress was not presented during index / re- index procedure, waiting for results became a pain in the neck. Selectable in Admin setting together with the update interval (1 - 10 seconds), AJAX technology was . . .
. . .
characters in front of words Warning: This option should be used with special care and not be activated for non ISO-8859 charsets. Some special characters as part of the word ending might be erased by accidental. Top 13. Media search for images, audio streams and videos 13.1 Media indexing Index of media files is enabled by separated Admin . . .
. . .
and 'Found at' - Total clicks - Last clicked - Query input 'Indexed Image Thumbnails' presenting: - Thumbnail 150 x 100 pixel - Image details like title, filename size of original image, link- and thumb-id - Option to delete the thumbnail In order to open the media files all tables contain active links. Media results are also stored in . . .
. . .
Settings' menu of the admin backend: First option: For results of XML product feeds, present the text extract of 350 characters as part of product attributes. If this option is not activated, all products of the XML product feed will be presented in search results, and the hits of the query string will be highlighted in all involved products. . . .
. . .
content of the database tables (those with the same table prefix) will be destroyed by the restore procedure. 16.5 Copy & Move This section of the Database Management will present only those databases, which are correctly configured, assigned and do have a set of installed tables as described in chapter Definition and configuration. This . . .
. . .
results. Never the less to find these x results, the complete database had to been browsed. Starting with version 2.5, an additional clean option is offered as part of the Admin backend. Main advantage of this option is a significant reduction of the search time for any query, because the content of the db could be limited to offer only x . . .
. . .
<link>http://www.abc.de/images/warp.gif</link> <title>warp.gif</title> <x_size>635</x_size> <y_size>98</y_size> </media_result> <media_result> <num>2</num> <type>audio</type> <url>http://www.abc.de/index.php</url> . . .
. . .
the following content: {"query":"colour","ip":"::1","host_name":"guard_007-hoster","query_time":"2016-03-18 10:56:23 AM","consumed":0.016,"total_results":2,"num_of_results":2,"from":1,"to":2,"text_results":[{"num":1,"weight":"100.0","url":"http:\/\/www.english.le-piaggie.info\/Info_eng.pdf","title":" Info_eng","fulltxt":" . . . . . .
. . .
placed on the ground floor, today's big kitchen with the heating fireplace for open and close mode of . . . ","page_size":"5,661.6kb","domain_name":"www.english.le-piaggie.info"},{"num":2,"weight":"50.0","url":"http:\/\/www.english.le-piaggie.info\/html\/description.html","title":" Description","fulltxt":" . . . cable. Additional active . . .
. . .
D.O.C.G. as far as to the Pratomagno mountains 160 olive-trees Detached house, about 800 m to reach the . . . ","page_size":"26.4 kb","domain_name":"www.english.le-piaggie.info"}]} Top . . .


5a2 All required information. Introduction Release and Legal Info Installation Documentation Change Log [ Documentation Summary ] Preamble: The info presented here is valid only for the latest release of Sphider-plus. At present version 4.2024a published February 12, 2024 is the actual release. 1. Settings, customizing and statistics 2. Indexing . . .
. . .
2. Indexing 2.1 Various options 2.2 Allow other hosts in same domain 2.3 Word stemming 2.4 Periodical Re-indexing 2.5 Preferred indexing 2.6 Multithreaded indexing 2.7 Create thumbnails during index procedure 2.8 Prevent indexing of known malware and pishing pages 2.9 Follow and create sitemap files 2.10 Use private sitemap instead of global . . .
. . .
Sitemap file 3. Using the indexer from command line 3.1 All options 3.2 Multithreaded indexing 4. Keeping pages, words and files from being indexed 4.1 robots.txt 4.2 Must include / must not include string list 4.3 Ignoring links 4.4 Ignoring parts of a page by <! sphider_noindex > 4.5 Ignoring parts of a page by <div id='abc'> . . .
. . .
charset' 6. Search modes 6.1 Search with wildcards * 6.2 Strict search ! 6.3 Tolerant search 6.4 Link search 6.5 Media search 6.6 Search only in one domain 6.7 Search in categories 6.8 Greek language support 6.9 Block queries 7. Chronological order for result listing 7.1 Text result listing 7.2 Media result listing 8. PDF converter 9. Clean . . .
. . .
of logging data 11. Error messages and Debug mode 12. Delete secondary characters 13. Media search for images, audio streams and videos 13.1 Media indexing 13.2 Not supported media content 13.3 Search for media content 13.4 Statistics for media content 14. Feed support 14.1 XML product feeds 14.2 RDF, RSD, RSS and Atom feeds 15. Result . . .
. . .
Overview 16.2 Definition and configuration 16.3 Activate / disable database 16.4 Backup & Restore of databases 16.5 Copy & and Move 16.6 Enhancing functionality of multiple database support 17. Search in categories 17.1 Hierachical structure 17.2 Parallel structure 18. User suggested sites 19. Vulnerability protection 19.1 Prevent queries . . .
. . .
templates 22.2 Embed the search engine into existing HTML code 22.3 The different style sheet files 23. JSON, XML and RSS result output [ Documentation ] 1. Settings, customizing and statistics If you want to change settings, behavior and design of Sphider-plus, you can do so by means of the Admin interface. There is a wide range of . . .
. . .
interface. There is a wide range of settings foreseen for Sphider-plus. Separated into different submenus like: Sites: - Add Site - Index only the new - Re-index all - Re-index only preferred URLs - Erase Re-index (available also for individual URLs) - Import/export URL list - Approve sites - Banned domains Categories: - Add, edit, delete - . . .
. . .
URL list - Approve sites - Banned domains Categories: - Add, edit, delete - Create new subcategory under Index: - Basic indexing options - Advanced options Clean: - Clean keywords not associated with any link - Clean links not associated with any site - Clean Category table not associated with any site - Clean Media links - Clear Temp table . . .
. . .
- Clear Search log - Clear 'Most Popular Page Links' log - Clear 'Most Popular Media Links' log - Clear Spider log, separate and bulk delete - Clear Thumbnail images, separate and bulk delete - Clear Text cache - Clear Media cache - Clear IDS log file - Clear flood attempts log file - Clear all entries in addurl or banned table - Truncate all . . .
. . .
flood attempts log file - Clear all entries in addurl or banned table - Truncate all tables in database Settings: - General Settings - Index Log Settings - Spider Settings - Search Settings - Order of Result listing - Suggest Options - Page Indexing Weights Database: - Configure up to 5 databases with unlimited number of table sets - Activate . . .
. . .
Database: - Configure up to 5 databases with unlimited number of table sets - Activate separately for 'Admin', 'Search' user and 'Suggest URL user' - Backup / Restore - Copy / Move - Optimize Templates: In order to enable customer's integration of Sphider-plus into existing sites, HTML templates are prepared for Search form Text result . . .
. . .
Search form Text result listing Media result listing Most popular queries etc. Three different designs are offered, which may be selected in submenu 'Settings'. If the layout does not fit the design of your site (which is normal), you may create new designs and modify the appropriate file /templates/My_template/adminstyle.css . . .
. . .
the appropriate file /templates/My_template/adminstyle.css /templates/My_template/userstyle.css Statistics output: - Top keywords (Top 50 with hit counter). - All indexed thumbnails w 3e80 ith ID3 and EXIF info. - Larges pages offering link URL and file size. - Most Popular Searches for text links offering: Link addr., total clicks, last . . .
. . .
Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Searches for media links offering: Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Links (click counter). - Search log offering: Query, Results, Queried at, Time taken, User IP, Country, Host name (Latest100) - Index log offering: . . .
. . .
Country, Host name (Latest100) - Index log offering: File-name, index date and delete option - sitemap log offering: sitemap.xml output sitemap list offering file/page suffixes - IDS log offering: IP, host, query, impact, involved tags, date and time of intrusion. - Flood attempts log offering: IP, query, date and time of flood attempt. - Auto . . .
. . .
attempts log offering: IP, query, date and time of flood attempt. - Auto Re-index log file - Server info offering: Server software, environment, MySQL, PDF-converter, image functions, php.ini file PHP integration, PHP security info. Each item holding lists of details. All text links, media links and thumbnails are active linked. As stated in . . .
. . .
individual Re-indexing, the periodical Re-indexer could be started and aborted in the "Options" menu of each site. 2.5 Preferred Re-indexing Each new URL added to the Admin backend, could be supplied with a priority level. This level will be used by the option 'Re-index only preferred sites'. Level 1 will be interpreted as most important, while . . .
. . .
less all index results will be stored in log files in sub folder /admin/log/ The names of the log files look like: db2_100524-21.47.56_1.html (log file of first thread) db2_100524-21.48.12_2.html (log file of second thread) and is build by the following items: db2 - Number of database. 100524 - Date (May 24, 2010) 21.47.56 - Time when this thread . . .
. . .
spider.php -new1 php spider.php -new2 etc. The IDs will be added to the name of the corresponding log files like: db2_100524-21.47.56_ID1.html (log file of first thread) db2_100524-21.48.12_ID2.html (log file of second thread) IDs could be defined by personal requirements, but the limitations for file names with respect to the OS should be taken . . .
. . .
but will not erase the content of all the other tables. So the check whether the content of a page has changed (MD5sum) is still available for a fast re-index procedure. Once prepared, multithreaded re-indexing could be invoked by starting several threads and adding individual IDs to the option parameter like: php spider.php -erased1 php . . .
. . .
4.4 Ignoring parts of a page Sphider-plus includes an option to exclude parts of pages from being indexed. Thi 775e s can for example be used to prevent search result flooding when certain keywords appear on certain part in most pages (like a header, footer or a menu). Any part of a page between <! sphider_noindex > and <! . . .
. . .
<! sphider_noindex > and <! /sphider_noindex > tags is not indexed, however links in it are followed. 4.5 Ignoring parts of a page by <div id='abc'> Ignoring parts of a page by the <! sphider_noindex > tags requires direct access to the page, because the tags need to be added (edited) to the page. A more flexible method, . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ menu[0-5] / 4.6 Indexing only parts of a page by <div id='abc'> If enabled in Admin settings, the values as defined in the list-file /include/common/divs_use.txt will be used to index only the content between <div . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ table[0-5] / 4.7 Ignore HTML elements defined by <tagname> . . </tagname> This option is foreseen to cooperate with the new HTML5 elements like section, nav, aside, hgroup, article, header, footer, etc. HTML elements . . .
. . .
contain a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */nav[0-5]/ Please keep in mind that element names placed in /include/common/elements_not.txt will be processed case-sensitive. 4.8 Index only HTML elements defined by <tagname> . . </tagname> This is the vice versa . . .
. . .
of all the duplicate content URLs: http://www.example.com/product.php?item=swedish-fish&category=gummy-candy http://www.example.com/product.php?item=swedish-fish&trackingid=1234&sessionid=5678 and Sphider-plus will understand that the duplicates all refer to the canonical URL: http://www.example.com/product.php?item=swedish-fish. The . . .
. . .
charset are supported by the ConvertCharset function and will be used to convert text into UTF-8 Unicode: WINDOWS windows-1250 - Central Europe windows-1251 - Cyrillic windows-1252 - Latin I windows-1253 - Greek windows-1254 - Turkish windows-1255 - Hebrew windows-1256 - Arabic windows-1257 - Baltic windows-1258 - Viet Nam cp874 - Thai - this . . .
. . .
- Baltic windows-1258 - Viet Nam cp874 - Thai - this file is also for DOS DOS cp437 - Latin US cp737 - Greek cp775 - BaltRim cp850 - Latin1 cp852 - Latin2 cp855 - Cyrylic cp857 - Turkish cp860 - Portuguese cp861 - Iceland cp862 - Hebrew cp863 - Canada cp864 - Arabic cp865 - Nordic cp866 - Cyrylic Russian (this is the one, used in IE . . .
. . .
IE Cyrillic (DOS) ) cp869 - Greek2 MAC (Apple) x-mac-cyrillic x-mac-greek x-mac-icelandic x-mac-ce x-mac-roman ISO iso-8859-1 iso-8859-2 iso-8859-3 iso-8859-4 iso-8859-5 iso-8859-6 iso-8859-7 iso-8859-8 iso-8859-9 iso-8859-10 iso-8859-11 iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 . . .
. . .
iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 cp500 cp856 cp875 cp1006 cp1026 koi8-r (Cyrillic) koi8-u (Cyrillic Ukrainian) nextstep us-ascii us-ascii-quotes DSP implementation for NeXT stdenc symbol zdingbat And specially for old Polish programs: mazovia This list is to be read . . .
. . .
Access denied; you need the RELOAD privilege. . . Top 10. Enable real-time output of logging data Up to version 1.5 of Sphider-plus, during index / re-index there was no printout available because: - Several servers, especially on Win32, buffer the output from the script until it terminates before transmitting the results to the browser. - . . .
. . .
is seen. - Some versions of Microsoft Internet Explorer only start to display the page after they have received 256 bytes of output. As progress was not presented during index / re- index procedure, waiting for results became a pain in the neck. Selectable in Admin setting together with the update interval (1 - 10 seconds), AJAX technology was . . .
. . .
characters in front of words Warning: This option should be used with special care and not be activated for non ISO-8859 charsets. Some special characters as part of the word ending might be erased by accidental. Top 13. Media search for images, audio streams and videos 13.1 Media indexing Index of media files is enabled by separated Admin . . .
. . .
and 'Found at' - Total clicks - Last clicked - Query input 'Indexed Image Thumbnails' presenting: - Thumbnail 150 x 100 pixel - Image details like title, filename size of original image, link- and thumb-id - Option to delete the thumbnail In order to open the media files all tables contain active links. Media results are also stored in . . .
. . .
Settings' menu of the admin backend: First option: For results of XML product feeds, present the text extract of 350 characters as part of product attributes. If this option is not activated, all products of the XML product feed will be presented in search results, and the hits of the query string will be highlighted in all involved products. . . .
. . .
content of the database tables (those with the same table prefix) will be destroyed by the restore procedure. 16.5 Copy & Move This section of the Database Management will present only those databases, which are correctly configured, assigned and do have a set of installed tables as described in chapter Definition and configuration. This . . .
. . .
results. Never the less to find these x results, the complete database had to been browsed. Starting with version 2.5, an additional clean option is offered as part of the Admin backend. Main advantage of this option is a significant reduction of the search time for any query, because the content of the db could be limited to offer only x . . .
. . .
<link>http://www.abc.de/images/warp.gif</link> <title>warp.gif</title> <x_size>635</x_size> <y_size>98</y_size> </media_result> <media_result> <num>2</num> <type>audio</type> <url>http://www.abc.de/index.php</url> . . .
. . .
the following content: {"query":"colour","ip":"::1","host_name":"guard_007-hoster","query_time":"2016-03-18 10:56:23 AM","consumed":0.016,"total_results":2,"num_of_results":2,"from":1,"to":2,"text_results":[{"num":1,"weight":"100.0","url":"http:\/\/www.english.le-piaggie.info\/Info_eng.pdf","title":" Info_eng","fulltxt":" . . . . . .
. . .
placed on the ground floor, today's big kitchen with the heating fireplace for open and close mode of . . . ","page_size":"5,661.6kb","domain_name":"www.english.le-piaggie.info"},{"num":2,"weight":"50.0","url":"http:\/\/www.english.le-piaggie.info\/html\/description.html","title":" Description","fulltxt":" . . . cable. Additional active . . .
. . .
D.O.C.G. as far as to the Pratomagno mountains 160 olive-trees Detached house, about 800 m to reach the . . . ","page_size":"26.4 kb","domain_name":"www.english.le-piaggie.info"}]} Top . . .


5a2 All required information. Introduction Release and Legal Info Installation Documentation Change Log [ Documentation Summary ] Preamble: The info presented here is valid only for the latest release of Sphider-plus. At present version 4.2024a published February 12, 2024 is the actual release. 1. Settings, customizing and statistics 2. Indexing . . .
. . .
2. Indexing 2.1 Various options 2.2 Allow other hosts in same domain 2.3 Word stemming 2.4 Periodical Re-indexing 2.5 Preferred indexing 2.6 Multithreaded indexing 2.7 Create thumbnails during index procedure 2.8 Prevent indexing of known malware and pishing pages 2.9 Follow and create sitemap files 2.10 Use private sitemap instead of global . . .
. . .
Sitemap file 3. Using the indexer from command line 3.1 All options 3.2 Multithreaded indexing 4. Keeping pages, words and files from being indexed 4.1 robots.txt 4.2 Must include / must not include string list 4.3 Ignoring links 4.4 Ignoring parts of a page by <! sphider_noindex > 4.5 Ignoring parts of a page by <div id='abc'> . . .
. . .
charset' 6. Search modes 6.1 Search with wildcards * 6.2 Strict search ! 6.3 Tolerant search 6.4 Link search 6.5 Media search 6.6 Search only in one domain 6.7 Search in categories 6.8 Greek language support 6.9 Block queries 7. Chronological order for result listing 7.1 Text result listing 7.2 Media result listing 8. PDF converter 9. Clean . . .
. . .
of logging data 11. Error messages and Debug mode 12. Delete secondary characters 13. Media search for images, audio streams and videos 13.1 Media indexing 13.2 Not supported media content 13.3 Search for media content 13.4 Statistics for media content 14. Feed support 14.1 XML product feeds 14.2 RDF, RSD, RSS and Atom feeds 15. Result . . .
. . .
Overview 16.2 Definition and configuration 16.3 Activate / disable database 16.4 Backup & Restore of databases 16.5 Copy & and Move 16.6 Enhancing functionality of multiple database support 17. Search in categories 17.1 Hierachical structure 17.2 Parallel structure 18. User suggested sites 19. Vulnerability protection 19.1 Prevent queries . . .
. . .
templates 22.2 Embed the search engine into existing HTML code 22.3 The different style sheet files 23. JSON, XML and RSS result output [ Documentation ] 1. Settings, customizing and statistics If you want to change settings, behavior and design of Sphider-plus, you can do so by means of the Admin interface. There is a wide range of . . .
. . .
interface. There is a wide range of settings foreseen for Sphider-plus. Separated into different submenus like: Sites: - Add Site - Index only the new - Re-index all - Re-index only preferred URLs - Erase Re-index (available also for individual URLs) - Import/export URL list - Approve sites - Banned domains Categories: - Add, edit, delete - . . .
. . .
URL list - Approve sites - Banned domains Categories: - Add, edit, delete - Create new subcategory under Index: - Basic indexing options - Advanced options Clean: - Clean keywords not associated with any link - Clean links not associated with any site - Clean Category table not associated with any site - Clean Media links - Clear Temp table . . .
. . .
- Clear Search log - Clear 'Most Popular Page Links' log - Clear 'Most Popular Media Links' log - Clear Spider log, separate and bulk delete - Clear Thumbnail images, separate and bulk delete - Clear Text cache - Clear Media cache - Clear IDS log file - Clear flood attempts log file - Clear all entries in addurl or banned table - Truncate all . . .
. . .
flood attempts log file - Clear all entries in addurl or banned table - Truncate all tables in database Settings: - General Settings - Index Log Settings - Spider Settings - Search Settings - Order of Result listing - Suggest Options - Page Indexing Weights Database: - Configure up to 5 databases with unlimited number of table sets - Activate . . .
. . .
Database: - Configure up to 5 databases with unlimited number of table sets - Activate separately for 'Admin', 'Search' user and 'Suggest URL user' - Backup / Restore - Copy / Move - Optimize Templates: In order to enable customer's integration of Sphider-plus into existing sites, HTML templates are prepared for Search form Text result . . .
. . .
Search form Text result listing Media result listing Most popular queries etc. Three different designs are offered, which may be selected in submenu 'Settings'. If the layout does not fit the design of your site (which is normal), you may create new designs and modify the appropriate file /templates/My_template/adminstyle.css . . .
. . .
the appropriate file /templates/My_template/adminstyle.css /templates/My_template/userstyle.css Statistics output: - Top keywords (Top 50 with hit counter). - All indexed thumbnails w 3e80 ith ID3 and EXIF info. - Larges pages offering link URL and file size. - Most Popular Searches for text links offering: Link addr., total clicks, last . . .
. . .
Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Searches for media links offering: Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Links (click counter). - Search log offering: Query, Results, Queried at, Time taken, User IP, Country, Host name (Latest100) - Index log offering: . . .
. . .
Country, Host name (Latest100) - Index log offering: File-name, index date and delete option - sitemap log offering: sitemap.xml output sitemap list offering file/page suffixes - IDS log offering: IP, host, query, impact, involved tags, date and time of intrusion. - Flood attempts log offering: IP, query, date and time of flood attempt. - Auto . . .
. . .
attempts log offering: IP, query, date and time of flood attempt. - Auto Re-index log file - Server info offering: Server software, environment, MySQL, PDF-converter, image functions, php.ini file PHP integration, PHP security info. Each item holding lists of details. All text links, media links and thumbnails are active linked. As stated in . . .
. . .
individual Re-indexing, the periodical Re-indexer could be started and aborted in the "Options" menu of each site. 2.5 Preferred Re-indexing Each new URL added to the Admin backend, could be supplied with a priority level. This level will be used by the option 'Re-index only preferred sites'. Level 1 will be interpreted as most important, while . . .
. . .
less all index results will be stored in log files in sub folder /admin/log/ The names of the log files look like: db2_100524-21.47.56_1.html (log file of first thread) db2_100524-21.48.12_2.html (log file of second thread) and is build by the following items: db2 - Number of database. 100524 - Date (May 24, 2010) 21.47.56 - Time when this thread . . .
. . .
spider.php -new1 php spider.php -new2 etc. The IDs will be added to the name of the corresponding log files like: db2_100524-21.47.56_ID1.html (log file of first thread) db2_100524-21.48.12_ID2.html (log file of second thread) IDs could be defined by personal requirements, but the limitations for file names with respect to the OS should be taken . . .
. . .
but will not erase the content of all the other tables. So the check whether the content of a page has changed (MD5sum) is still available for a fast re-index procedure. Once prepared, multithreaded re-indexing could be invoked by starting several threads and adding individual IDs to the option parameter like: php spider.php -erased1 php . . .
. . .
4.4 Ignoring parts of a page Sphider-plus includes an option to exclude parts of pages from being indexed. Thi 775e s can for example be used to prevent search result flooding when certain keywords appear on certain part in most pages (like a header, footer or a menu). Any part of a page between <! sphider_noindex > and <! . . .
. . .
<! sphider_noindex > and <! /sphider_noindex > tags is not indexed, however links in it are followed. 4.5 Ignoring parts of a page by <div id='abc'> Ignoring parts of a page by the <! sphider_noindex > tags requires direct access to the page, because the tags need to be added (edited) to the page. A more flexible method, . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ menu[0-5] / 4.6 Indexing only parts of a page by <div id='abc'> If enabled in Admin settings, the values as defined in the list-file /include/common/divs_use.txt will be used to index only the content between <div . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ table[0-5] / 4.7 Ignore HTML elements defined by <tagname> . . </tagname> This option is foreseen to cooperate with the new HTML5 elements like section, nav, aside, hgroup, article, header, footer, etc. HTML elements . . .
. . .
contain a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */nav[0-5]/ Please keep in mind that element names placed in /include/common/elements_not.txt will be processed case-sensitive. 4.8 Index only HTML elements defined by <tagname> . . </tagname> This is the vice versa . . .
. . .
of all the duplicate content URLs: http://www.example.com/product.php?item=swedish-fish&category=gummy-candy http//wwwexamplecom/productphpitem=swedish-fish&trackingid=1234&sessionid=5678 and Sphider-plus will understand that the duplicates all refer to the canonical URL: http://www.example.com/product.php?item=swedish-fish. The . . .
. . .
charset are supported by the ConvertCharset function and will be used to convert text into UTF-8 Unicode: WINDOWS windows-1250 - Central Europe windows-1251 - Cyrillic windows-1252 - Latin I windows-1253 - Greek windows-1254 - Turkish windows-1255 - Hebrew windows-1256 - Arabic windows-1257 - Baltic windows-1258 - Viet Nam cp874 - Thai - this . . .
. . .
- Baltic windows-1258 - Viet Nam cp874 - Thai - this file is also for DOS DOS cp437 - Latin US cp737 - Greek cp775 - BaltRim cp850 - Latin1 cp852 - Latin2 cp855 - Cyrylic cp857 - Turkish cp860 - Portuguese cp861 - Iceland cp862 - Hebrew cp863 - Canada cp864 - Arabic cp865 - Nordic cp866 - Cyrylic Russian (this is the one, used in IE . . .
. . .
IE Cyrillic (DOS) ) cp869 - Greek2 MAC (Apple) x-mac-cyrillic x-mac-greek x-mac-icelandic x-mac-ce x-mac-roman ISO iso-8859-1 iso-8859-2 iso-8859-3 iso-8859-4 iso-8859-5 iso-8859-6 iso-8859-7 iso-8859-8 iso-8859-9 iso-8859-10 iso-8859-11 iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 . . .
. . .
iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 cp500 cp856 cp875 cp1006 cp1026 koi8-r (Cyrillic) koi8-u (Cyrillic Ukrainian) nextstep us-ascii us-ascii-quotes DSP implementation for NeXT stdenc symbol zdingbat And specially for old Polish programs: mazovia This list is to be read . . .
. . .
Access denied; you need the RELOAD privilege. . . Top 10. Enable real-time output of logging data Up to version 1.5 of Sphider-plus, during index / re-index there was no printout available because: - Several servers, especially on Win32, buffer the output from the script until it terminates before transmitting the results to the browser. - . . .
. . .
is seen. - Some versions of Microsoft Internet Explorer only start to display the page after they have received 256 bytes of output. As progress was not presented during index / re- index procedure, waiting for results became a pain in the neck. Selectable in Admin setting together with the update interval (1 - 10 seconds), AJAX technology was . . .
. . .
characters in front of words Warning: This option should be used with special care and not be activated for non ISO-8859 charsets. Some special characters as part of the word ending might be erased by accidental. Top 13. Media search for images, audio streams and videos 13.1 Media indexing Index of media files is enabled by separated Admin . . .
. . .
and 'Found at' - Total clicks - Last clicked - Query input 'Indexed Image Thumbnails' presenting: - Thumbnail 150 x 100 pixel - Image details like title, filename size of original image, link- and thumb-id - Option to delete the thumbnail In order to open the media files all tables contain active links. Media results are also stored in . . .
. . .
Settings' menu of the admin backend: First option: For results of XML product feeds, present the text extract of 350 characters as part of product attributes. If this option is not activated, all products of the XML product feed will be presented in search results, and the hits of the query string will be highlighted in all involved products. . . .
. . .
content of the database tables (those with the same table prefix) will be destroyed by the restore procedure. 16.5 Copy & Move This section of the Database Management will present only those databases, which are correctly configured, assigned and do have a set of installed tables as described in chapter Definition and configuration. This . . .
. . .
results. Never the less to find these x results, the complete database had to been browsed. Starting with version 2.5, an additional clean option is offered as part of the Admin backend. Main advantage of this option is a significant reduction of the search time for any query, because the content of the db could be limited to offer only x . . .
. . .
<link>http://www.abc.de/images/warp.gif</link> <title>warp.gif</title> <x_size>635</x_size> <y_size>98</y_size> </media_result> <media_result> <num>2</num> <type>audio</type> <url>http://www.abc.de/index.php</url> . . .
. . .
the following content: {"query":"colour","ip":"::1","host_name":"guard_007-hoster","query_time":"2016-03-18 10:56:23 AM","consumed":0.016,"total_results":2,"num_of_results":2,"from":1,"to":2,"text_results":[{"num":1,"weight":"100.0","url":"http:\/\/www.english.le-piaggie.info\/Info_eng.pdf","title":" Info_eng","fulltxt":" . . . . . .
. . .
placed on the ground floor, today's big kitchen with the heating fireplace for open and close mode of . . . ","page_size":"5,661.6kb","domain_name":"www.english.le-piaggie.info"},{"num":2,"weight":"50.0","url":"http:\/\/www.english.le-piaggie.info\/html\/description.html","title":" Description","fulltxt":" . . . cable. Additional active . . .
. . .
D.O.C.G. as far as to the Pratomagno mountains 160 olive-trees Detached house, about 800 m to reach the . . . ","page_size":"26.4 kb","domain_name":"www.english.le-piaggie.info"}]} Top . . .


5a2 All required information. Introduction Release and Legal Info Installation Documentation Change Log [ Documentation Summary ] Preamble: The info presented here is valid only for the latest release of Sphider-plus. At present version 4.2024a published February 12, 2024 is the actual release. 1. Settings, customizing and statistics 2. Indexing . . .
. . .
2. Indexing 2.1 Various options 2.2 Allow other hosts in same domain 2.3 Word stemming 2.4 Periodical Re-indexing 2.5 Preferred indexing 2.6 Multithreaded indexing 2.7 Create thumbnails during index procedure 2.8 Prevent indexing of known malware and pishing pages 2.9 Follow and create sitemap files 2.10 Use private sitemap instead of global . . .
. . .
Sitemap file 3. Using the indexer from command line 3.1 All options 3.2 Multithreaded indexing 4. Keeping pages, words and files from being indexed 4.1 robots.txt 4.2 Must include / must not include string list 4.3 Ignoring links 4.4 Ignoring parts of a page by <! sphider_noindex > 4.5 Ignoring parts of a page by <div id='abc'> . . .
. . .
charset' 6. Search modes 6.1 Search with wildcards * 6.2 Strict search ! 6.3 Tolerant search 6.4 Link search 6.5 Media search 6.6 Search only in one domain 6.7 Search in categories 6.8 Greek language support 6.9 Block queries 7. Chronological order for result listing 7.1 Text result listing 7.2 Media result listing 8. PDF converter 9. Clean . . .
. . .
of logging data 11. Error messages and Debug mode 12. Delete secondary characters 13. Media search for images, audio streams and videos 13.1 Media indexing 13.2 Not supported media content 13.3 Search for media content 13.4 Statistics for media content 14. Feed support 14.1 XML product feeds 14.2 RDF, RSD, RSS and Atom feeds 15. Result . . .
. . .
Overview 16.2 Definition and configuration 16.3 Activate / disable database 16.4 Backup & Restore of databases 16.5 Copy & and Move 16.6 Enhancing functionality of multiple database support 17. Search in categories 17.1 Hierachical structure 17.2 Parallel structure 18. User suggested sites 19. Vulnerability protection 19.1 Prevent queries . . .
. . .
templates 22.2 Embed the search engine into existing HTML code 22.3 The different style sheet files 23. JSON, XML and RSS result output [ Documentation ] 1. Settings, customizing and statistics If you want to change settings, behavior and design of Sphider-plus, you can do so by means of the Admin interface. There is a wide range of . . .
. . .
interface. There is a wide range of settings foreseen for Sphider-plus. Separated into different submenus like: Sites: - Add Site - Index only the new - Re-index all - Re-index only preferred URLs - Erase Re-index (available also for individual URLs) - Import/export URL list - Approve sites - Banned domains Categories: - Add, edit, delete - . . .
. . .
URL list - Approve sites - Banned domains Categories: - Add, edit, delete - Create new subcategory under Index: - Basic indexing options - Advanced options Clean: - Clean keywords not associated with any link - Clean links not associated with any site - Clean Category table not associated with any site - Clean Media links - Clear Temp table . . .
. . .
- Clear Search log - Clear 'Most Popular Page Links' log - Clear 'Most Popular Media Links' log - Clear Spider log, separate and bulk delete - Clear Thumbnail images, separate and bulk delete - Clear Text cache - Clear Media cache - Clear IDS log file - Clear flood attempts log file - Clear all entries in addurl or banned table - Truncate all . . .
. . .
flood attempts log file - Clear all entries in addurl or banned table - Truncate all tables in database Settings: - General Settings - Index Log Settings - Spider Settings - Search Settings - Order of Result listing - Suggest Options - Page Indexing Weights Database: - Configure up to 5 databases with unlimited number of table sets - Activate . . .
. . .
Database: - Configure up to 5 databases with unlimited number of table sets - Activate separately for 'Admin', 'Search' user and 'Suggest URL user' - Backup / Restore - Copy / Move - Optimize Templates: In order to enable customer's integration of Sphider-plus into existing sites, HTML templates are prepared for Search form Text result . . .
. . .
Search form Text result listing Media result listing Most popular queries etc. Three different designs are offered, which may be selected in submenu 'Settings'. If the layout does not fit the design of your site (which is normal), you may create new designs and modify the appropriate file /templates/My_template/adminstyle.css . . .
. . .
the appropriate file /templates/My_template/adminstyle.css /templates/My_template/userstyle.css Statistics output: - Top keywords (Top 50 with hit counter). - All indexed thumbnails w 3e80 ith ID3 and EXIF info. - Larges pages offering link URL and file size. - Most Popular Searches for text links offering: Link addr., total clicks, last . . .
. . .
Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Searches for media links offering: Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Links (click counter). - Search log offering: Query, Results, Queried at, Time taken, User IP, Country, Host name (Latest100) - Index log offering: . . .
. . .
Country, Host name (Latest100) - Index log offering: File-name, index date and delete option - sitemap log offering: sitemap.xml output sitemap list offering file/page suffixes - IDS log offering: IP, host, query, impact, involved tags, date and time of intrusion. - Flood attempts log offering: IP, query, date and time of flood attempt. - Auto . . .
. . .
attempts log offering: IP, query, date and time of flood attempt. - Auto Re-index log file - Server info offering: Server software, environment, MySQL, PDF-converter, image functions, php.ini file PHP integration, PHP security info. Each item holding lists of details. All text links, media links and thumbnails are active linked. As stated in . . .
. . .
individual Re-indexing, the periodical Re-indexer could be started and aborted in the "Options" menu of each site. 2.5 Preferred Re-indexing Each new URL added to the Admin backend, could be supplied with a priority level. This level will be used by the option 'Re-index only preferred sites'. Level 1 will be interpreted as most important, while . . .
. . .
less all index results will be stored in log files in sub folder /admin/log/ The names of the log files look like: db2_100524-21.47.56_1.html (log file of first thread) db2_100524-21.48.12_2.html (log file of second thread) and is build by the following items: db2 - Number of database. 100524 - Date (May 24, 2010) 21.47.56 - Time when this thread . . .
. . .
spider.php -new1 php spider.php -new2 etc. The IDs will be added to the name of the corresponding log files like: db2_100524-21.47.56_ID1.html (log file of first thread) db2_100524-21.48.12_ID2.html (log file of second thread) IDs could be defined by personal requirements, but the limitations for file names with respect to the OS should be taken . . .
. . .
but will not erase the content of all the other tables. So the check whether the content of a page has changed (MD5sum) is still available for a fast re-index procedure. Once prepared, multithreaded re-indexing could be invoked by starting several threads and adding individual IDs to the option parameter like: php spider.php -erased1 php . . .
. . .
4.4 Ignoring parts of a page Sphider-plus includes an option to exclude parts of pages from being indexed. Thi 775e s can for example be used to prevent search result flooding when certain keywords appear on certain part in most pages (like a header, footer or a menu). Any part of a page between <! sphider_noindex > and <! . . .
. . .
<! sphider_noindex > and <! /sphider_noindex > tags is not indexed, however links in it are followed. 4.5 Ignoring parts of a page by <div id='abc'> Ignoring parts of a page by the <! sphider_noindex > tags requires direct access to the page, because the tags need to be added (edited) to the page. A more flexible method, . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ menu[0-5] / 4.6 Indexing only parts of a page by <div id='abc'> If enabled in Admin settings, the values as defined in the list-file /include/common/divs_use.txt will be used to index only the content between <div . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ table[0-5] / 4.7 Ignore HTML elements defined by <tagname> . . </tagname> This option is foreseen to cooperate with the new HTML5 elements like section, nav, aside, hgroup, article, header, footer, etc. HTML elements . . .
. . .
contain a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */nav[0-5]/ Please keep in mind that element names placed in /include/common/elements_not.txt will be processed case-sensitive. 4.8 Index only HTML elements defined by <tagname> . . </tagname> This is the vice versa . . .
. . .
of all the duplicate content URLs: http://www.example.com/product.php?item=swedish-fish&category=gummy-candy http://www.example.com/product.php?item=swedish-fish&trackingid=1234&sessionid=5678 and Sphider-plus will understand that the duplicates all refer to the canonical URL: http://www.example.com/product.php?item=swedish-fish. The . . .
. . .
charset are supported by the ConvertCharset function and will be used to convert text into UTF-8 Unicode: WINDOWS windows-1250 - Central Europe windows-1251 - Cyrillic windows-1252 - Latin I windows-1253 - Greek windows-1254 - Turkish windows-1255 - Hebrew windows-1256 - Arabic windows-1257 - Baltic windows-1258 - Viet Nam cp874 - Thai - this . . .
. . .
- Baltic windows-1258 - Viet Nam cp874 - Thai - this file is also for DOS DOS cp437 - Latin US cp737 - Greek cp775 - BaltRim cp850 - Latin1 cp852 - Latin2 cp855 - Cyrylic cp857 - Turkish cp860 - Portuguese cp861 - Iceland cp862 - Hebrew cp863 - Canada cp864 - Arabic cp865 - Nordic cp866 - Cyrylic Russian (this is the one, used in IE . . .
. . .
IE Cyrillic (DOS) ) cp869 - Greek2 MAC (Apple) x-mac-cyrillic x-mac-greek x-mac-icelandic x-mac-ce x-mac-roman ISO iso-8859-1-1 iso-8859-1-2 iso-8859-1-3 iso-8859-1-4 iso-8859-1-5 iso-8859-1-6 iso-8859-1-7 iso-8859-1-8 iso-8859-1-9 iso-8859-1-10 iso-8859-1-11 iso-8859-1-12 iso-8859-1-13 iso-8859-1-14 iso-8859-1-15 iso-8859-1-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 . . .
. . .
iso-8859-1-12 iso-8859-1-13 iso-8859-1-14 iso-8859-1-15 iso-8859-1-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 cp500 cp856 cp875 cp1006 cp1026 koi8-r (Cyrillic) koi8-u (Cyrillic Ukrainian) nextstep us-ascii us-ascii-quotes DSP implementation for NeXT stdenc symbol zdingbat And specially for old Polish programs: mazovia This list is to be read . . .
. . .
Access denied; you need the RELOAD privilege. . . Top 10. Enable real-time output of logging data Up to version 1.5 of Sphider-plus, during index / re-index there was no printout available because: - Several servers, especially on Win32, buffer the output from the script until it terminates before transmitting the results to the browser. - . . .
. . .
is seen. - Some versions of Microsoft Internet Explorer only start to display the page after they have received 256 bytes of output. As progress was not presented during index / re- index procedure, waiting for results became a pain in the neck. Selectable in Admin setting together with the update interval (1 - 10 seconds), AJAX technology was . . .
. . .
characters in front of words Warning: This option should be used with special care and not be activated for non iso-8859-1 charsets. Some special characters as part of the word ending might be erased by accidental. Top 13. Media search for images, audio streams and videos 13.1 Media indexing Index of media files is enabled by separated Admin . . .
. . .
and 'Found at' - Total clicks - Last clicked - Query input 'Indexed Image Thumbnails' presenting: - Thumbnail 150 x 100 pixel - Image details like title, filename size of original image, link- and thumb-id - Option to delete the thumbnail In order to open the media files all tables contain active links. Media results are also stored in . . .
. . .
Settings' menu of the admin backend: First option: For results of XML product feeds, present the text extract of 350 characters as part of product attributes. If this option is not activated, all products of the XML product feed will be presented in search results, and the hits of the query string will be highlighted in all involved products. . . .
. . .
content of the database tables (those with the same table prefix) will be destroyed by the restore procedure. 16.5 Copy & Move This section of the Database Management will present only those databases, which are correctly configured, assigned and do have a set of installed tables as described in chapter Definition and configuration. This . . .
. . .
results. Never the less to find these x results, the complete database had to been browsed. Starting with version 2.5, an additional clean option is offered as part of the Admin backend. Main advantage of this option is a significant reduction of the search time for any query, because the content of the db could be limited to offer only x . . .
. . .
<link>http://www.abc.de/images/warp.gif</link> <title>warp.gif</title> <x_size>635</x_size> <y_size>98</y_size> </media_result> <media_result> <num>2</num> <type>audio</type> <url>http://www.abc.de/index.php</url> . . .
. . .
the following content: {"query":"colour","ip":"::1","host_name":"guard_007-hoster","query_time":"2016-03-18 10:56:23 AM","consumed":0.016,"total_results":2,"num_of_results":2,"from":1,"to":2,"text_results":[{"num":1,"weight":"100.0","url":"http:\/\/www.english.le-piaggie.info\/Info_eng.pdf","title":" Info_eng","fulltxt":" . . . . . .
. . .
placed on the ground floor, today's big kitchen with the heating fireplace for open and close mode of . . . ","page_size":"5,661.6kb","domain_name":"www.english.le-piaggie.info"},{"num":2,"weight":"50.0","url":"http:\/\/www.english.le-piaggie.info\/html\/description.html","title":" Description","fulltxt":" . . . cable. Additional active . . .
. . .
D.O.C.G. as far as to the Pratomagno mountains 160 olive-trees Detached house, about 800 m to reach the . . . ","page_size":"26.4 kb","domain_name":"www.english.le-piaggie.info"}]} Top . . .


5a2 All required information. Introduction Release and Legal Info Installation Documentation Change Log [ Documentation Summary ] Preamble: The info presented here is valid only for the latest release of Sphider-plus. At present version 4.2024a published February 12, 2024 is the actual release. 1. Settings, customizing and statistics 2. Indexing . . .
. . .
2. Indexing 2.1 Various options 2.2 Allow other hosts in same domain 2.3 Word stemming 2.4 Periodical Re-indexing 2.5 Preferred indexing 2.6 Multithreaded indexing 2.7 Create thumbnails during index procedure 2.8 Prevent indexing of known malware and pishing pages 2.9 Follow and create sitemap files 2.10 Use private sitemap instead of global . . .
. . .
Sitemap file 3. Using the indexer from command line 3.1 All options 3.2 Multithreaded indexing 4. Keeping pages, words and files from being indexed 4.1 robots.txt 4.2 Must include / must not include string list 4.3 Ignoring links 4.4 Ignoring parts of a page by <! sphider_noindex > 4.5 Ignoring parts of a page by <div id='abc'> . . .
. . .
charset' 6. Search modes 6.1 Search with wildcards * 6.2 Strict search ! 6.3 Tolerant search 6.4 Link search 6.5 Media search 6.6 Search only in one domain 6.7 Search in categories 6.8 Greek language support 6.9 Block queries 7. Chronological order for result listing 7.1 Text result listing 7.2 Media result listing 8. PDF converter 9. Clean . . .
. . .
of logging data 11. Error messages and Debug mode 12. Delete secondary characters 13. Media search for images, audio streams and videos 13.1 Media indexing 13.2 Not supported media content 13.3 Search for media content 13.4 Statistics for media content 14. Feed support 14.1 XML product feeds 14.2 RDF, RSD, RSS and Atom feeds 15. Result . . .
. . .
Overview 16.2 Definition and configuration 16.3 Activate / disable database 16.4 Backup & Restore of databases 16.5 Copy & and Move 16.6 Enhancing functionality of multiple database support 17. Search in categories 17.1 Hierachical structure 17.2 Parallel structure 18. User suggested sites 19. Vulnerability protection 19.1 Prevent queries . . .
. . .
templates 22.2 Embed the search engine into existing HTML code 22.3 The different style sheet files 23. JSON, XML and RSS result output [ Documentation ] 1. Settings, customizing and statistics If you want to change settings, behavior and design of Sphider-plus, you can do so by means of the Admin interface. There is a wide range of . . .
. . .
interface. There is a wide range of settings foreseen for Sphider-plus. Separated into different submenus like: Sites: - Add Site - Index only the new - Re-index all - Re-index only preferred URLs - Erase Re-index (available also for individual URLs) - Import/export URL list - Approve sites - Banned domains Categories: - Add, edit, delete - . . .
. . .
URL list - Approve sites - Banned domains Categories: - Add, edit, delete - Create new subcategory under Index: - Basic indexing options - Advanced options Clean: - Clean keywords not associated with any link - Clean links not associated with any site - Clean Category table not associated with any site - Clean Media links - Clear Temp table . . .
. . .
- Clear Search log - Clear 'Most Popular Page Links' log - Clear 'Most Popular Media Links' log - Clear Spider log, separate and bulk delete - Clear Thumbnail images, separate and bulk delete - Clear Text cache - Clear Media cache - Clear IDS log file - Clear flood attempts log file - Clear all entries in addurl or banned table - Truncate all . . .
. . .
flood attempts log file - Clear all entries in addurl or banned table - Truncate all tables in database Settings: - General Settings - Index Log Settings - Spider Settings - Search Settings - Order of Result listing - Suggest Options - Page Indexing Weights Database: - Configure up to 5 databases with unlimited number of table sets - Activate . . .
. . .
Database: - Configure up to 5 databases with unlimited number of table sets - Activate separately for 'Admin', 'Search' user and 'Suggest URL user' - Backup / Restore - Copy / Move - Optimize Templates: In order to enable customer's integration of Sphider-plus into existing sites, HTML templates are prepared for Search form Text result . . .
. . .
Search form Text result listing Media result listing Most popular queries etc. Three different designs are offered, which may be selected in submenu 'Settings'. If the layout does not fit the design of your site (which is normal), you may create new designs and modify the appropriate file /templates/My_template/adminstyle.css . . .
. . .
the appropriate file /templates/My_template/adminstyle.css /templates/My_template/userstyle.css Statistics output: - Top keywords (Top 50 with hit counter). - All indexed thumbnails w 3e80 ith ID3 and EXIF info. - Larges pages offering link URL and file size. - Most Popular Searches for text links offering: Link addr., total clicks, last . . .
. . .
Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Searches for media links offering: Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Links (click counter). - Search log offering: Query, Results, Queried at, Time taken, User IP, Country, Host name (Latest100) - Index log offering: . . .
. . .
Country, Host name (Latest100) - Index log offering: File-name, index date and delete option - sitemap log offering: sitemap.xml output sitemap list offering file/page suffixes - IDS log offering: IP, host, query, impact, involved tags, date and time of intrusion. - Flood attempts log offering: IP, query, date and time of flood attempt. - Auto . . .
. . .
attempts log offering: IP, query, date and time of flood attempt. - Auto Re-index log file - Server info offering: Server software, environment, MySQL, PDF-converter, image functions, php.ini file PHP integration, PHP security info. Each item holding lists of details. All text links, media links and thumbnails are active linked. As stated in . . .
. . .
individual Re-indexing, the periodical Re-indexer could be started and aborted in the "Options" menu of each site. 2.5 Preferred Re-indexing Each new URL added to the Admin backend, could be supplied with a priority level. This level will be used by the option 'Re-index only preferred sites'. Level 1 will be interpreted as most important, while . . .
. . .
less all index results will be stored in log files in sub folder /admin/log/ The names of the log files look like: db2_100524-21.47.56_1.html (log file of first thread) db2_100524-21.48.12_2.html (log file of second thread) and is build by the following items: db2 - Number of database. 100524 - Date (May 24, 2010) 21.47.56 - Time when this thread . . .
. . .
spider.php -new1 php spider.php -new2 etc. The IDs will be added to the name of the corresponding log files like: db2_100524-21.47.56_ID1.html (log file of first thread) db2_100524-21.48.12_ID2.html (log file of second thread) IDs could be defined by personal requirements, but the limitations for file names with respect to the OS should be taken . . .
. . .
but will not erase the content of all the other tables. So the check whether the content of a page has changed (MD5sum) is still available for a fast re-index procedure. Once prepared, multithreaded re-indexing could be invoked by starting several threads and adding individual IDs to the option parameter like: php spider.php -erased1 php . . .
. . .
4.4 Ignoring parts of a page Sphider-plus includes an option to exclude parts of pages from being indexed. Thi 775e s can for example be used to prevent search result flooding when certain keywords appear on certain part in most pages (like a header, footer or a menu). Any part of a page between <! sphider_noindex > and <! . . .
. . .
<! sphider_noindex > and <! /sphider_noindex > tags is not indexed, however links in it are followed. 4.5 Ignoring parts of a page by <div id='abc'> Ignoring parts of a page by the <! sphider_noindex > tags requires direct access to the page, because the tags need to be added (edited) to the page. A more flexible method, . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ menu[0-5] / 4.6 Indexing only parts of a page by <div id='abc'> If enabled in Admin settings, the values as defined in the list-file /include/common/divs_use.txt will be used to index only the content between <div . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ table[0-5] / 4.7 Ignore HTML elements defined by <tagname> . . </tagname> This option is foreseen to cooperate with the new HTML5 elements like section, nav, aside, hgroup, article, header, footer, etc. HTML elements . . .
. . .
contain a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */nav[0-5]/ Please keep in mind that element names placed in /include/common/elements_not.txt will be processed case-sensitive. 4.8 Index only HTML elements defined by <tagname> . . </tagname> This is the vice versa . . .
. . .
of all the duplicate content URLs: http://www.example.com/product.php?item=swedish-fish&category=gummy-candy http://www.example.com/product.php?item=swedish-fish&trackingid=1234&sessionid=5678 and Sphider-plus will understand that the duplicates all refer to the canonical URL: http://www.example.com/product.php?item=swedish-fish. The . . .
. . .
charset are supported by the ConvertCharset function and will be used to convert text into UTF-8 Unicode: WINDOWS windows-1250 - Central Europe windows-1251 - Cyrillic windows-1252 - Latin I windows-1253 - Greek windows-1254 - Turkish windows-1255 - Hebrew windows-1256 - Arabic windows-1257 - Baltic windows-1258 - Viet Nam cp874 - Thai - this . . .
. . .
- Baltic windows-1258 - Viet Nam cp874 - Thai - this file is also for DOS DOS cp437 - Latin US cp737 - Greek cp775 - BaltRim cp850 - Latin1 cp852 - Latin2 cp855 - Cyrylic cp857 - Turkish cp860 - Portuguese cp861 - Iceland cp862 - Hebrew cp863 - Canada cp864 - Arabic cp865 - Nordic cp866 - Cyrylic Russian (this is the one, used in IE . . .
. . .
IE Cyrillic (DOS) ) cp869 - Greek2 MAC (Apple) x-mac-cyrillic x-mac-greek x-mac-icelandic x-mac-ce x-mac-roman ISO iso-8859-1 iso-8859-2 iso-8859-3 iso-8859-4 iso-8859-5 iso-8859-6 iso-8859-7 iso-8859-8 iso-8859-9 iso-8859-10 iso-8859-11 iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 . . .
. . .
iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 cp500 cp856 cp875 cp1006 cp1026 koi8-r (Cyrillic) koi8-u (Cyrillic Ukrainian) nextstep us-ascii us-ascii-quotes DSP implementation for NeXT stdenc symbol zdingbat And specially for old Polish programs: mazovia This list is to be read . . .
. . .
Access denied; you need the RELOAD privilege. . . Top 10. Enable real-time output of logging data Up to version 1.5 of Sphider-plus, during index / re-index there was no printout available because: - Several servers, especially on Win32, buffer the output from the script until it terminates before transmitting the results to the browser. - . . .
. . .
is seen. - Some versions of Microsoft Internet Explorer only start to display the page after they have received 256 bytes of output. As progress was not presented during index / re- index procedure, waiting for results became a pain in the neck. Selectable in Admin setting together with the update interval (1 - 10 seconds), AJAX technology was . . .
. . .
characters in front of words Warning: This option should be used with special care and not be activated for non ISO-8859 charsets. Some special characters as part of the word ending might be erased by accidental. Top 13. Media search for images, audio streams and videos 13.1 Media indexing Index of media files is enabled by separated Admin . . .
. . .
and 'Found at' - Total clicks - Last clicked - Query input 'Indexed Image Thumbnails' presenting: - Thumbnail 150 x 100 pixel - Image details like title, filename size of original image, link- and thumb-id - Option to delete the thumbnail In order to open the media files all tables contain active links. Media results are also stored in . . .
. . .
Settings' menu of the admin backend: First option: For results of XML product feeds, present the text extract of 350 characters as part of product attributes. If this option is not activated, all products of the XML product feed will be presented in search results, and the hits of the query string will be highlighted in all involved products. . . .
. . .
content of the database tables (those with the same table prefix) will be destroyed by the restore procedure. 16.5 Copy & Move This section of the Database Management will present only those databases, which are correctly configured, assigned and do have a set of installed tables as described in chapter Definition and configuration. This . . .
. . .
results. Never the less to find these x results, the complete database had to been browsed. Starting with version 2.5, an additional clean option is offered as part of the Admin backend. Main advantage of this option is a significant reduction of the search time for any query, because the content of the db could be limited to offer only x . . .
. . .
<link>http://www.abc.de/images/warp.gif</link> <title>warp.gif</title> <x_size>635</x_size> <y_size>98</y_size> </media_result> <media_result> <num>2</num> <type>audio</type> <url>http://www.abc.de/index.php</url> . . .
. . .
the following content: {"query":"colour","ip":"::1","host_name":"guard_007-hoster","query_time":"2016-03-18 10:56:23 AM","consumed":0.016,"total_results":2,"num_of_results":2,"from":1,"to":2,"text_results":[{"num":1,"weight":"100.0","url":"http:\/\/www.english.le-piaggie.info\/Info_eng.pdf","title":" Info_eng","fulltxt":" . . . . . .
. . .
placed on the ground floor, today's big kitchen with the heating fireplace for open and close mode of . . . ","page_size":"5,661.6kb","domain_name":"www.english.le-piaggie.info"},{"num":2,"weight":"50.0","url":"http:\/\/www.english.le-piaggie.info\/html\/description.html","title":" Description","fulltxt":" . . . cable. Additional active . . .
. . .
D.O.C.G. as far as to the Pratomagno mountains 160 olive-trees Detached house, about 800 m to reach the . . . ","page_size":"26.4 kb","domain_name":"www.english.le-piaggie.info"}]} Top . . .


5a2 All required information. Introduction Release and Legal Info Installation Documentation Change Log [ Documentation Summary ] Preamble: The info presented here is valid only for the latest release of Sphider-plus. At present version 4.2024a published February 12, 2024 is the actual release. 1. Settings, customizing and statistics 2. Indexing . . .
. . .
2. Indexing 2.1 Various options 2.2 Allow other hosts in same domain 2.3 Word stemming 2.4 Periodical Re-indexing 2.5 Preferred indexing 2.6 Multithreaded indexing 2.7 Create thumbnails during index procedure 2.8 Prevent indexing of known malware and pishing pages 2.9 Follow and create sitemap files 2.10 Use private sitemap instead of global . . .
. . .
Sitemap file 3. Using the indexer from command line 3.1 All options 3.2 Multithreaded indexing 4. Keeping pages, words and files from being indexed 4.1 robots.txt 4.2 Must include / must not include string list 4.3 Ignoring links 4.4 Ignoring parts of a page by <! sphider_noindex > 4.5 Ignoring parts of a page by <div id='abc'> . . .
. . .
charset' 6. Search modes 6.1 Search with wildcards * 6.2 Strict search ! 6.3 Tolerant search 6.4 Link search 6.5 Media search 6.6 Search only in one domain 6.7 Search in categories 6.8 Greek language support 6.9 Block queries 7. Chronological order for result listing 7.1 Text result listing 7.2 Media result listing 8. PDF converter 9. Clean . . .
. . .
of logging data 11. Error messages and Debug mode 12. Delete secondary characters 13. Media search for images, audio streams and videos 13.1 Media indexing 13.2 Not supported media content 13.3 Search for media content 13.4 Statistics for media content 14. Feed support 14.1 XML product feeds 14.2 RDF, RSD, RSS and Atom feeds 15. Result . . .
. . .
Overview 16.2 Definition and configuration 16.3 Activate / disable database 16.4 Backup & Restore of databases 16.5 Copy & and Move 16.6 Enhancing functionality of multiple database support 17. Search in categories 17.1 Hierachical structure 17.2 Parallel structure 18. User suggested sites 19. Vulnerability protection 19.1 Prevent queries . . .
. . .
templates 22.2 Embed the search engine into existing HTML code 22.3 The different style sheet files 23. JSON, XML and RSS result output [ Documentation ] 1. Settings, customizing and statistics If you want to change settings, behavior and design of Sphider-plus, you can do so by means of the Admin interface. There is a wide range of . . .
. . .
interface. There is a wide range of settings foreseen for Sphider-plus. Separated into different submenus like: Sites: - Add Site - Index only the new - Re-index all - Re-index only preferred URLs - Erase Re-index (available also for individual URLs) - Import/export URL list - Approve sites - Banned domains Categories: - Add, edit, delete - . . .
. . .
URL list - Approve sites - Banned domains Categories: - Add, edit, delete - Create new subcategory under Index: - Basic indexing options - Advanced options Clean: - Clean keywords not associated with any link - Clean links not associated with any site - Clean Category table not associated with any site - Clean Media links - Clear Temp table . . .
. . .
- Clear Search log - Clear 'Most Popular Page Links' log - Clear 'Most Popular Media Links' log - Clear Spider log, separate and bulk delete - Clear Thumbnail images, separate and bulk delete - Clear Text cache - Clear Media cache - Clear IDS log file - Clear flood attempts log file - Clear all entries in addurl or banned table - Truncate all . . .
. . .
flood attempts log file - Clear all entries in addurl or banned table - Truncate all tables in database Settings: - General Settings - Index Log Settings - Spider Settings - Search Settings - Order of Result listing - Suggest Options - Page Indexing Weights Database: - Configure up to 5 databases with unlimited number of table sets - Activate . . .
. . .
Database: - Configure up to 5 databases with unlimited number of table sets - Activate separately for 'Admin', 'Search' user and 'Suggest URL user' - Backup / Restore - Copy / Move - Optimize Templates: In order to enable customer's integration of Sphider-plus into existing sites, HTML templates are prepared for Search form Text result . . .
. . .
Search form Text result listing Media result listing Most popular queries etc. Three different designs are offered, which may be selected in submenu 'Settings'. If the layout does not fit the design of your site (which is normal), you may create new designs and modify the appropriate file /templates/My_template/adminstyle.css . . .
. . .
the appropriate file /templates/My_template/adminstyle.css /templates/My_template/userstyle.css Statistics output: - Top keywords (Top 50 with hit counter). - All indexed thumbnails w 3e80 ith ID3 and EXIF info. - Larges pages offering link URL and file size. - Most Popular Searches for text links offering: Link addr., total clicks, last . . .
. . .
Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Searches for media links offering: Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Links (click counter). - Search log offering: Query, Results, Queried at, Time taken, User IP, Country, Host name (Latest100) - Index log offering: . . .
. . .
Country, Host name (Latest100) - Index log offering: File-name, index date and delete option - sitemap log offering: sitemap.xml output sitemap list offering file/page suffixes - IDS log offering: IP, host, query, impact, involved tags, date and time of intrusion. - Flood attempts log offering: IP, query, date and time of flood attempt. - Auto . . .
. . .
attempts log offering: IP, query, date and time of flood attempt. - Auto Re-index log file - Server info offering: Server software, environment, MySQL, PDF-converter, image functions, php.ini file PHP integration, PHP security info. Each item holding lists of details. All text links, media links and thumbnails are active linked. As stated in . . .
. . .
individual Re-indexing, the periodical Re-indexer could be started and aborted in the "Options" menu of each site. 2.5 Preferred Re-indexing Each new URL added to the Admin backend, could be supplied with a priority level. This level will be used by the option 'Re-index only preferred sites'. Level 1 will be interpreted as most important, while . . .
. . .
less all index results will be stored in log files in sub folder /admin/log/ The names of the log files look like: db2_100524-21.47.56_1.html (log file of first thread) db2_100524-21.48.12_2.html (log file of second thread) and is build by the following items: db2 - Number of database. 100524 - Date (May 24, 2010) 21.47.56 - Time when this thread . . .
. . .
spider.php -new1 php spider.php -new2 etc. The IDs will be added to the name of the corresponding log files like: db2_100524-21.47.56_ID1.html (log file of first thread) db2_100524-21.48.12_ID2.html (log file of second thread) IDs could be defined by personal requirements, but the limitations for file names with respect to the OS should be taken . . .
. . .
but will not erase the content of all the other tables. So the check whether the content of a page has changed (MD5sum) is still available for a fast re-index procedure. Once prepared, multithreaded re-indexing could be invoked by starting several threads and adding individual IDs to the option parameter like: php spider.php -erased1 php . . .
. . .
4.4 Ignoring parts of a page Sphider-plus includes an option to exclude parts of pages from being indexed. Thi 775e s can for example be used to prevent search result flooding when certain keywords appear on certain part in most pages (like a header, footer or a menu). Any part of a page between <! sphider_noindex > and <! . . .
. . .
<! sphider_noindex > and <! /sphider_noindex > tags is not indexed, however links in it are followed. 4.5 Ignoring parts of a page by <div id='abc'> Ignoring parts of a page by the <! sphider_noindex > tags requires direct access to the page, because the tags need to be added (edited) to the page. A more flexible method, . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ menu[0-5] / 4.6 Indexing only parts of a page by <div id='abc'> If enabled in Admin settings, the values as defined in the list-file /include/common/divs_use.txt will be used to index only the content between <div . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ table[0-5] / 4.7 Ignore HTML elements defined by <tagname> . . </tagname> This option is foreseen to cooperate with the new HTML5 elements like section, nav, aside, hgroup, article, header, footer, etc. HTML elements . . .
. . .
contain a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */nav[0-5]/ Please keep in mind that element names placed in /include/common/elements_not.txt will be processed case-sensitive. 4.8 Index only HTML elements defined by <tagname> . . </tagname> This is the vice versa . . .
. . .
of all the duplicate content URLs: http://www.example.com/product.php?item=swedish-fish&category=gummy-candy http://www.example.com/product.php?item=swedish-fish&trackingid=1234&sessionid=5678 and Sphider-plus will understand that the duplicates all refer to the canonical URL: http://www.example.com/product.php?item=swedish-fish. The . . .
. . .
charset are supported by the ConvertCharset function and will be used to convert text into UTF-8 Unicode: WINDOWS windows-1250 - Central Europe windows-1251 - Cyrillic windows-1252 - Latin I windows-1253 - Greek windows-1254 - Turkish windows-1255 - Hebrew windows-1256 - Arabic windows-1257 - Baltic windows-1258 - Viet Nam cp874 - Thai - this . . .
. . .
- Baltic windows-1258 - Viet Nam cp874 - Thai - this file is also for DOS DOS cp437 - Latin US cp737 - Greek cp775 - BaltRim cp850 - Latin1 cp852 - Latin2 cp855 - Cyrylic cp857 - Turkish cp860 - Portuguese cp861 - Iceland cp862 - Hebrew cp863 - Canada cp864 - Arabic cp865 - Nordic cp866 - Cyrylic Russian (this is the one, used in IE . . .
. . .
IE Cyrillic (DOS) ) cp869 - Greek2 MAC (Apple) x-mac-cyrillic x-mac-greek x-mac-icelandic x-mac-ce x-mac-roman ISO iso-8859-1 iso-8859-2 iso-8859-3 iso-8859-4 iso-8859-5 iso-8859-6 iso-8859-7 iso-8859-8 iso-8859-9 iso-8859-10 iso-8859-11 iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 . . .
. . .
iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 cp500 cp856 cp875 cp1006 cp1026 koi8-r (Cyrillic) koi8-u (Cyrillic Ukrainian) nextstep us-ascii us-ascii-quotes DSP implementation for NeXT stdenc symbol zdingbat And specially for old Polish programs: mazovia This list is to be read . . .
. . .
Access denied; you need the RELOAD privilege. . . Top 10. Enable real-time output of logging data Up to version 1.5 of Sphider-plus, during index / re-index there was no printout available because: - Several servers, especially on Win32, buffer the output from the script until it terminates before transmitting the results to the browser. - . . .
. . .
is seen. - Some versions of Microsoft Internet Explorer only start to display the page after they have received 256 bytes of output. As progress was not presented during index / re- index procedure, waiting for results became a pain in the neck. Selectable in Admin setting together with the update interval (1 - 10 seconds), AJAX technology was . . .
. . .
characters in front of words Warning: This option should be used with special care and not be activated for non ISO-8859 charsets. Some special characters as part of the word ending might be erased by accidental. Top 13. Media search for images, audio streams and videos 13.1 Media indexing Index of media files is enabled by separated Admin . . .
. . .
and 'Found at' - Total clicks - Last clicked - Query input 'Indexed Image Thumbnails' presenting: - Thumbnail 150 x 100 pixel - Image details like title, filename size of original image, link- and thumb-id - Option to delete the thumbnail In order to open the media files all tables contain active links. Media results are also stored in . . .
. . .
Settings' menu of the admin backend: First option: For results of XML product feeds, present the text extract of 350 characters as part of product attributes. If this option is not activated, all products of the XML product feed will be presented in search results, and the hits of the query string will be highlighted in all involved products. . . .
. . .
content of the database tables (those with the same table prefix) will be destroyed by the restore procedure. 16.5 Copy & Move This section of the Database Management will present only those databases, which are correctly configured, assigned and do have a set of installed tables as described in chapter Definition and configuration. This . . .
. . .
results. Never the less to find these x results, the complete database had to been browsed. Starting with version 2.5, an additional clean option is offered as part of the Admin backend. Main advantage of this option is a significant reduction of the search time for any query, because the content of the db could be limited to offer only x . . .
. . .
<link>http://www.abc.de/images/warp.gif</link> <title>warp.gif</title> <x_size>635</x_size> <y_size>98</y_size> </media_result> <media_result> <num>2</num> <type>audio</type> <url>http://www.abc.de/index.php</url> . . .
. . .
the following content: {"query":"colour","ip":"::1","host_name":"guard_007-hoster","query_time":"2016-03-18 10:56:23 AM","consumed":0.016,"total_results":2,"num_of_results":2,"from":1,"to":2,"text_results":[{"num":1,"weight":"100.0","url":"http:\/\/www.english.le-piaggie.info\/Info_eng.pdf","title":" Info_eng","fulltxt":" . . . . . .
. . .
placed on the ground floor, today's big kitchen with the heating fireplace for open and close mode of . . . ","page_size":"5,661.6kb","domain_name":"www.english.le-piaggie.info"},{"num":2,"weight":"50.0","url":"http:\/\/www.english.le-piaggie.info\/html\/description.html","title":" Description","fulltxt":" . . . cable. Additional active . . .
. . .
D.O.C.G. as far as to the Pratomagno mountains 160 olive-trees Detached house, about 800 m to reach the . . . ","page_size":"26.4 kb","domain_name":"www.english.le-piaggie.info"}]} Top . . .


5a2 All required information. Introduction Release and Legal Info Installation Documentation Change Log [ Documentation Summary ] Preamble: The info presented here is valid only for the latest release of Sphider-plus. At present version 4.2024a published February 12, 2024 is the actual release. 1. Settings, customizing and statistics 2. Indexing . . .
. . .
2. Indexing 2.1 Various options 2.2 Allow other hosts in same domain 2.3 Word stemming 2.4 Periodical Re-indexing 2.5 Preferred indexing 2.6 Multithreaded indexing 2.7 Create thumbnails during index procedure 2.8 Prevent indexing of known malware and pishing pages 2.9 Follow and create sitemap files 2.10 Use private sitemap instead of global . . .
. . .
Sitemap file 3. Using the indexer from command line 3.1 All options 3.2 Multithreaded indexing 4. Keeping pages, words and files from being indexed 4.1 robots.txt 4.2 Must include / must not include string list 4.3 Ignoring links 4.4 Ignoring parts of a page by <! sphider_noindex > 4.5 Ignoring parts of a page by <div id='abc'> . . .
. . .
charset' 6. Search modes 6.1 Search with wildcards * 6.2 Strict search ! 6.3 Tolerant search 6.4 Link search 6.5 Media search 6.6 Search only in one domain 6.7 Search in categories 6.8 Greek language support 6.9 Block queries 7. Chronological order for result listing 7.1 Text result listing 7.2 Media result listing 8. PDF converter 9. Clean . . .
. . .
of logging data 11. Error messages and Debug mode 12. Delete secondary characters 13. Media search for images, audio streams and videos 13.1 Media indexing 13.2 Not supported media content 13.3 Search for media content 13.4 Statistics for media content 14. Feed support 14.1 XML product feeds 14.2 RDF, RSD, RSS and Atom feeds 15. Result . . .
. . .
Overview 16.2 Definition and configuration 16.3 Activate / disable database 16.4 Backup & Restore of databases 16.5 Copy & and Move 16.6 Enhancing functionality of multiple database support 17. Search in categories 17.1 Hierachical structure 17.2 Parallel structure 18. User suggested sites 19. Vulnerability protection 19.1 Prevent queries . . .
. . .
templates 22.2 Embed the search engine into existing HTML code 22.3 The different style sheet files 23. JSON, XML and RSS result output [ Documentation ] 1. Settings, customizing and statistics If you want to change settings, behavior and design of Sphider-plus, you can do so by means of the Admin interface. There is a wide range of . . .
. . .
interface. There is a wide range of settings foreseen for Sphider-plus. Separated into different submenus like: Sites: - Add Site - Index only the new - Re-index all - Re-index only preferred URLs - Erase Re-index (available also for individual URLs) - Import/export URL list - Approve sites - Banned domains Categories: - Add, edit, delete - . . .
. . .
URL list - Approve sites - Banned domains Categories: - Add, edit, delete - Create new subcategory under Index: - Basic indexing options - Advanced options Clean: - Clean keywords not associated with any link - Clean links not associated with any site - Clean Category table not associated with any site - Clean Media links - Clear Temp table . . .
. . .
- Clear Search log - Clear 'Most Popular Page Links' log - Clear 'Most Popular Media Links' log - Clear Spider log, separate and bulk delete - Clear Thumbnail images, separate and bulk delete - Clear Text cache - Clear Media cache - Clear IDS log file - Clear flood attempts log file - Clear all entries in addurl or banned table - Truncate all . . .
. . .
flood attempts log file - Clear all entries in addurl or banned table - Truncate all tables in database Settings: - General Settings - Index Log Settings - Spider Settings - Search Settings - Order of Result listing - Suggest Options - Page Indexing Weights Database: - Configure up to 5 databases with unlimited number of table sets - Activate . . .
. . .
Database: - Configure up to 5 databases with unlimited number of table sets - Activate separately for 'Admin', 'Search' user and 'Suggest URL user' - Backup / Restore - Copy / Move - Optimize Templates: In order to enable customer's integration of Sphider-plus into existing sites, HTML templates are prepared for Search form Text result . . .
. . .
Search form Text result listing Media result listing Most popular queries etc. Three different designs are offered, which may be selected in submenu 'Settings'. If the layout does not fit the design of your site (which is normal), you may create new designs and modify the appropriate file /templates/My_template/adminstyle.css . . .
. . .
the appropriate file /templates/My_template/adminstyle.css /templates/My_template/userstyle.css Statistics output: - Top keywords (Top 50 with hit counter). - All indexed thumbnails w 3e80 ith ID3 and EXIF info. - Larges pages offering link URL and file size. - Most Popular Searches for text links offering: Link addr., total clicks, last . . .
. . .
Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Searches for media links offering: Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Links (click counter). - Search log offering: Query, Results, Queried at, Time taken, User IP, Country, Host name (Latest100) - Index log offering: . . .
. . .
Country, Host name (Latest100) - Index log offering: File-name, index date and delete option - sitemap log offering: sitemap.xml output sitemap list offering file/page suffixes - IDS log offering: IP, host, query, impact, involved tags, date and time of intrusion. - Flood attempts log offering: IP, query, date and time of flood attempt. - Auto . . .
. . .
attempts log offering: IP, query, date and time of flood attempt. - Auto Re-index log file - Server info offering: Server software, environment, MySQL, PDF-converter, image functions, php.ini file PHP integration, PHP security info. Each item holding lists of details. All text links, media links and thumbnails are active linked. As stated in . . .
. . .
individual Re-indexing, the periodical Re-indexer could be started and aborted in the "Options" menu of each site. 2.5 Preferred Re-indexing Each new URL added to the Admin backend, could be supplied with a priority level. This level will be used by the option 'Re-index only preferred sites'. Level 1 will be interpreted as most important, while . . .
. . .
less all index results will be stored in log files in sub folder /admin/log/ The names of the log files look like: db2_100524-21.47.56_1.html (log file of first thread) db2_100524-21.48.12_2.html (log file of second thread) and is build by the following items: db2 - Number of database. 100524 - Date (May 24, 2010) 21.47.56 - Time when this thread . . .
. . .
spider.php -new1 php spider.php -new2 etc. The IDs will be added to the name of the corresponding log files like: db2_100524-21.47.56_ID1.html (log file of first thread) db2_100524-21.48.12_ID2.html (log file of second thread) IDs could be defined by personal requirements, but the limitations for file names with respect to the OS should be taken . . .
. . .
but will not erase the content of all the other tables. So the check whether the content of a page has changed (MD5sum) is still available for a fast re-index procedure. Once prepared, multithreaded re-indexing could be invoked by starting several threads and adding individual IDs to the option parameter like: php spider.php -erased1 php . . .
. . .
4.4 Ignoring parts of a page Sphider-plus includes an option to exclude parts of pages from being indexed. Thi 775e s can for example be used to prevent search result flooding when certain keywords appear on certain part in most pages (like a header, footer or a menu). Any part of a page between <! sphider_noindex > and <! . . .
. . .
<! sphider_noindex > and <! /sphider_noindex > tags is not indexed, however links in it are followed. 4.5 Ignoring parts of a page by <div id='abc'> Ignoring parts of a page by the <! sphider_noindex > tags requires direct access to the page, because the tags need to be added (edited) to the page. A more flexible method, . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ menu[0-5] / 4.6 Indexing only parts of a page by <div id='abc'> If enabled in Admin settings, the values as defined in the list-file /include/common/divs_use.txt will be used to index only the content between <div . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ table[0-5] / 4.7 Ignore HTML elements defined by <tagname> . . </tagname> This option is foreseen to cooperate with the new HTML5 elements like section, nav, aside, hgroup, article, header, footer, etc. HTML elements . . .
. . .
contain a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */nav[0-5]/ Please keep in mind that element names placed in /include/common/elements_not.txt will be processed case-sensitive. 4.8 Index only HTML elements defined by <tagname> . . </tagname> This is the vice versa . . .
. . .
of all the duplicate content URLs: http://www.example.com/product.php?item=swedish-fish&category=gummy-candy http://www.example.com/product.php?item=swedish-fish&trackingid=1234&sessionid=5678 and Sphider-plus will understand that the duplicates all refer to the canonical URL: http://www.example.com/product.php?item=swedish-fish. The . . .
. . .
charset are supported by the ConvertCharset function and will be used to convert text into UTF-8 Unicode: WINDOWS windows-1250 - Central Europe windows-1251 - Cyrillic windows-1252 - Latin I windows-1253 - Greek windows-1254 - Turkish windows-1255 - Hebrew windows-1256 - Arabic windows-1257 - Baltic windows-1258 - Viet Nam cp874 - Thai - this . . .
. . .
- Baltic windows-1258 - Viet Nam cp874 - Thai - this file is also for DOS DOS cp437 - Latin US cp737 - Greek cp775 - BaltRim cp850 - Latin1 cp852 - Latin2 cp855 - Cyrylic cp857 - Turkish cp860 - Portuguese cp861 - Iceland cp862 - Hebrew cp863 - Canada cp864 - Arabic cp865 - Nordic cp866 - Cyrylic Russian (this is the one, used in IE . . .
. . .
IE Cyrillic (DOS) ) cp869 - Greek2 MAC (Apple) x-mac-cyrillic x-mac-greek x-mac-icelandic x-mac-ce x-mac-roman ISO iso-8859-1 iso-8859-2 iso-8859-3 iso-8859-4 iso-8859-5 iso-8859-6 iso-8859-7 iso-8859-8 iso-8859-9 iso-8859-10 iso-8859-11 iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 . . .
. . .
iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 cp500 cp856 cp875 cp1006 cp1026 koi8-r (Cyrillic) koi8-u (Cyrillic Ukrainian) nextstep us-ascii us-ascii-quotes DSP implementation for NeXT stdenc symbol zdingbat And specially for old Polish programs: mazovia This list is to be read . . .
. . .
Access denied; you need the RELOAD privilege. . . Top 10. Enable real-time output of logging data Up to version 1.5 of Sphider-plus, during index / re-index there was no printout available because: - Several servers, especially on Win32, buffer the output from the script until it terminates before transmitting the results to the browser. - . . .
. . .
is seen. - Some versions of Microsoft Internet Explorer only start to display the page after they have received 256 bytes of output. As progress was not presented during index / re- index procedure, waiting for results became a pain in the neck. Selectable in Admin setting together with the update interval (1 - 10 seconds), AJAX technology was . . .
. . .
characters in front of words Warning: This option should be used with special care and not be activated for non ISO-8859 charsets. Some special characters as part of the word ending might be erased by accidental. Top 13. Media search for images, audio streams and videos 13.1 Media indexing Index of media files is enabled by separated Admin . . .
. . .
and 'Found at' - Total clicks - Last clicked - Query input 'Indexed Image Thumbnails' presenting: - Thumbnail 150 x 100 pixel - Image details like title, filename size of original image, link- and thumb-id - Option to delete the thumbnail In order to open the media files all tables contain active links. Media results are also stored in . . .
. . .
Settings' menu of the admin backend: First option: For results of XML product feeds, present the text extract of 350 characters as part of product attributes. If this option is not activated, all products of the XML product feed will be presented in search results, and the hits of the query string will be highlighted in all involved products. . . .
. . .
content of the database tables (those with the same table prefix) will be destroyed by the restore procedure. 16.5 Copy & Move This section of the Database Management will present only those databases, which are correctly configured, assigned and do have a set of installed tables as described in chapter Definition and configuration. This . . .
. . .
results. Never the less to find these x results, the complete database had to been browsed. Starting with version 2.5, an additional clean option is offered as part of the Admin backend. Main advantage of this option is a significant reduction of the search time for any query, because the content of the db could be limited to offer only x . . .
. . .
<link>http://www.abc.de/images/warp.gif</link> <title>warp.gif</title> <x_size>635</x_size> <y_size>98</y_size> </media_result> <media_result> <num>2</num> <type>audio</type> <url>http://www.abc.de/index.php</url> . . .
. . .
the following content: {"query":"colour","ip":"::1","host_name":"guard_007-hoster","query_time":"2016-03-18 10:56:23 AM","consumed":0.016,"total_results":2,"num_of_results":2,"from":1,"to":2,"text_results":[{"num":1,"weight":"100.0","url":"http:\/\/www.english.le-piaggie.info\/Info_eng.pdf","title":" Info_eng","fulltxt":" . . . . . .
. . .
placed on the ground floor, today's big kitchen with the heating fireplace for open and close mode of . . . ","page_size":"5,661.6kb","domain_name":"www.english.le-piaggie.info"},{"num":2,"weight":"50.0","url":"http:\/\/www.english.le-piaggie.info\/html\/description.html","title":" Description","fulltxt":" . . . cable. Additional active . . .
. . .
D.O.C.G. as far as to the Pratomagno mountains 160 olive-trees Detached house, about 800 m to reach the . . . ","page_size":"26.4 kb","domain_name":"www.english.le-piaggie.info"}]} Top . . .


5a2 All required information. Introduction Release and Legal Info Installation Documentation Change Log [ Documentation Summary ] Preamble: The info presented here is valid only for the latest release of Sphider-plus. At present version 4.2024a published February 12, 2024 is the actual release. 1. Settings, customizing and statistics 2. Indexing . . .
. . .
2. Indexing 2.1 Various options 2.2 Allow other hosts in same domain 2.3 Word stemming 2.4 Periodical Re-indexing 2.5 Preferred indexing 2.6 Multithreaded indexing 2.7 Create thumbnails during index procedure 2.8 Prevent indexing of known malware and pishing pages 2.9 Follow and create sitemap files 2.10 Use private sitemap instead of global . . .
. . .
Sitemap file 3. Using the indexer from command line 3.1 All options 3.2 Multithreaded indexing 4. Keeping pages, words and files from being indexed 4.1 robots.txt 4.2 Must include / must not include string list 4.3 Ignoring links 4.4 Ignoring parts of a page by <! sphider_noindex > 4.5 Ignoring parts of a page by <div id='abc'> . . .
. . .
charset' 6. Search modes 6.1 Search with wildcards * 6.2 Strict search ! 6.3 Tolerant search 6.4 Link search 6.5 Media search 6.6 Search only in one domain 6.7 Search in categories 6.8 Greek language support 6.9 Block queries 7. Chronological order for result listing 7.1 Text result listing 7.2 Media result listing 8. PDF converter 9. Clean . . .
. . .
of logging data 11. Error messages and Debug mode 12. Delete secondary characters 13. Media search for images, audio streams and videos 13.1 Media indexing 13.2 Not supported media content 13.3 Search for media content 13.4 Statistics for media content 14. Feed support 14.1 XML product feeds 14.2 RDF, RSD, RSS and Atom feeds 15. Result . . .
. . .
Overview 16.2 Definition and configuration 16.3 Activate / disable database 16.4 Backup & Restore of databases 16.5 Copy & and Move 16.6 Enhancing functionality of multiple database support 17. Search in categories 17.1 Hierachical structure 17.2 Parallel structure 18. User suggested sites 19. Vulnerability protection 19.1 Prevent queries . . .
. . .
templates 22.2 Embed the search engine into existing HTML code 22.3 The different style sheet files 23. JSON, XML and RSS result output [ Documentation ] 1. Settings, customizing and statistics If you want to change settings, behavior and design of Sphider-plus, you can do so by means of the Admin interface. There is a wide range of . . .
. . .
interface. There is a wide range of settings foreseen for Sphider-plus. Separated into different submenus like: Sites: - Add Site - Index only the new - Re-index all - Re-index only preferred URLs - Erase Re-index (available also for individual URLs) - Import/export URL list - Approve sites - Banned domains Categories: - Add, edit, delete - . . .
. . .
URL list - Approve sites - Banned domains Categories: - Add, edit, delete - Create new subcategory under Index: - Basic indexing options - Advanced options Clean: - Clean keywords not associated with any link - Clean links not associated with any site - Clean Category table not associated with any site - Clean Media links - Clear Temp table . . .
. . .
- Clear Search log - Clear 'Most Popular Page Links' log - Clear 'Most Popular Media Links' log - Clear Spider log, separate and bulk delete - Clear Thumbnail images, separate and bulk delete - Clear Text cache - Clear Media cache - Clear IDS log file - Clear flood attempts log file - Clear all entries in addurl or banned table - Truncate all . . .
. . .
flood attempts log file - Clear all entries in addurl or banned table - Truncate all tables in database Settings: - General Settings - Index Log Settings - Spider Settings - Search Settings - Order of Result listing - Suggest Options - Page Indexing Weights Database: - Configure up to 5 databases with unlimited number of table sets - Activate . . .
. . .
Database: - Configure up to 5 databases with unlimited number of table sets - Activate separately for 'Admin', 'Search' user and 'Suggest URL user' - Backup / Restore - Copy / Move - Optimize Templates: In order to enable customer's integration of Sphider-plus into existing sites, HTML templates are prepared for Search form Text result . . .
. . .
Search form Text result listing Media result listing Most popular queries etc. Three different designs are offered, which may be selected in submenu 'Settings'. If the layout does not fit the design of your site (which is normal), you may create new designs and modify the appropriate file /templates/My_template/adminstyle.css . . .
. . .
the appropriate file /templates/My_template/adminstyle.css /templates/My_template/userstyle.css Statistics output: - Top keywords (Top 50 with hit counter). - All indexed thumbnails w 3e80 ith ID3 and EXIF info. - Larges pages offering link URL and file size. - Most Popular Searches for text links offering: Link addr., total clicks, last . . .
. . .
Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Searches for media links offering: Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Links (click counter). - Search log offering: Query, Results, Queried at, Time taken, User IP, Country, Host name (Latest100) - Index log offering: . . .
. . .
Country, Host name (Latest100) - Index log offering: File-name, index date and delete option - sitemap log offering: sitemap.xml output sitemap list offering file/page suffixes - IDS log offering: IP, host, query, impact, involved tags, date and time of intrusion. - Flood attempts log offering: IP, query, date and time of flood attempt. - Auto . . .
. . .
attempts log offering: IP, query, date and time of flood attempt. - Auto Re-index log file - Server info offering: Server software, environment, MySQL, PDF-converter, image functions, php.ini file PHP integration, PHP security info. Each item holding lists of details. All text links, media links and thumbnails are active linked. As stated in . . .
. . .
individual Re-indexing, the periodical Re-indexer could be started and aborted in the "Options" menu of each site. 2.5 Preferred Re-indexing Each new URL added to the Admin backend, could be supplied with a priority level. This level will be used by the option 'Re-index only preferred sites'. Level 1 will be interpreted as most important, while . . .
. . .
less all index results will be stored in log files in sub folder /admin/log/ The names of the log files look like: db2_100524-21.47.56_1.html (log file of first thread) db2_100524-21.48.12_2.html (log file of second thread) and is build by the following items: db2 - Number of database. 100524 - Date (May 24, 2010) 21.47.56 - Time when this thread . . .
. . .
spider.php -new1 php spider.php -new2 etc. The IDs will be added to the name of the corresponding log files like: db2_100524-21.47.56_ID1.html (log file of first thread) db2_100524-21.48.12_ID2.html (log file of second thread) IDs could be defined by personal requirements, but the limitations for file names with respect to the OS should be taken . . .
. . .
but will not erase the content of all the other tables. So the check whether the content of a page has changed (MD5sum) is still available for a fast re-index procedure. Once prepared, multithreaded re-indexing could be invoked by starting several threads and adding individual IDs to the option parameter like: php spider.php -erased1 php . . .
. . .
4.4 Ignoring parts of a page Sphider-plus includes an option to exclude parts of pages from being indexed. Thi 775e s can for example be used to prevent search result flooding when certain keywords appear on certain part in most pages (like a header, footer or a menu). Any part of a page between <! sphider_noindex > and <! . . .
. . .
<! sphider_noindex > and <! /sphider_noindex > tags is not indexed, however links in it are followed. 4.5 Ignoring parts of a page by <div id='abc'> Ignoring parts of a page by the <! sphider_noindex > tags requires direct access to the page, because the tags need to be added (edited) to the page. A more flexible method, . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ menu[0-5] / 4.6 Indexing only parts of a page by <div id='abc'> If enabled in Admin settings, the values as defined in the list-file /include/common/divs_use.txt will be used to index only the content between <div . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ table[0-5] / 4.7 Ignore HTML elements defined by <tagname> . . </tagname> This option is foreseen to cooperate with the new HTML5 elements like section, nav, aside, hgroup, article, header, footer, etc. HTML elements . . .
. . .
contain a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */nav[0-5]/ Please keep in mind that element names placed in /include/common/elements_not.txt will be processed case-sensitive. 4.8 Index only HTML elements defined by <tagname> . . </tagname> This is the vice versa . . .
. . .
of all the duplicate content URLs: http://www.example.com/product.php?item=swedish-fish&category=gummy-candy http://www.example.com/product.php?item=swedish-fish&trackingid=1234&sessionid=5678 and Sphider-plus will understand that the duplicates all refer to the canonical URL: http://www.example.com/product.php?item=swedish-fish. The . . .
. . .
charset are supported by the ConvertCharset function and will be used to convert text into UTF-8 Unicode: WINDOWS windows-1250 - Central Europe windows-1251 - Cyrillic windows-1252 - Latin I windows-1253 - Greek windows-1254 - Turkish windows-1255 - Hebrew windows-1256 - Arabic windows-1257 - Baltic windows-1258 - Viet Nam cp874 - Thai - this . . .
. . .
- Baltic windows-1258 - Viet Nam cp874 - Thai - this file is also for DOS DOS cp437 - Latin US cp737 - Greek cp775 - BaltRim cp850 - Latin1 cp852 - Latin2 cp855 - Cyrylic cp857 - Turkish cp860 - Portuguese cp861 - Iceland cp862 - Hebrew cp863 - Canada cp864 - Arabic cp865 - Nordic cp866 - Cyrylic Russian (this is the one, used in IE . . .
. . .
IE Cyrillic (DOS) ) cp869 - Greek2 MAC (Apple) x-mac-cyrillic x-mac-greek x-mac-icelandic x-mac-ce x-mac-roman ISO iso-8859-1 iso-8859-2 iso-8859-3 iso-8859-4 iso-8859-5 iso-8859-6 iso-8859-7 iso-8859-8 iso-8859-9 iso-8859-10 iso-8859-11 iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 . . .
. . .
iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 cp500 cp856 cp875 cp1006 cp1026 koi8-r (Cyrillic) koi8-u (Cyrillic Ukrainian) nextstep us-ascii us-ascii-quotes DSP implementation for NeXT stdenc symbol zdingbat And specially for old Polish programs: mazovia This list is to be read . . .
. . .
Access denied; you need the RELOAD privilege. . . Top 10. Enable real-time output of logging data Up to version 1.5 of Sphider-plus, during index / re-index there was no printout available because: - Several servers, especially on Win32, buffer the output from the script until it terminates before transmitting the results to the browser. - . . .
. . .
is seen. - Some versions of Microsoft Internet Explorer only start to display the page after they have received 256 bytes of output. As progress was not presented during index / re- index procedure, waiting for results became a pain in the neck. Selectable in Admin setting together with the update interval (1 - 10 seconds), AJAX technology was . . .
. . .
characters in front of words Warning: This option should be used with special care and not be activated for non ISO-8859 charsets. Some special characters as part of the word ending might be erased by accidental. Top 13. Media search for images, audio streams and videos 13.1 Media indexing Index of media files is enabled by separated Admin . . .
. . .
and 'Found at' - Total clicks - Last clicked - Query input 'Indexed Image Thumbnails' presenting: - Thumbnail 150 x 100 pixel - Image details like title, filename size of original image, link- and thumb-id - Option to delete the thumbnail In order to open the media files all tables contain active links. Media results are also stored in . . .
. . .
Settings' menu of the admin backend: First option: For results of XML product feeds, present the text extract of 350 characters as part of product attributes. If this option is not activated, all products of the XML product feed will be presented in search results, and the hits of the query string will be highlighted in all involved products. . . .
. . .
content of the database tables (those with the same table prefix) will be destroyed by the restore procedure. 16.5 Copy & Move This section of the Database Management will present only those databases, which are correctly configured, assigned and do have a set of installed tables as described in chapter Definition and configuration. This . . .
. . .
results. Never the less to find these x results, the complete database had to been browsed. Starting with version 2.5, an additional clean option is offered as part of the Admin backend. Main advantage of this option is a significant reduction of the search time for any query, because the content of the db could be limited to offer only x . . .
. . .
<link>http://www.abc.de/images/warp.gif</link> <title>warp.gif</title> <x_size>635</x_size> <y_size>98</y_size> </media_result> <media_result> <num>2</num> <type>audio</type> <url>http://www.abc.de/index.php</url> . . .
. . .
the following content: {"query":"colour","ip":"::1","host_name":"guard_007-hoster","query_time":"2016-03-18 10:56:23 AM","consumed":0.016,"total_results":2,"num_of_results":2,"from":1,"to":2,"text_results":[{"num":1,"weight":"100.0","url":"http:\/\/www.english.le-piaggie.info\/Info_eng.pdf","title":" Info_eng","fulltxt":" . . . . . .
. . .
placed on the ground floor, today's big kitchen with the heating fireplace for open and close mode of . . . ","page_size":"5,661.6kb","domain_name":"www.english.le-piaggie.info"},{"num":2,"weight":"50.0","url":"http:\/\/www.english.le-piaggie.info\/html\/description.html","title":" Description","fulltxt":" . . . cable. Additional active . . .
. . .
D.O.C.G. as far as to the Pratomagno mountains 160 olive-trees Detached house, about 800 m to reach the . . . ","page_size":"26.4 kb","domain_name":"www.english.le-piaggie.info"}]} Top . . .


5a2 All required information. Introduction Release and Legal Info Installation Documentation Change Log [ Documentation Summary ] Preamble: The info presented here is valid only for the latest release of Sphider-plus. At present version 4.2024a published February 12, 2024 is the actual release. 1. Settings, customizing and statistics 2. Indexing . . .
. . .
2. Indexing 2.1 Various options 2.2 Allow other hosts in same domain 2.3 Word stemming 2.4 Periodical Re-indexing 2.5 Preferred indexing 2.6 Multithreaded indexing 2.7 Create thumbnails during index procedure 2.8 Prevent indexing of known malware and pishing pages 2.9 Follow and create sitemap files 2.10 Use private sitemap instead of global . . .
. . .
Sitemap file 3. Using the indexer from command line 3.1 All options 3.2 Multithreaded indexing 4. Keeping pages, words and files from being indexed 4.1 robots.txt 4.2 Must include / must not include string list 4.3 Ignoring links 4.4 Ignoring parts of a page by <! sphider_noindex > 4.5 Ignoring parts of a page by <div id='abc'> . . .
. . .
charset' 6. Search modes 6.1 Search with wildcards * 6.2 Strict search ! 6.3 Tolerant search 6.4 Link search 6.5 Media search 6.6 Search only in one domain 6.7 Search in categories 6.8 Greek language support 6.9 Block queries 7. Chronological order for result listing 7.1 Text result listing 7.2 Media result listing 8. PDF converter 9. Clean . . .
. . .
of logging data 11. Error messages and Debug mode 12. Delete secondary characters 13. Media search for images, audio streams and videos 13.1 Media indexing 13.2 Not supported media content 13.3 Search for media content 13.4 Statistics for media content 14. Feed support 14.1 XML product feeds 14.2 RDF, RSD, RSS and Atom feeds 15. Result . . .
. . .
Overview 16.2 Definition and configuration 16.3 Activate / disable database 16.4 Backup & Restore of databases 16.5 Copy & and Move 16.6 Enhancing functionality of multiple database support 17. Search in categories 17.1 Hierachical structure 17.2 Parallel structure 18. User suggested sites 19. Vulnerability protection 19.1 Prevent queries . . .
. . .
templates 22.2 Embed the search engine into existing HTML code 22.3 The different style sheet files 23. JSON, XML and RSS result output [ Documentation ] 1. Settings, customizing and statistics If you want to change settings, behavior and design of Sphider-plus, you can do so by means of the Admin interface. There is a wide range of . . .
. . .
interface. There is a wide range of settings foreseen for Sphider-plus. Separated into different submenus like: Sites: - Add Site - Index only the new - Re-index all - Re-index only preferred URLs - Erase Re-index (available also for individual URLs) - Import/export URL list - Approve sites - Banned domains Categories: - Add, edit, delete - . . .
. . .
URL list - Approve sites - Banned domains Categories: - Add, edit, delete - Create new subcategory under Index: - Basic indexing options - Advanced options Clean: - Clean keywords not associated with any link - Clean links not associated with any site - Clean Category table not associated with any site - Clean Media links - Clear Temp table . . .
. . .
- Clear Search log - Clear 'Most Popular Page Links' log - Clear 'Most Popular Media Links' log - Clear Spider log, separate and bulk delete - Clear Thumbnail images, separate and bulk delete - Clear Text cache - Clear Media cache - Clear IDS log file - Clear flood attempts log file - Clear all entries in addurl or banned table - Truncate all . . .
. . .
flood attempts log file - Clear all entries in addurl or banned table - Truncate all tables in database Settings: - General Settings - Index Log Settings - Spider Settings - Search Settings - Order of Result listing - Suggest Options - Page Indexing Weights Database: - Configure up to 5 databases with unlimited number of table sets - Activate . . .
. . .
Database: - Configure up to 5 databases with unlimited number of table sets - Activate separately for 'Admin', 'Search' user and 'Suggest URL user' - Backup / Restore - Copy / Move - Optimize Templates: In order to enable customer's integration of Sphider-plus into existing sites, HTML templates are prepared for Search form Text result . . .
. . .
Search form Text result listing Media result listing Most popular queries etc. Three different designs are offered, which may be selected in submenu 'Settings'. If the layout does not fit the design of your site (which is normal), you may create new designs and modify the appropriate file /templates/My_template/adminstyle.css . . .
. . .
the appropriate file /templates/My_template/adminstyle.css /templates/My_template/userstyle.css Statistics output: - Top keywords (Top 50 with hit counter). - All indexed thumbnails w 3e80 ith ID3 and EXIF info. - Larges pages offering link URL and file size. - Most Popular Searches for text links offering: Link addr., total clicks, last . . .
. . .
Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Searches for media links offering: Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Links (click counter). - Search log offering: Query, Results, Queried at, Time taken, User IP, Country, Host name (Latest100) - Index log offering: . . .
. . .
Country, Host name (Latest100) - Index log offering: File-name, index date and delete option - sitemap log offering: sitemap.xml output sitemap list offering file/page suffixes - IDS log offering: IP, host, query, impact, involved tags, date and time of intrusion. - Flood attempts log offering: IP, query, date and time of flood attempt. - Auto . . .
. . .
attempts log offering: IP, query, date and time of flood attempt. - Auto Re-index log file - Server info offering: Server software, environment, MySQL, PDF-converter, image functions, php.ini file PHP integration, PHP security info. Each item holding lists of details. All text links, media links and thumbnails are active linked. As stated in . . .
. . .
individual Re-indexing, the periodical Re-indexer could be started and aborted in the "Options" menu of each site. 2.5 Preferred Re-indexing Each new URL added to the Admin backend, could be supplied with a priority level. This level will be used by the option 'Re-index only preferred sites'. Level 1 will be interpreted as most important, while . . .
. . .
less all index results will be stored in log files in sub folder /admin/log/ The names of the log files look like: db2_100524-21.47.56_1.html (log file of first thread) db2_100524-21.48.12_2.html (log file of second thread) and is build by the following items: db2 - Number of database. 100524 - Date (May 24, 2010) 21.47.56 - Time when this thread . . .
. . .
spider.php -new1 php spider.php -new2 etc. The IDs will be added to the name of the corresponding log files like: db2_100524-21.47.56_ID1.html (log file of first thread) db2_100524-21.48.12_ID2.html (log file of second thread) IDs could be defined by personal requirements, but the limitations for file names with respect to the OS should be taken . . .
. . .
but will not erase the content of all the other tables. So the check whether the content of a page has changed (MD5sum) is still available for a fast re-index procedure. Once prepared, multithreaded re-indexing could be invoked by starting several threads and adding individual IDs to the option parameter like: php spider.php -erased1 php . . .
. . .
4.4 Ignoring parts of a page Sphider-plus includes an option to exclude parts of pages from being indexed. Thi 775e s can for example be used to prevent search result flooding when certain keywords appear on certain part in most pages (like a header, footer or a menu). Any part of a page between <! sphider_noindex > and <! . . .
. . .
<! sphider_noindex > and <! /sphider_noindex > tags is not indexed, however links in it are followed. 4.5 Ignoring parts of a page by <div id='abc'> Ignoring parts of a page by the <! sphider_noindex > tags requires direct access to the page, because the tags need to be added (edited) to the page. A more flexible method, . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ menu[0-5] / 4.6 Indexing only parts of a page by <div id='abc'> If enabled in Admin settings, the values as defined in the list-file /include/common/divs_use.txt will be used to index only the content between <div . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ table[0-5] / 4.7 Ignore HTML elements defined by <tagname> . . </tagname> This option is foreseen to cooperate with the new HTML5 elements like section, nav, aside, hgroup, article, header, footer, etc. HTML elements . . .
. . .
contain a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */nav[0-5]/ Please keep in mind that element names placed in /include/common/elements_not.txt will be processed case-sensitive. 4.8 Index only HTML elements defined by <tagname> . . </tagname> This is the vice versa . . .
. . .
of all the duplicate content URLs: http://www.example.com/product.php?item=swedish-fish&category=gummy-candy http://www.example.com/product.php?item=swedish-fish&trackingid=1234&sessionid=5678 and Sphider-plus will understand that the duplicates all refer to the canonical URL: http://www.example.com/product.php?item=swedish-fish. The . . .
. . .
charset are supported by the ConvertCharset function and will be used to convert text into UTF-8 Unicode: WINDOWS windows-1250 - Central Europe windows-1251 - Cyrillic windows-1252 - Latin I windows-1253 - Greek windows-1254 - Turkish windows-1255 - Hebrew windows-1256 - Arabic windows-1257 - Baltic windows-1258 - Viet Nam cp874 - Thai - this . . .
. . .
- Baltic windows-1258 - Viet Nam cp874 - Thai - this file is also for DOS DOS cp437 - Latin US cp737 - Greek cp775 - BaltRim cp850 - Latin1 cp852 - Latin2 cp855 - Cyrylic cp857 - Turkish cp860 - Portuguese cp861 - Iceland cp862 - Hebrew cp863 - Canada cp864 - Arabic cp865 - Nordic cp866 - Cyrylic Russian (this is the one, used in IE . . .
. . .
IE Cyrillic (DOS) ) cp869 - Greek2 MAC (Apple) x-mac-cyrillic x-mac-greek x-mac-icelandic x-mac-ce x-mac-roman ISO iso-8859-1 iso-8859-2 iso-8859-3 iso-8859-4 iso-8859-5 iso-8859-6 iso-8859-7 iso-8859-8 iso-8859-9 iso-8859-10 iso-8859-11 iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 . . .
. . .
iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 cp500 cp856 cp875 cp1006 cp1026 koi8-r (Cyrillic) koi8-u (Cyrillic Ukrainian) nextstep us-ascii us-ascii-quotes DSP implementation for NeXT stdenc symbol zdingbat And specially for old Polish programs: mazovia This list is to be read . . .
. . .
Access denied; you need the RELOAD privilege. . . Top 10. Enable real-time output of logging data Up to version 1.5 of Sphider-plus, during index / re-index there was no printout available because: - Several servers, especially on Win32, buffer the output from the script until it terminates before transmitting the results to the browser. - . . .
. . .
is seen. - Some versions of Microsoft Internet Explorer only start to display the page after they have received 256 bytes of output. As progress was not presented during index / re- index procedure, waiting for results became a pain in the neck. Selectable in Admin setting together with the update interval (1 - 10 seconds), AJAX technology was . . .
. . .
characters in front of words Warning: This option should be used with special care and not be activated for non ISO-8859 charsets. Some special characters as part of the word ending might be erased by accidental. Top 13. Media search for images, audio streams and videos 13.1 Media indexing Index of media files is enabled by separated Admin . . .
. . .
and 'Found at' - Total clicks - Last clicked - Query input 'Indexed Image Thumbnails' presenting: - Thumbnail 150 x 100 pixel - Image details like title, filename size of original image, link- and thumb-id - Option to delete the thumbnail In order to open the media files all tables contain active links. Media results are also stored in . . .
. . .
Settings' menu of the admin backend: First option: For results of XML product feeds, present the text extract of 350 characters as part of product attributes. If this option is not activated, all products of the XML product feed will be presented in search results, and the hits of the query string will be highlighted in all involved products. . . .
. . .
content of the database tables (those with the same table prefix) will be destroyed by the restore procedure. 16.5 Copy & Move This section of the Database Management will present only those databases, which are correctly configured, assigned and do have a set of installed tables as described in chapter Definition and configuration. This . . .
. . .
results. Never the less to find these x results, the complete database had to been browsed. Starting with version 2.5, an additional clean option is offered as part of the Admin backend. Main advantage of this option is a significant reduction of the search time for any query, because the content of the db could be limited to offer only x . . .
. . .
<link>http://www.abc.de/images/warp.gif</link> <title>warp.gif</title> <x_size>635</x_size> <y_size>98</y_size> </media_result> <media_result> <num>2</num> <type>audio</type> <url>http://www.abc.de/index.php</url> . . .
. . .
the following content: {"query":"colour","ip":"::1","host_name":"guard_007-hoster","query_time":"2016-03-18 10:56:23 AM","consumed":0.016,"total_results":2,"num_of_results":2,"from":1,"to":2,"text_results":[{"num":1,"weight":"100.0","url":"http:\/\/www.english.le-piaggie.info\/Info_eng.pdf","title":" Info_eng","fulltxt":" . . . . . .
. . .
placed on the ground floor, today's big kitchen with the heating fireplace for open and close mode of . . . ","page_size":"5,661.6kb","domain_name":"www.english.le-piaggie.info"},{"num":2,"weight":"50.0","url":"http:\/\/www.english.le-piaggie.info\/html\/description.html","title":" Description","fulltxt":" . . . cable. Additional active . . .
. . .
D.O.C.G. as far as to the Pratomagno mountains 160 olive-trees Detached house, about 800 m to reach the . . . ","page_size":"26.4 kb","domain_name":"www.english.le-piaggie.info"}]} Top . . .


5a2 All required information. Introduction Release and Legal Info Installation Documentation Change Log [ Documentation Summary ] Preamble: The info presented here is valid only for the latest release of Sphider-plus. At present version 4.2024a published February 12, 2024 is the actual release. 1. Settings, customizing and statistics 2. Indexing . . .
. . .
2. Indexing 2.1 Various options 2.2 Allow other hosts in same domain 2.3 Word stemming 2.4 Periodical Re-indexing 2.5 Preferred indexing 2.6 Multithreaded indexing 2.7 Create thumbnails during index procedure 2.8 Prevent indexing of known malware and pishing pages 2.9 Follow and create sitemap files 2.10 Use private sitemap instead of global . . .
. . .
Sitemap file 3. Using the indexer from command line 3.1 All options 3.2 Multithreaded indexing 4. Keeping pages, words and files from being indexed 4.1 robots.txt 4.2 Must include / must not include string list 4.3 Ignoring links 4.4 Ignoring parts of a page by <! sphider_noindex > 4.5 Ignoring parts of a page by <div id='abc'> . . .
. . .
charset' 6. Search modes 6.1 Search with wildcards * 6.2 Strict search ! 6.3 Tolerant search 6.4 Link search 6.5 Media search 6.6 Search only in one domain 6.7 Search in categories 6.8 Greek language support 6.9 Block queries 7. Chronological order for result listing 7.1 Text result listing 7.2 Media result listing 8. PDF converter 9. Clean . . .
. . .
of logging data 11. Error messages and Debug mode 12. Delete secondary characters 13. Media search for images, audio streams and videos 13.1 Media indexing 13.2 Not supported media content 13.3 Search for media content 13.4 Statistics for media content 14. Feed support 14.1 XML product feeds 14.2 RDF, RSD, RSS and Atom feeds 15. Result . . .
. . .
Overview 16.2 Definition and configuration 16.3 Activate / disable database 16.4 Backup & Restore of databases 16.5 Copy & and Move 16.6 Enhancing functionality of multiple database support 17. Search in categories 17.1 Hierachical structure 17.2 Parallel structure 18. User suggested sites 19. Vulnerability protection 19.1 Prevent queries . . .
. . .
templates 22.2 Embed the search engine into existing HTML code 22.3 The different style sheet files 23. JSON, XML and RSS result output [ Documentation ] 1. Settings, customizing and statistics If you want to change settings, behavior and design of Sphider-plus, you can do so by means of the Admin interface. There is a wide range of . . .
. . .
interface. There is a wide range of settings foreseen for Sphider-plus. Separated into different submenus like: Sites: - Add Site - Index only the new - Re-index all - Re-index only preferred URLs - Erase Re-index (available also for individual URLs) - Import/export URL list - Approve sites - Banned domains Categories: - Add, edit, delete - . . .
. . .
URL list - Approve sites - Banned domains Categories: - Add, edit, delete - Create new subcategory under Index: - Basic indexing options - Advanced options Clean: - Clean keywords not associated with any link - Clean links not associated with any site - Clean Category table not associated with any site - Clean Media links - Clear Temp table . . .
. . .
- Clear Search log - Clear 'Most Popular Page Links' log - Clear 'Most Popular Media Links' log - Clear Spider log, separate and bulk delete - Clear Thumbnail images, separate and bulk delete - Clear Text cache - Clear Media cache - Clear IDS log file - Clear flood attempts log file - Clear all entries in addurl or banned table - Truncate all . . .
. . .
flood attempts log file - Clear all entries in addurl or banned table - Truncate all tables in database Settings: - General Settings - Index Log Settings - Spider Settings - Search Settings - Order of Result listing - Suggest Options - Page Indexing Weights Database: - Configure up to 5 databases with unlimited number of table sets - Activate . . .
. . .
Database: - Configure up to 5 databases with unlimited number of table sets - Activate separately for 'Admin', 'Search' user and 'Suggest URL user' - Backup / Restore - Copy / Move - Optimize Templates: In order to enable customer's integration of Sphider-plus into existing sites, HTML templates are prepared for Search form Text result . . .
. . .
Search form Text result listing Media result listing Most popular queries etc. Three different designs are offered, which may be selected in submenu 'Settings'. If the layout does not fit the design of your site (which is normal), you may create new designs and modify the appropriate file /templates/My_template/adminstyle.css . . .
. . .
the appropriate file /templates/My_template/adminstyle.css /templates/My_template/userstyle.css Statistics output: - Top keywords (Top 50 with hit counter). - All indexed thumbnails w 3e80 ith ID3 and EXIF info. - Larges pages offering link URL and file size. - Most Popular Searches for text links offering: Link addr., total clicks, last . . .
. . .
Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Searches for media links offering: Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Links (click counter). - Search log offering: Query, Results, Queried at, Time taken, User IP, Country, Host name (Latest100) - Index log offering: . . .
. . .
Country, Host name (Latest100) - Index log offering: File-name, index date and delete option - sitemap log offering: sitemap.xml output sitemap list offering file/page suffixes - IDS log offering: IP, host, query, impact, involved tags, date and time of intrusion. - Flood attempts log offering: IP, query, date and time of flood attempt. - Auto . . .
. . .
attempts log offering: IP, query, date and time of flood attempt. - Auto Re-index log file - Server info offering: Server software, environment, MySQL, PDF-converter, image functions, php.ini file PHP integration, PHP security info. Each item holding lists of details. All text links, media links and thumbnails are active linked. As stated in . . .
. . .
individual Re-indexing, the periodical Re-indexer could be started and aborted in the "Options" menu of each site. 2.5 Preferred Re-indexing Each new URL added to the Admin backend, could be supplied with a priority level. This level will be used by the option 'Re-index only preferred sites'. Level 1 will be interpreted as most important, while . . .
. . .
less all index results will be stored in log files in sub folder /admin/log/ The names of the log files look like: db2_100524-21.47.56_1.html (log file of first thread) db2_100524-21.48.12_2.html (log file of second thread) and is build by the following items: db2 - Number of database. 100524 - Date (May 24, 2010) 21.47.56 - Time when this thread . . .
. . .
spider.php -new1 php spider.php -new2 etc. The IDs will be added to the name of the corresponding log files like: db2_100524-21.47.56_ID1.html (log file of first thread) db2_100524-21.48.12_ID2.html (log file of second thread) IDs could be defined by personal requirements, but the limitations for file names with respect to the OS should be taken . . .
. . .
but will not erase the content of all the other tables. So the check whether the content of a page has changed (MD5sum) is still available for a fast re-index procedure. Once prepared, multithreaded re-indexing could be invoked by starting several threads and adding individual IDs to the option parameter like: php spider.php -erased1 php . . .
. . .
4.4 Ignoring parts of a page Sphider-plus includes an option to exclude parts of pages from being indexed. Thi 775e s can for example be used to prevent search result flooding when certain keywords appear on certain part in most pages (like a header, footer or a menu). Any part of a page between <! sphider_noindex > and <! . . .
. . .
<! sphider_noindex > and <! /sphider_noindex > tags is not indexed, however links in it are followed. 4.5 Ignoring parts of a page by <div id='abc'> Ignoring parts of a page by the <! sphider_noindex > tags requires direct access to the page, because the tags need to be added (edited) to the page. A more flexible method, . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ menu[0-5] / 4.6 Indexing only parts of a page by <div id='abc'> If enabled in Admin settings, the values as defined in the list-file /include/common/divs_use.txt will be used to index only the content between <div . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ table[0-5] / 4.7 Ignore HTML elements defined by <tagname> . . </tagname> This option is foreseen to cooperate with the new HTML5 elements like section, nav, aside, hgroup, article, header, footer, etc. HTML elements . . .
. . .
contain a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */nav[0-5]/ Please keep in mind that element names placed in /include/common/elements_not.txt will be processed case-sensitive. 4.8 Index only HTML elements defined by <tagname> . . </tagname> This is the vice versa . . .
. . .
of all the duplicate content URLs: http://www.example.com/product.php?item=swedish-fish&category=gummy-candy http://www.example.com/product.php?item=swedish-fish&trackingid=1234&sessionid=5678 and Sphider-plus will understand that the duplicates all refer to the canonical URL: http://www.example.com/product.php?item=swedish-fish. The . . .
. . .
charset are supported by the ConvertCharset function and will be used to convert text into UTF-8 Unicode: WINDOWS windows-1250 - Central Europe windows-1251 - Cyrillic windows-1252 - Latin I windows-1253 - Greek windows-1254 - Turkish windows-1255 - Hebrew windows-1256 - Arabic windows-1257 - Baltic windows-1258 - Viet Nam cp874 - Thai - this . . .
. . .
- Baltic windows-1258 - Viet Nam cp874 - Thai - this file is also for DOS DOS cp437 - Latin US cp737 - Greek cp775 - BaltRim cp850 - Latin1 cp852 - Latin2 cp855 - Cyrylic cp857 - Turkish cp860 - Portuguese cp861 - Iceland cp862 - Hebrew cp863 - Canada cp864 - Arabic cp865 - Nordic cp866 - Cyrylic Russian (this is the one, used in IE . . .
. . .
IE Cyrillic (DOS) ) cp869 - Greek2 MAC (Apple) x-mac-cyrillic x-mac-greek x-mac-icelandic x-mac-ce x-mac-roman ISO iso-8859-1 iso-8859-2 iso-8859-3 iso-8859-4 iso-8859-5 iso-8859-6 iso-8859-7 iso-8859-8 iso-8859-9 iso-8859-10 iso-8859-11 iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 . . .
. . .
iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 cp500 cp856 cp875 cp1006 cp1026 koi8-r (Cyrillic) koi8-u (Cyrillic Ukrainian) nextstep us-ascii us-ascii-quotes DSP implementation for NeXT stdenc symbol zdingbat And specially for old Polish programs: mazovia This list is to be read . . .
. . .
Access denied; you need the RELOAD privilege. . . Top 10. Enable real-time output of logging data Up to version 1.5 of Sphider-plus, during index / re-index there was no printout available because: - Several servers, especially on Win32, buffer the output from the script until it terminates before transmitting the results to the browser. - . . .
. . .
is seen. - Some versions of Microsoft Internet Explorer only start to display the page after they have received 256 bytes of output. As progress was not presented during index / re- index procedure, waiting for results became a pain in the neck. Selectable in Admin setting together with the update interval (1 - 10 seconds), AJAX technology was . . .
. . .
characters in front of words Warning: This option should be used with special care and not be activated for non ISO-8859 charsets. Some special characters as part of the word ending might be erased by accidental. Top 13. Media search for images, audio streams and videos 13.1 Media indexing Index of media files is enabled by separated Admin . . .
. . .
and 'Found at' - Total clicks - Last clicked - Query input 'Indexed Image Thumbnails' presenting: - Thumbnail 150 x 100 pixel - Image details like title, filename size of original image, link- and thumb-id - Option to delete the thumbnail In order to open the media files all tables contain active links. Media results are also stored in . . .
. . .
Settings' menu of the admin backend: First option: For results of XML product feeds, present the text extract of 350 characters as part of product attributes. If this option is not activated, all products of the XML product feed will be presented in search results, and the hits of the query string will be highlighted in all involved products. . . .
. . .
content of the database tables (those with the same table prefix) will be destroyed by the restore procedure. 16.5 Copy & Move This section of the Database Management will present only those databases, which are correctly configured, assigned and do have a set of installed tables as described in chapter Definition and configuration. This . . .
. . .
results. Never the less to find these x results, the complete database had to been browsed. Starting with version 2.5, an additional clean option is offered as part of the Admin backend. Main advantage of this option is a significant reduction of the search time for any query, because the content of the db could be limited to offer only x . . .
. . .
<link>http://www.abc.de/images/warp.gif</link> <title>warp.gif</title> <x_size>635</x_size> <y_size>98</y_size> </media_result> <media_result> <num>2</num> <type>audio</type> <url>http://www.abc.de/index.php</url> . . .
. . .
the following content: {"query":"colour","ip":"::1","host_name":"guard_007-hoster","query_time":"2016-03-18 10:56:23 AM","consumed":0.016,"total_results":2,"num_of_results":2,"from":1,"to":2,"text_results":[{"num":1,"weight":"100.0","url":"http:\/\/www.english.le-piaggie.info\/Info_eng.pdf","title":" Info_eng","fulltxt":" . . . . . .
. . .
placed on the ground floor, today's big kitchen with the heating fireplace for open and close mode of . . . ","page_size":"5,661.6kb","domain_name":"www.english.le-piaggie.info"},{"num":2,"weight":"50.0","url":"http:\/\/www.english.le-piaggie.info\/html\/description.html","title":" Description","fulltxt":" . . . cable. Additional active . . .
. . .
D.O.C.G. as far as to the Pratomagno mountains 160 olive-trees Detached house, about 800 m to reach the . . . ","page_size":"26.4 kb","domain_name":"www.english.le-piaggie.info"}]} Top . . .


5a2 All required information. Introduction Release and Legal Info Installation Documentation Change Log [ Documentation Summary ] Preamble: The info presented here is valid only for the latest release of Sphider-plus. At present version 4.2024a published February 12, 2024 is the actual release. 1. Settings, customizing and statistics 2. Indexing . . .
. . .
2. Indexing 2.1 Various options 2.2 Allow other hosts in same domain 2.3 Word stemming 2.4 Periodical Re-indexing 2.5 Preferred indexing 2.6 Multithreaded indexing 2.7 Create thumbnails during index procedure 2.8 Prevent indexing of known malware and pishing pages 2.9 Follow and create sitemap files 2.10 Use private sitemap instead of global . . .
. . .
Sitemap file 3. Using the indexer from command line 3.1 All options 3.2 Multithreaded indexing 4. Keeping pages, words and files from being indexed 4.1 robots.txt 4.2 Must include / must not include string list 4.3 Ignoring links 4.4 Ignoring parts of a page by <! sphider_noindex > 4.5 Ignoring parts of a page by <div id='abc'> . . .
. . .
charset' 6. Search modes 6.1 Search with wildcards * 6.2 Strict search ! 6.3 Tolerant search 6.4 Link search 6.5 Media search 6.6 Search only in one domain 6.7 Search in categories 6.8 Greek language support 6.9 Block queries 7. Chronological order for result listing 7.1 Text result listing 7.2 Media result listing 8. PDF converter 9. Clean . . .
. . .
of logging data 11. Error messages and Debug mode 12. Delete secondary characters 13. Media search for images, audio streams and videos 13.1 Media indexing 13.2 Not supported media content 13.3 Search for media content 13.4 Statistics for media content 14. Feed support 14.1 XML product feeds 14.2 RDF, RSD, RSS and Atom feeds 15. Result . . .
. . .
Overview 16.2 Definition and configuration 16.3 Activate / disable database 16.4 Backup & Restore of databases 16.5 Copy & and Move 16.6 Enhancing functionality of multiple database support 17. Search in categories 17.1 Hierachical structure 17.2 Parallel structure 18. User suggested sites 19. Vulnerability protection 19.1 Prevent queries . . .
. . .
templates 22.2 Embed the search engine into existing HTML code 22.3 The different style sheet files 23. JSON, XML and RSS result output [ Documentation ] 1. Settings, customizing and statistics If you want to change settings, behavior and design of Sphider-plus, you can do so by means of the Admin interface. There is a wide range of . . .
. . .
interface. There is a wide range of settings foreseen for Sphider-plus. Separated into different submenus like: Sites: - Add Site - Index only the new - Re-index all - Re-index only preferred URLs - Erase Re-index (available also for individual URLs) - Import/export URL list - Approve sites - Banned domains Categories: - Add, edit, delete - . . .
. . .
URL list - Approve sites - Banned domains Categories: - Add, edit, delete - Create new subcategory under Index: - Basic indexing options - Advanced options Clean: - Clean keywords not associated with any link - Clean links not associated with any site - Clean Category table not associated with any site - Clean Media links - Clear Temp table . . .
. . .
- Clear Search log - Clear 'Most Popular Page Links' log - Clear 'Most Popular Media Links' log - Clear Spider log, separate and bulk delete - Clear Thumbnail images, separate and bulk delete - Clear Text cache - Clear Media cache - Clear IDS log file - Clear flood attempts log file - Clear all entries in addurl or banned table - Truncate all . . .
. . .
flood attempts log file - Clear all entries in addurl or banned table - Truncate all tables in database Settings: - General Settings - Index Log Settings - Spider Settings - Search Settings - Order of Result listing - Suggest Options - Page Indexing Weights Database: - Configure up to 5 databases with unlimited number of table sets - Activate . . .
. . .
Database: - Configure up to 5 databases with unlimited number of table sets - Activate separately for 'Admin', 'Search' user and 'Suggest URL user' - Backup / Restore - Copy / Move - Optimize Templates: In order to enable customer's integration of Sphider-plus into existing sites, HTML templates are prepared for Search form Text result . . .
. . .
Search form Text result listing Media result listing Most popular queries etc. Three different designs are offered, which may be selected in submenu 'Settings'. If the layout does not fit the design of your site (which is normal), you may create new designs and modify the appropriate file /templates/My_template/adminstyle.css . . .
. . .
the appropriate file /templates/My_template/adminstyle.css /templates/My_template/userstyle.css Statistics output: - Top keywords (Top 50 with hit counter). - All indexed thumbnails w 3e80 ith ID3 and EXIF info. - Larges pages offering link URL and file size. - Most Popular Searches for text links offering: Link addr., total clicks, last . . .
. . .
Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Searches for media links offering: Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Links (click counter). - Search log offering: Query, Results, Queried at, Time taken, User IP, Country, Host name (Latest100) - Index log offering: . . .
. . .
Country, Host name (Latest100) - Index log offering: File-name, index date and delete option - sitemap log offering: sitemap.xml output sitemap list offering file/page suffixes - IDS log offering: IP, host, query, impact, involved tags, date and time of intrusion. - Flood attempts log offering: IP, query, date and time of flood attempt. - Auto . . .
. . .
attempts log offering: IP, query, date and time of flood attempt. - Auto Re-index log file - Server info offering: Server software, environment, MySQL, PDF-converter, image functions, php.ini file PHP integration, PHP security info. Each item holding lists of details. All text links, media links and thumbnails are active linked. As stated in . . .
. . .
individual Re-indexing, the periodical Re-indexer could be started and aborted in the "Options" menu of each site. 2.5 Preferred Re-indexing Each new URL added to the Admin backend, could be supplied with a priority level. This level will be used by the option 'Re-index only preferred sites'. Level 1 will be interpreted as most important, while . . .
. . .
less all index results will be stored in log files in sub folder /admin/log/ The names of the log files look like: db2_100524-21.47.56_1.html (log file of first thread) db2_100524-21.48.12_2.html (log file of second thread) and is build by the following items: db2 - Number of database. 100524 - Date (May 24, 2010) 21.47.56 - Time when this thread . . .
. . .
spider.php -new1 php spider.php -new2 etc. The IDs will be added to the name of the corresponding log files like: db2_100524-21.47.56_ID1.html (log file of first thread) db2_100524-21.48.12_ID2.html (log file of second thread) IDs could be defined by personal requirements, but the limitations for file names with respect to the OS should be taken . . .
. . .
but will not erase the content of all the other tables. So the check whether the content of a page has changed (MD5sum) is still available for a fast re-index procedure. Once prepared, multithreaded re-indexing could be invoked by starting several threads and adding individual IDs to the option parameter like: php spider.php -erased1 php . . .
. . .
4.4 Ignoring parts of a page Sphider-plus includes an option to exclude parts of pages from being indexed. Thi 775e s can for example be used to prevent search result flooding when certain keywords appear on certain part in most pages (like a header, footer or a menu). Any part of a page between <! sphider_noindex > and <! . . .
. . .
<! sphider_noindex > and <! /sphider_noindex > tags is not indexed, however links in it are followed. 4.5 Ignoring parts of a page by <div id='abc'> Ignoring parts of a page by the <! sphider_noindex > tags requires direct access to the page, because the tags need to be added (edited) to the page. A more flexible method, . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ menu[0-5] / 4.6 Indexing only parts of a page by <div id='abc'> If enabled in Admin settings, the values as defined in the list-file /include/common/divs_use.txt will be used to index only the content between <div . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ table[0-5] / 4.7 Ignore HTML elements defined by <tagname> . . </tagname> This option is foreseen to cooperate with the new HTML5 elements like section, nav, aside, hgroup, article, header, footer, etc. HTML elements . . .
. . .
contain a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */nav[0-5]/ Please keep in mind that element names placed in /include/common/elements_not.txt will be processed case-sensitive. 4.8 Index only HTML elements defined by <tagname> . . </tagname> This is the vice versa . . .
. . .
of all the duplicate content URLs: http://www.example.com/product.php?item=swedish-fish&category=gummy-candy http://www.example.com/product.php?item=swedish-fish&trackingid=1234&sessionid=5678 and Sphider-plus will understand that the duplicates all refer to the canonical URL: http://www.example.com/product.php?item=swedish-fish. The . . .
. . .
charset are supported by the ConvertCharset function and will be used to convert text into UTF-8 Unicode: WINDOWS windows-1250 - Central Europe windows-1251 - Cyrillic windows-1252 - Latin I windows-1253 - Greek windows-1254 - Turkish windows-1255 - Hebrew windows-1256 - Arabic windows-1257 - Baltic windows-1258 - Viet Nam cp874 - Thai - this . . .
. . .
- Baltic windows-1258 - Viet Nam cp874 - Thai - this file is also for DOS DOS cp437 - Latin US cp737 - Greek cp775 - BaltRim cp850 - Latin1 cp852 - Latin2 cp855 - Cyrylic cp857 - Turkish cp860 - Portuguese cp861 - Iceland cp862 - Hebrew cp863 - Canada cp864 - Arabic cp865 - Nordic cp866 - Cyrylic Russian (this is the one, used in IE . . .
. . .
IE Cyrillic (DOS) ) cp869 - Greek2 MAC (Apple) x-mac-cyrillic x-mac-greek x-mac-icelandic x-mac-ce x-mac-roman ISO iso-8859-1 iso-8859-2 iso-8859-3 iso-8859-4 iso-8859-5 iso-8859-6 iso-8859-7 iso-8859-8 iso-8859-9 iso-8859-10 iso-8859-11 iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 . . .
. . .
iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 cp500 cp856 cp875 cp1006 cp1026 koi8-r (Cyrillic) koi8-u (Cyrillic Ukrainian) nextstep us-ascii us-ascii-quotes DSP implementation for NeXT stdenc symbol zdingbat And specially for old Polish programs: mazovia This list is to be read . . .
. . .
Access denied; you need the RELOAD privilege. . . Top 10. Enable real-time output of logging data Up to version 1.5 of Sphider-plus, during index / re-index there was no printout available because: - Several servers, especially on Win32, buffer the output from the script until it terminates before transmitting the results to the browser. - . . .
. . .
is seen. - Some versions of Microsoft Internet Explorer only start to display the page after they have received 256 bytes of output. As progress was not presented during index / re- index procedure, waiting for results became a pain in the neck. Selectable in Admin setting together with the update interval (1 - 10 seconds), AJAX technology was . . .
. . .
characters in front of words Warning: This option should be used with special care and not be activated for non ISO-8859 charsets. Some special characters as part of the word ending might be erased by accidental. Top 13. Media search for images, audio streams and videos 13.1 Media indexing Index of media files is enabled by separated Admin . . .
. . .
and 'Found at' - Total clicks - Last clicked - Query input 'Indexed Image Thumbnails' presenting: - Thumbnail 150 x 100 pixel - Image details like title, filename size of original image, link- and thumb-id - Option to delete the thumbnail In order to open the media files all tables contain active links. Media results are also stored in . . .
. . .
Settings' menu of the admin backend: First option: For results of XML product feeds, present the text extract of 350 characters as part of product attributes. If this option is not activated, all products of the XML product feed will be presented in search results, and the hits of the query string will be highlighted in all involved products. . . .
. . .
content of the database tables (those with the same table prefix) will be destroyed by the restore procedure. 16.5 Copy & Move This section of the Database Management will present only those databases, which are correctly configured, assigned and do have a set of installed tables as described in chapter Definition and configuration. This . . .
. . .
results. Never the less to find these x results, the complete database had to been browsed. Starting with version 2.5, an additional clean option is offered as part of the Admin backend. Main advantage of this option is a significant reduction of the search time for any query, because the content of the db could be limited to offer only x . . .
. . .
<link>http://www.abc.de/images/warp.gif</link> <title>warp.gif</title> <x_size>635</x_size> <y_size>98</y_size> </media_result> <media_result> <num>2</num> <type>audio</type> <url>http://www.abc.de/index.php</url> . . .
. . .
the following content: {"query":"colour","ip":"::1","host_name":"guard_007-hoster","query_time":"2016-03-18 10:56:23 AM","consumed":0.016,"total_results":2,"num_of_results":2,"from":1,"to":2,"text_results":[{"num":1,"weight":"100.0","url":"http:\/\/www.english.le-piaggie.info\/Info_eng.pdf","title":" Info_eng","fulltxt":" . . . . . .
. . .
placed on the ground floor, today's big kitchen with the heating fireplace for open and close mode of . . . ","page_size":"5,661.6kb","domain_name":"www.english.le-piaggie.info"},{"num":2,"weight":"50.0","url":"http:\/\/www.english.le-piaggie.info\/html\/description.html","title":" Description","fulltxt":" . . . cable. Additional active . . .
. . .
D.O.C.G. as far as to the Pratomagno mountains 160 olive-trees Detached house, about 800 m to reach the . . . ","page_size":"26.4 kb","domain_name":"www.english.le-piaggie.info"}]} Top . . .


5a2 All required information. Introduction Release and Legal Info Installation Documentation Change Log [ Documentation Summary ] Preamble: The info presented here is valid only for the latest release of Sphider-plus. At present version 4.2024a published February 12, 2024 is the actual release. 1. Settings, customizing and statistics 2. Indexing . . .
. . .
2. Indexing 2.1 Various options 2.2 Allow other hosts in same domain 2.3 Word stemming 2.4 Periodical Re-indexing 2.5 Preferred indexing 2.6 Multithreaded indexing 2.7 Create thumbnails during index procedure 2.8 Prevent indexing of known malware and pishing pages 2.9 Follow and create sitemap files 2.10 Use private sitemap instead of global . . .
. . .
Sitemap file 3. Using the indexer from command line 3.1 All options 3.2 Multithreaded indexing 4. Keeping pages, words and files from being indexed 4.1 robots.txt 4.2 Must include / must not include string list 4.3 Ignoring links 4.4 Ignoring parts of a page by <! sphider_noindex > 4.5 Ignoring parts of a page by <div id='abc'> . . .
. . .
charset' 6. Search modes 6.1 Search with wildcards * 6.2 Strict search ! 6.3 Tolerant search 6.4 Link search 6.5 Media search 6.6 Search only in one domain 6.7 Search in categories 6.8 Greek language support 6.9 Block queries 7. Chronological order for result listing 7.1 Text result listing 7.2 Media result listing 8. PDF converter 9. Clean . . .
. . .
of logging data 11. Error messages and Debug mode 12. Delete secondary characters 13. Media search for images, audio streams and videos 13.1 Media indexing 13.2 Not supported media content 13.3 Search for media content 13.4 Statistics for media content 14. Feed support 14.1 XML product feeds 14.2 RDF, RSD, RSS and Atom feeds 15. Result . . .
. . .
Overview 16.2 Definition and configuration 16.3 Activate / disable database 16.4 Backup & Restore of databases 16.5 Copy & and Move 16.6 Enhancing functionality of multiple database support 17. Search in categories 17.1 Hierachical structure 17.2 Parallel structure 18. User suggested sites 19. Vulnerability protection 19.1 Prevent queries . . .
. . .
templates 22.2 Embed the search engine into existing HTML code 22.3 The different style sheet files 23. JSON, XML and RSS result output [ Documentation ] 1. Settings, customizing and statistics If you want to change settings, behavior and design of Sphider-plus, you can do so by means of the Admin interface. There is a wide range of . . .
. . .
interface. There is a wide range of settings foreseen for Sphider-plus. Separated into different submenus like: Sites: - Add Site - Index only the new - Re-index all - Re-index only preferred URLs - Erase Re-index (available also for individual URLs) - Import/export URL list - Approve sites - Banned domains Categories: - Add, edit, delete - . . .
. . .
URL list - Approve sites - Banned domains Categories: - Add, edit, delete - Create new subcategory under Index: - Basic indexing options - Advanced options Clean: - Clean keywords not associated with any link - Clean links not associated with any site - Clean Category table not associated with any site - Clean Media links - Clear Temp table . . .
. . .
- Clear Search log - Clear 'Most Popular Page Links' log - Clear 'Most Popular Media Links' log - Clear Spider log, separate and bulk delete - Clear Thumbnail images, separate and bulk delete - Clear Text cache - Clear Media cache - Clear IDS log file - Clear flood attempts log file - Clear all entries in addurl or banned table - Truncate all . . .
. . .
flood attempts log file - Clear all entries in addurl or banned table - Truncate all tables in database Settings: - General Settings - Index Log Settings - Spider Settings - Search Settings - Order of Result listing - Suggest Options - Page Indexing Weights Database: - Configure up to 5 databases with unlimited number of table sets - Activate . . .
. . .
Database: - Configure up to 5 databases with unlimited number of table sets - Activate separately for 'Admin', 'Search' user and 'Suggest URL user' - Backup / Restore - Copy / Move - Optimize Templates: In order to enable customer's integration of Sphider-plus into existing sites, HTML templates are prepared for Search form Text result . . .
. . .
Search form Text result listing Media result listing Most popular queries etc. Three different designs are offered, which may be selected in submenu 'Settings'. If the layout does not fit the design of your site (which is normal), you may create new designs and modify the appropriate file /templates/My_template/adminstyle.css . . .
. . .
the appropriate file /templates/My_template/adminstyle.css /templates/My_template/userstyle.css Statistics output: - Top keywords (Top 50 with hit counter). - All indexed thumbnails w 3e80 ith ID3 and EXIF info. - Larges pages offering link URL and file size. - Most Popular Searches for text links offering: Link addr., total clicks, last . . .
. . .
Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Searches for media links offering: Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Links (click counter). - Search log offering: Query, Results, Queried at, Time taken, User IP, Country, Host name (Latest100) - Index log offering: . . .
. . .
Country, Host name (Latest100) - Index log offering: File-name, index date and delete option - sitemap log offering: sitemap.xml output sitemap list offering file/page suffixes - IDS log offering: IP, host, query, impact, involved tags, date and time of intrusion. - Flood attempts log offering: IP, query, date and time of flood attempt. - Auto . . .
. . .
attempts log offering: IP, query, date and time of flood attempt. - Auto Re-index log file - Server info offering: Server software, environment, MySQL, PDF-converter, image functions, php.ini file PHP integration, PHP security info. Each item holding lists of details. All text links, media links and thumbnails are active linked. As stated in . . .
. . .
individual Re-indexing, the periodical Re-indexer could be started and aborted in the "Options" menu of each site. 2.5 Preferred Re-indexing Each new URL added to the Admin backend, could be supplied with a priority level. This level will be used by the option 'Re-index only preferred sites'. Level 1 will be interpreted as most important, while . . .
. . .
less all index results will be stored in log files in sub folder /admin/log/ The names of the log files look like: db2_100524-21.47.56_1.html (log file of first thread) db2_100524-21.48.12_2.html (log file of second thread) and is build by the following items: db2 - Number of database. 100524 - Date (May 24, 2010) 21.47.56 - Time when this thread . . .
. . .
spider.php -new1 php spider.php -new2 etc. The IDs will be added to the name of the corresponding log files like: db2_100524-21.47.56_ID1.html (log file of first thread) db2_100524-21.48.12_ID2.html (log file of second thread) IDs could be defined by personal requirements, but the limitations for file names with respect to the OS should be taken . . .
. . .
but will not erase the content of all the other tables. So the check whether the content of a page has changed (MD5sum) is still available for a fast re-index procedure. Once prepared, multithreaded re-indexing could be invoked by starting several threads and adding individual IDs to the option parameter like: php spider.php -erased1 php . . .
. . .
4.4 Ignoring parts of a page Sphider-plus includes an option to exclude parts of pages from being indexed. Thi 775e s can for example be used to prevent search result flooding when certain keywords appear on certain part in most pages (like a header, footer or a menu). Any part of a page between <! sphider_noindex > and <! . . .
. . .
<! sphider_noindex > and <! /sphider_noindex > tags is not indexed, however links in it are followed. 4.5 Ignoring parts of a page by <div id='abc'> Ignoring parts of a page by the <! sphider_noindex > tags requires direct access to the page, because the tags need to be added (edited) to the page. A more flexible method, . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ menu[0-5] / 4.6 Indexing only parts of a page by <div id='abc'> If enabled in Admin settings, the values as defined in the list-file /include/common/divs_use.txt will be used to index only the content between <div . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ table[0-5] / 4.7 Ignore HTML elements defined by <tagname> . . </tagname> This option is foreseen to cooperate with the new HTML5 elements like section, nav, aside, hgroup, article, header, footer, etc. HTML elements . . .
. . .
contain a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */nav[0-5]/ Please keep in mind that element names placed in /include/common/elements_not.txt will be processed case-sensitive. 4.8 Index only HTML elements defined by <tagname> . . </tagname> This is the vice versa . . .
. . .
of all the duplicate content URLs: http://www.example.com/product.php?item=swedish-fish&category=gummy-candy http://www.example.com/product.php?item=swedish-fish&trackingid=1234&sessionid=5678 and Sphider-plus will understand that the duplicates all refer to the canonical URL: http://www.example.com/product.php?item=swedish-fish. The . . .
. . .
charset are supported by the ConvertCharset function and will be used to convert text into UTF-8 Unicode: WINDOWS windows-1250 - Central Europe windows-1251 - Cyrillic windows-1252 - Latin I windows-1253 - Greek windows-1254 - Turkish windows-1255 - Hebrew windows-1256 - Arabic windows-1257 - Baltic windows-1258 - Viet Nam cp874 - Thai - this . . .
. . .
- Baltic windows-1258 - Viet Nam cp874 - Thai - this file is also for DOS DOS cp437 - Latin US cp737 - Greek cp775 - BaltRim cp850 - Latin1 cp852 - Latin2 cp855 - Cyrylic cp857 - Turkish cp860 - Portuguese cp861 - Iceland cp862 - Hebrew cp863 - Canada cp864 - Arabic cp865 - Nordic cp866 - Cyrylic Russian (this is the one, used in IE . . .
. . .
IE Cyrillic (DOS) ) cp869 - Greek2 MAC (Apple) x-mac-cyrillic x-mac-greek x-mac-icelandic x-mac-ce x-mac-roman ISO iso-8859-1 iso-8859-2 iso-8859-3 iso-8859-4 iso-8859-5 iso-8859-6 iso-8859-7 iso-8859-8 iso-8859-9 iso-8859-10 iso-8859-11 iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 . . .
. . .
iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 cp500 cp856 cp875 cp1006 cp1026 koi8-r (Cyrillic) koi8-u (Cyrillic Ukrainian) nextstep us-ascii us-ascii-quotes DSP implementation for NeXT stdenc symbol zdingbat And specially for old Polish programs: mazovia This list is to be read . . .
. . .
Access denied; you need the RELOAD privilege. . . Top 10. Enable real-time output of logging data Up to version 1.5 of Sphider-plus, during index / re-index there was no printout available because: - Several servers, especially on Win32, buffer the output from the script until it terminates before transmitting the results to the browser. - . . .
. . .
is seen. - Some versions of Microsoft Internet Explorer only start to display the page after they have received 256 bytes of output. As progress was not presented during index / re- index procedure, waiting for results became a pain in the neck. Selectable in Admin setting together with the update interval (1 - 10 seconds), AJAX technology was . . .
. . .
characters in front of words Warning: This option should be used with special care and not be activated for non ISO-8859 charsets. Some special characters as part of the word ending might be erased by accidental. Top 13. Media search for images, audio streams and videos 13.1 Media indexing Index of media files is enabled by separated Admin . . .
. . .
and 'Found at' - Total clicks - Last clicked - Query input 'Indexed Image Thumbnails' presenting: - Thumbnail 150 x 100 pixel - Image details like title, filename size of original image, link- and thumb-id - Option to delete the thumbnail In order to open the media files all tables contain active links. Media results are also stored in . . .
. . .
Settings' menu of the admin backend: First option: For results of XML product feeds, present the text extract of 350 characters as part of product attributes. If this option is not activated, all products of the XML product feed will be presented in search results, and the hits of the query string will be highlighted in all involved products. . . .
. . .
content of the database tables (those with the same table prefix) will be destroyed by the restore procedure. 16.5 Copy & Move This section of the Database Management will present only those databases, which are correctly configured, assigned and do have a set of installed tables as described in chapter Definition and configuration. This . . .
. . .
results. Never the less to find these x results, the complete database had to been browsed. Starting with version 2.5, an additional clean option is offered as part of the Admin backend. Main advantage of this option is a significant reduction of the search time for any query, because the content of the db could be limited to offer only x . . .
. . .
<link>http://www.abc.de/images/warp.gif</link> <title>warp.gif</title> <x_size>635</x_size> <y_size>98</y_size> </media_result> <media_result> <num>2</num> <type>audio</type> <url>http://www.abc.de/index.php</url> . . .
. . .
the following content: {"query":"colour","ip":"::1","host_name":"guard_007-hoster","query_time":"2016-03-18 10:56:23 AM","consumed":0.016,"total_results":2,"num_of_results":2,"from":1,"to":2,"text_results":[{"num":1,"weight":"100.0","url":"http:\/\/www.english.le-piaggie.info\/Info_eng.pdf","title":" Info_eng","fulltxt":" . . . . . .
. . .
placed on the ground floor, today's big kitchen with the heating fireplace for open and close mode of . . . ","page_size":"5,661.6kb","domain_name":"www.english.le-piaggie.info"},{"num":2,"weight":"50.0","url":"http:\/\/www.english.le-piaggie.info\/html\/description.html","title":" Description","fulltxt":" . . . cable. Additional active . . .
. . .
D.O.C.G. as far as to the Pratomagno mountains 160 olive-trees Detached house, about 800 m to reach the . . . ","page_size":"26.4 kb","domain_name":"www.english.le-piaggie.info"}]} Top . . .


5a2 All required information. Introduction Release and Legal Info Installation Documentation Change Log [ Documentation Summary ] Preamble: The info presented here is valid only for the latest release of Sphider-plus. At present version 4.2024a published February 12, 2024 is the actual release. 1. Settings, customizing and statistics 2. Indexing . . .
. . .
2. Indexing 2.1 Various options 2.2 Allow other hosts in same domain 2.3 Word stemming 2.4 Periodical Re-indexing 2.5 Preferred indexing 2.6 Multithreaded indexing 2.7 Create thumbnails during index procedure 2.8 Prevent indexing of known malware and pishing pages 2.9 Follow and create sitemap files 2.10 Use private sitemap instead of global . . .
. . .
Sitemap file 3. Using the indexer from command line 3.1 All options 3.2 Multithreaded indexing 4. Keeping pages, words and files from being indexed 4.1 robots.txt 4.2 Must include / must not include string list 4.3 Ignoring links 4.4 Ignoring parts of a page by <! sphider_noindex > 4.5 Ignoring parts of a page by <div id='abc'> . . .
. . .
charset' 6. Search modes 6.1 Search with wildcards * 6.2 Strict search ! 6.3 Tolerant search 6.4 Link search 6.5 Media search 6.6 Search only in one domain 6.7 Search in categories 6.8 Greek language support 6.9 Block queries 7. Chronological order for result listing 7.1 Text result listing 7.2 Media result listing 8. PDF converter 9. Clean . . .
. . .
of logging data 11. Error messages and Debug mode 12. Delete secondary characters 13. Media search for images, audio streams and videos 13.1 Media indexing 13.2 Not supported media content 13.3 Search for media content 13.4 Statistics for media content 14. Feed support 14.1 XML product feeds 14.2 RDF, RSD, RSS and Atom feeds 15. Result . . .
. . .
Overview 16.2 Definition and configuration 16.3 Activate / disable database 16.4 Backup & Restore of databases 16.5 Copy & and Move 16.6 Enhancing functionality of multiple database support 17. Search in categories 17.1 Hierachical structure 17.2 Parallel structure 18. User suggested sites 19. Vulnerability protection 19.1 Prevent queries . . .
. . .
templates 22.2 Embed the search engine into existing HTML code 22.3 The different style sheet files 23. JSON, XML and RSS result output [ Documentation ] 1. Settings, customizing and statistics If you want to change settings, behavior and design of Sphider-plus, you can do so by means of the Admin interface. There is a wide range of . . .
. . .
interface. There is a wide range of settings foreseen for Sphider-plus. Separated into different submenus like: Sites: - Add Site - Index only the new - Re-index all - Re-index only preferred URLs - Erase Re-index (available also for individual URLs) - Import/export URL list - Approve sites - Banned domains Categories: - Add, edit, delete - . . .
. . .
URL list - Approve sites - Banned domains Categories: - Add, edit, delete - Create new subcategory under Index: - Basic indexing options - Advanced options Clean: - Clean keywords not associated with any link - Clean links not associated with any site - Clean Category table not associated with any site - Clean Media links - Clear Temp table . . .
. . .
- Clear Search log - Clear 'Most Popular Page Links' log - Clear 'Most Popular Media Links' log - Clear Spider log, separate and bulk delete - Clear Thumbnail images, separate and bulk delete - Clear Text cache - Clear Media cache - Clear IDS log file - Clear flood attempts log file - Clear all entries in addurl or banned table - Truncate all . . .
. . .
flood attempts log file - Clear all entries in addurl or banned table - Truncate all tables in database Settings: - General Settings - Index Log Settings - Spider Settings - Search Settings - Order of Result listing - Suggest Options - Page Indexing Weights Database: - Configure up to 5 databases with unlimited number of table sets - Activate . . .
. . .
Database: - Configure up to 5 databases with unlimited number of table sets - Activate separately for 'Admin', 'Search' user and 'Suggest URL user' - Backup / Restore - Copy / Move - Optimize Templates: In order to enable customer's integration of Sphider-plus into existing sites, HTML templates are prepared for Search form Text result . . .
. . .
Search form Text result listing Media result listing Most popular queries etc. Three different designs are offered, which may be selected in submenu 'Settings'. If the layout does not fit the design of your site (which is normal), you may create new designs and modify the appropriate file /templates/My_template/adminstyle.css . . .
. . .
the appropriate file /templates/My_template/adminstyle.css /templates/My_template/userstyle.css Statistics output: - Top keywords (Top 50 with hit counter). - All indexed thumbnails w 3e80 ith ID3 and EXIF info. - Larges pages offering link URL and file size. - Most Popular Searches for text links offering: Link addr., total clicks, last . . .
. . .
Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Searches for media links offering: Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Links (click counter). - Search log offering: Query, Results, Queried at, Time taken, User IP, Country, Host name (Latest100) - Index log offering: . . .
. . .
Country, Host name (Latest100) - Index log offering: File-name, index date and delete option - sitemap log offering: sitemap.xml output sitemap list offering file/page suffixes - IDS log offering: IP, host, query, impact, involved tags, date and time of intrusion. - Flood attempts log offering: IP, query, date and time of flood attempt. - Auto . . .
. . .
attempts log offering: IP, query, date and time of flood attempt. - Auto Re-index log file - Server info offering: Server software, environment, MySQL, PDF-converter, image functions, php.ini file PHP integration, PHP security info. Each item holding lists of details. All text links, media links and thumbnails are active linked. As stated in . . .
. . .
individual Re-indexing, the periodical Re-indexer could be started and aborted in the "Options" menu of each site. 2.5 Preferred Re-indexing Each new URL added to the Admin backend, could be supplied with a priority level. This level will be used by the option 'Re-index only preferred sites'. Level 1 will be interpreted as most important, while . . .
. . .
less all index results will be stored in log files in sub folder /admin/log/ The names of the log files look like: db2_100524-21.47.56_1.html (log file of first thread) db2_100524-21.48.12_2.html (log file of second thread) and is build by the following items: db2 - Number of database. 100524 - Date (May 24, 2010) 21.47.56 - Time when this thread . . .
. . .
spider.php -new1 php spider.php -new2 etc. The IDs will be added to the name of the corresponding log files like: db2_100524-21.47.56_ID1.html (log file of first thread) db2_100524-21.48.12_ID2.html (log file of second thread) IDs could be defined by personal requirements, but the limitations for file names with respect to the OS should be taken . . .
. . .
but will not erase the content of all the other tables. So the check whether the content of a page has changed (MD5sum) is still available for a fast re-index procedure. Once prepared, multithreaded re-indexing could be invoked by starting several threads and adding individual IDs to the option parameter like: php spider.php -erased1 php . . .
. . .
4.4 Ignoring parts of a page Sphider-plus includes an option to exclude parts of pages from being indexed. Thi 775e s can for example be used to prevent search result flooding when certain keywords appear on certain part in most pages (like a header, footer or a menu). Any part of a page between <! sphider_noindex > and <! . . .
. . .
<! sphider_noindex > and <! /sphider_noindex > tags is not indexed, however links in it are followed. 4.5 Ignoring parts of a page by <div id='abc'> Ignoring parts of a page by the <! sphider_noindex > tags requires direct access to the page, because the tags need to be added (edited) to the page. A more flexible method, . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ menu[0-5] / 4.6 Indexing only parts of a page by <div id='abc'> If enabled in Admin settings, the values as defined in the list-file /include/common/divs_use.txt will be used to index only the content between <div . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ table[0-5] / 4.7 Ignore HTML elements defined by <tagname> . . </tagname> This option is foreseen to cooperate with the new HTML5 elements like section, nav, aside, hgroup, article, header, footer, etc. HTML elements . . .
. . .
contain a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */nav[0-5]/ Please keep in mind that element names placed in /include/common/elements_not.txt will be processed case-sensitive. 4.8 Index only HTML elements defined by <tagname> . . </tagname> This is the vice versa . . .
. . .
of all the duplicate content URLs: http://www.example.com/product.php?item=swedish-fish&category=gummy-candy http://www.example.com/product.php?item=swedish-fish&trackingid=1234&sessionid=5678 and Sphider-plus will understand that the duplicates all refer to the canonical URL: http://www.example.com/product.php?item=swedish-fish. The . . .
. . .
charset are supported by the ConvertCharset function and will be used to convert text into UTF-8 Unicode: WINDOWS windows-1250 - Central Europe windows-1251 - Cyrillic windows-1252 - Latin I windows-1253 - Greek windows-1254 - Turkish windows-1255 - Hebrew windows-1256 - Arabic windows-1257 - Baltic windows-1258 - Viet Nam cp874 - Thai - this . . .
. . .
- Baltic windows-1258 - Viet Nam cp874 - Thai - this file is also for DOS DOS cp437 - Latin US cp737 - Greek cp775 - BaltRim cp850 - Latin1 cp852 - Latin2 cp855 - Cyrylic cp857 - Turkish cp860 - Portuguese cp861 - Iceland cp862 - Hebrew cp863 - Canada cp864 - Arabic cp865 - Nordic cp866 - Cyrylic Russian (this is the one, used in IE . . .
. . .
IE Cyrillic (DOS) ) cp869 - Greek2 MAC (Apple) x-mac-cyrillic x-mac-greek x-mac-icelandic x-mac-ce x-mac-roman ISO iso-8859-1 iso-8859-2 iso-8859-3 iso-8859-4 iso-8859-5 iso-8859-6 iso-8859-7 iso-8859-8 iso-8859-9 iso-8859-10 iso-8859-11 iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 . . .
. . .
iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 cp500 cp856 cp875 cp1006 cp1026 koi8-r (Cyrillic) koi8-u (Cyrillic Ukrainian) nextstep us-ascii us-ascii-quotes DSP implementation for NeXT stdenc symbol zdingbat And specially for old Polish programs: mazovia This list is to be read . . .
. . .
Access denied; you need the RELOAD privilege. . . Top 10. Enable real-time output of logging data Up to version 1.5 of Sphider-plus, during index / re-index there was no printout available because: - Several servers, especially on Win32, buffer the output from the script until it terminates before transmitting the results to the browser. - . . .
. . .
is seen. - Some versions of Microsoft Internet Explorer only start to display the page after they have received 256 bytes of output. As progress was not presented during index / re- index procedure, waiting for results became a pain in the neck. Selectable in Admin setting together with the update interval (1 - 10 seconds), AJAX technology was . . .
. . .
characters in front of words Warning: This option should be used with special care and not be activated for non ISO-8859 charsets. Some special characters as part of the word ending might be erased by accidental. Top 13. Media search for images, audio streams and videos 13.1 Media indexing Index of media files is enabled by separated Admin . . .
. . .
and 'Found at' - Total clicks - Last clicked - Query input 'Indexed Image Thumbnails' presenting: - Thumbnail 150 x 100 pixel - Image details like title, filename size of original image, link- and thumb-id - Option to delete the thumbnail In order to open the media files all tables contain active links. Media results are also stored in . . .
. . .
Settings' menu of the admin backend: First option: For results of XML product feeds, present the text extract of 350 characters as part of product attributes. If this option is not activated, all products of the XML product feed will be presented in search results, and the hits of the query string will be highlighted in all involved products. . . .
. . .
content of the database tables (those with the same table prefix) will be destroyed by the restore procedure. 16.5 Copy & Move This section of the Database Management will present only those databases, which are correctly configured, assigned and do have a set of installed tables as described in chapter Definition and configuration. This . . .
. . .
results. Never the less to find these x results, the complete database had to been browsed. Starting with version 2.5, an additional clean option is offered as part of the Admin backend. Main advantage of this option is a significant reduction of the search time for any query, because the content of the db could be limited to offer only x . . .
. . .
<link>http://www.abc.de/images/warp.gif</link> <title>warp.gif</title> <x_size>635</x_size> <y_size>98</y_size> </media_result> <media_result> <num>2</num> <type>audio</type> <url>http://www.abc.de/index.php</url> . . .
. . .
the following content: {"query":"colour","ip":"::1","host_name":"guard_007-hoster","query_time":"2016-03-18 10:56:23 AM","consumed":0.016,"total_results":2,"num_of_results":2,"from":1,"to":2,"text_results":[{"num":1,"weight":"100.0","url":"http:\/\/www.english.le-piaggie.info\/Info_eng.pdf","title":" Info_eng","fulltxt":" . . . . . .
. . .
placed on the ground floor, today's big kitchen with the heating fireplace for open and close mode of . . . ","page_size":"5,661.6kb","domain_name":"www.english.le-piaggie.info"},{"num":2,"weight":"50.0","url":"http:\/\/www.english.le-piaggie.info\/html\/description.html","title":" Description","fulltxt":" . . . cable. Additional active . . .
. . .
D.O.C.G. as far as to the Pratomagno mountains 160 olive-trees Detached house, about 800 m to reach the . . . ","page_size":"26.4 kb","domain_name":"www.english.le-piaggie.info"}]} Top . . .


5a2 All required information. Introduction Release and Legal Info Installation Documentation Change Log [ Documentation Summary ] Preamble: The info presented here is valid only for the latest release of Sphider-plus. At present version 4.2024a published February 12, 2024 is the actual release. 1. Settings, customizing and statistics 2. Indexing . . .
. . .
2. Indexing 2.1 Various options 2.2 Allow other hosts in same domain 2.3 Word stemming 2.4 Periodical Re-indexing 2.5 Preferred indexing 2.6 Multithreaded indexing 2.7 Create thumbnails during index procedure 2.8 Prevent indexing of known malware and pishing pages 2.9 Follow and create sitemap files 2.10 Use private sitemap instead of global . . .
. . .
Sitemap file 3. Using the indexer from command line 3.1 All options 3.2 Multithreaded indexing 4. Keeping pages, words and files from being indexed 4.1 robots.txt 4.2 Must include / must not include string list 4.3 Ignoring links 4.4 Ignoring parts of a page by <! sphider_noindex > 4.5 Ignoring parts of a page by <div id='abc'> . . .
. . .
charset' 6. Search modes 6.1 Search with wildcards * 6.2 Strict search ! 6.3 Tolerant search 6.4 Link search 6.5 Media search 6.6 Search only in one domain 6.7 Search in categories 6.8 Greek language support 6.9 Block queries 7. Chronological order for result listing 7.1 Text result listing 7.2 Media result listing 8. PDF converter 9. Clean . . .
. . .
of logging data 11. Error messages and Debug mode 12. Delete secondary characters 13. Media search for images, audio streams and videos 13.1 Media indexing 13.2 Not supported media content 13.3 Search for media content 13.4 Statistics for media content 14. Feed support 14.1 XML product feeds 14.2 RDF, RSD, RSS and Atom feeds 15. Result . . .
. . .
Overview 16.2 Definition and configuration 16.3 Activate / disable database 16.4 Backup & Restore of databases 16.5 Copy & and Move 16.6 Enhancing functionality of multiple database support 17. Search in categories 17.1 Hierachical structure 17.2 Parallel structure 18. User suggested sites 19. Vulnerability protection 19.1 Prevent queries . . .
. . .
templates 22.2 Embed the search engine into existing HTML code 22.3 The different style sheet files 23. JSON, XML and RSS result output [ Documentation ] 1. Settings, customizing and statistics If you want to change settings, behavior and design of Sphider-plus, you can do so by means of the Admin interface. There is a wide range of . . .
. . .
interface. There is a wide range of settings foreseen for Sphider-plus. Separated into different submenus like: Sites: - Add Site - Index only the new - Re-index all - Re-index only preferred URLs - Erase Re-index (available also for individual URLs) - Import/export URL list - Approve sites - Banned domains Categories: - Add, edit, delete - . . .
. . .
URL list - Approve sites - Banned domains Categories: - Add, edit, delete - Create new subcategory under Index: - Basic indexing options - Advanced options Clean: - Clean keywords not associated with any link - Clean links not associated with any site - Clean Category table not associated with any site - Clean Media links - Clear Temp table . . .
. . .
- Clear Search log - Clear 'Most Popular Page Links' log - Clear 'Most Popular Media Links' log - Clear Spider log, separate and bulk delete - Clear Thumbnail images, separate and bulk delete - Clear Text cache - Clear Media cache - Clear IDS log file - Clear flood attempts log file - Clear all entries in addurl or banned table - Truncate all . . .
. . .
flood attempts log file - Clear all entries in addurl or banned table - Truncate all tables in database Settings: - General Settings - Index Log Settings - Spider Settings - Search Settings - Order of Result listing - Suggest Options - Page Indexing Weights Database: - Configure up to 5 databases with unlimited number of table sets - Activate . . .
. . .
Database: - Configure up to 5 databases with unlimited number of table sets - Activate separately for 'Admin', 'Search' user and 'Suggest URL user' - Backup / Restore - Copy / Move - Optimize Templates: In order to enable customer's integration of Sphider-plus into existing sites, HTML templates are prepared for Search form Text result . . .
. . .
Search form Text result listing Media result listing Most popular queries etc. Three different designs are offered, which may be selected in submenu 'Settings'. If the layout does not fit the design of your site (which is normal), you may create new designs and modify the appropriate file /templates/My_template/adminstyle.css . . .
. . .
the appropriate file /templates/My_template/adminstyle.css /templates/My_template/userstyle.css Statistics output: - Top keywords (Top 50 with hit counter). - All indexed thumbnails w 3e80 ith ID3 and EXIF info. - Larges pages offering link URL and file size. - Most Popular Searches for text links offering: Link addr., total clicks, last . . .
. . .
Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Searches for media links offering: Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Links (click counter). - Search log offering: Query, Results, Queried at, Time taken, User IP, Country, Host name (Latest100) - Index log offering: . . .
. . .
Country, Host name (Latest100) - Index log offering: File-name, index date and delete option - sitemap log offering: sitemap.xml output sitemap list offering file/page suffixes - IDS log offering: IP, host, query, impact, involved tags, date and time of intrusion. - Flood attempts log offering: IP, query, date and time of flood attempt. - Auto . . .
. . .
attempts log offering: IP, query, date and time of flood attempt. - Auto Re-index log file - Server info offering: Server software, environment, MySQL, PDF-converter, image functions, php.ini file PHP integration, PHP security info. Each item holding lists of details. All text links, media links and thumbnails are active linked. As stated in . . .
. . .
individual Re-indexing, the periodical Re-indexer could be started and aborted in the "Options" menu of each site. 2.5 Preferred Re-indexing Each new URL added to the Admin backend, could be supplied with a priority level. This level will be used by the option 'Re-index only preferred sites'. Level 1 will be interpreted as most important, while . . .
. . .
less all index results will be stored in log files in sub folder /admin/log/ The names of the log files look like: db2_100524-21.47.56_1.html (log file of first thread) db2_100524-21.48.12_2.html (log file of second thread) and is build by the following items: db2 - Number of database. 100524 - Date (May 24, 2010) 21.47.56 - Time when this thread . . .
. . .
spider.php -new1 php spider.php -new2 etc. The IDs will be added to the name of the corresponding log files like: db2_100524-21.47.56_ID1.html (log file of first thread) db2_100524-21.48.12_ID2.html (log file of second thread) IDs could be defined by personal requirements, but the limitations for file names with respect to the OS should be taken . . .
. . .
but will not erase the content of all the other tables. So the check whether the content of a page has changed (MD5sum) is still available for a fast re-index procedure. Once prepared, multithreaded re-indexing could be invoked by starting several threads and adding individual IDs to the option parameter like: php spider.php -erased1 php . . .
. . .
4.4 Ignoring parts of a page Sphider-plus includes an option to exclude parts of pages from being indexed. Thi 775e s can for example be used to prevent search result flooding when certain keywords appear on certain part in most pages (like a header, footer or a menu). Any part of a page between <! sphider_noindex > and <! . . .
. . .
<! sphider_noindex > and <! /sphider_noindex > tags is not indexed, however links in it are followed. 4.5 Ignoring parts of a page by <div id='abc'> Ignoring parts of a page by the <! sphider_noindex > tags requires direct access to the page, because the tags need to be added (edited) to the page. A more flexible method, . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ menu[0-5] / 4.6 Indexing only parts of a page by <div id='abc'> If enabled in Admin settings, the values as defined in the list-file /include/common/divs_use.txt will be used to index only the content between <div . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ table[0-5] / 4.7 Ignore HTML elements defined by <tagname> . . </tagname> This option is foreseen to cooperate with the new HTML5 elements like section, nav, aside, hgroup, article, header, footer, etc. HTML elements . . .
. . .
contain a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */nav[0-5]/ Please keep in mind that element names placed in /include/common/elements_not.txt will be processed case-sensitive. 4.8 Index only HTML elements defined by <tagname> . . </tagname> This is the vice versa . . .
. . .
of all the duplicate content URLs: http://www.example.com/product.php?item=swedish-fish&category=gummy-candy http://www.example.com/product.php?item=swedish-fish&trackingid=1234&sessionid=5678 and Sphider-plus will understand that the duplicates all refer to the canonical URL: http://www.example.com/product.php?item=swedish-fish. The . . .
. . .
charset are supported by the ConvertCharset function and will be used to convert text into UTF-8 Unicode: WINDOWS windows-1250 - Central Europe windows-1251 - Cyrillic windows-1252 - Latin I windows-1253 - Greek windows-1254 - Turkish windows-1255 - Hebrew windows-1256 - Arabic windows-1257 - Baltic windows-1258 - Viet Nam cp874 - Thai - this . . .
. . .
- Baltic windows-1258 - Viet Nam cp874 - Thai - this file is also for DOS DOS cp437 - Latin US cp737 - Greek cp775 - BaltRim cp850 - Latin1 cp852 - Latin2 cp855 - Cyrylic cp857 - Turkish cp860 - Portuguese cp861 - Iceland cp862 - Hebrew cp863 - Canada cp864 - Arabic cp865 - Nordic cp866 - Cyrylic Russian (this is the one, used in IE . . .
. . .
IE Cyrillic (DOS) ) cp869 - Greek2 MAC (Apple) x-mac-cyrillic x-mac-greek x-mac-icelandic x-mac-ce x-mac-roman ISO iso-8859-1 iso-8859-2 iso-8859-3 iso-8859-4 iso-8859-5 iso-8859-6 iso-8859-7 iso-8859-8 iso-8859-9 iso-8859-10 iso-8859-11 iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 . . .
. . .
iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 cp500 cp856 cp875 cp1006 cp1026 koi8-r (Cyrillic) koi8-u (Cyrillic Ukrainian) nextstep us-ascii us-ascii-quotes DSP implementation for NeXT stdenc symbol zdingbat And specially for old Polish programs: mazovia This list is to be read . . .
. . .
Access denied; you need the RELOAD privilege. . . Top 10. Enable real-time output of logging data Up to version 1.5 of Sphider-plus, during index / re-index there was no printout available because: - Several servers, especially on Win32, buffer the output from the script until it terminates before transmitting the results to the browser. - . . .
. . .
is seen. - Some versions of Microsoft Internet Explorer only start to display the page after they have received 256 bytes of output. As progress was not presented during index / re- index procedure, waiting for results became a pain in the neck. Selectable in Admin setting together with the update interval (1 - 10 seconds), AJAX technology was . . .
. . .
characters in front of words Warning: This option should be used with special care and not be activated for non ISO-8859 charsets. Some special characters as part of the word ending might be erased by accidental. Top 13. Media search for images, audio streams and videos 13.1 Media indexing Index of media files is enabled by separated Admin . . .
. . .
and 'Found at' - Total clicks - Last clicked - Query input 'Indexed Image Thumbnails' presenting: - Thumbnail 150 x 100 pixel - Image details like title, filename size of original image, link- and thumb-id - Option to delete the thumbnail In order to open the media files all tables contain active links. Media results are also stored in . . .
. . .
Settings' menu of the admin backend: First option: For results of XML product feeds, present the text extract of 350 characters as part of product attributes. If this option is not activated, all products of the XML product feed will be presented in search results, and the hits of the query string will be highlighted in all involved products. . . .
. . .
content of the database tables (those with the same table prefix) will be destroyed by the restore procedure. 16.5 Copy & Move This section of the Database Management will present only those databases, which are correctly configured, assigned and do have a set of installed tables as described in chapter Definition and configuration. This . . .
. . .
results. Never the less to find these x results, the complete database had to been browsed. Starting with version 2.5, an additional clean option is offered as part of the Admin backend. Main advantage of this option is a significant reduction of the search time for any query, because the content of the db could be limited to offer only x . . .
. . .
<link>http://www.abc.de/images/warp.gif</link> <title>warp.gif</title> <x_size>635</x_size> <y_size>98</y_size> </media_result> <media_result> <num>2</num> <type>audio</type> <url>http://www.abc.de/index.php</url> . . .
. . .
the following content: {"query":"colour","ip":"::1","host_name":"guard_007-hoster","query_time":"2016-03-18 10:56:23 AM","consumed":0.016,"total_results":2,"num_of_results":2,"from":1,"to":2,"text_results":[{"num":1,"weight":"100.0","url":"http:\/\/www.english.le-piaggie.info\/Info_eng.pdf","title":" Info_eng","fulltxt":" . . . . . .
. . .
placed on the ground floor, today's big kitchen with the heating fireplace for open and close mode of . . . ","page_size":"5,661.6kb","domain_name":"www.english.le-piaggie.info"},{"num":2,"weight":"50.0","url":"http:\/\/www.english.le-piaggie.info\/html\/description.html","title":" Description","fulltxt":" . . . cable. Additional active . . .
. . .
D.O.C.G. as far as to the Pratomagno mountains 160 olive-trees Detached house, about 800 m to reach the . . . ","page_size":"26.4 kb","domain_name":"www.english.le-piaggie.info"}]} Top . . .


5a2 All required information. Introduction Release and Legal Info Installation Documentation Change Log [ Documentation Summary ] Preamble: The info presented here is valid only for the latest release of Sphider-plus. At present version 4.2024a published February 12, 2024 is the actual release. 1. Settings, customizing and statistics 2. Indexing . . .
. . .
2. Indexing 2.1 Various options 2.2 Allow other hosts in same domain 2.3 Word stemming 2.4 Periodical Re-indexing 2.5 Preferred indexing 2.6 Multithreaded indexing 2.7 Create thumbnails during index procedure 2.8 Prevent indexing of known malware and pishing pages 2.9 Follow and create sitemap files 2.10 Use private sitemap instead of global . . .
. . .
Sitemap file 3. Using the indexer from command line 3.1 All options 3.2 Multithreaded indexing 4. Keeping pages, words and files from being indexed 4.1 robots.txt 4.2 Must include / must not include string list 4.3 Ignoring links 4.4 Ignoring parts of a page by <! sphider_noindex > 4.5 Ignoring parts of a page by <div id='abc'> . . .
. . .
charset' 6. Search modes 6.1 Search with wildcards * 6.2 Strict search ! 6.3 Tolerant search 6.4 Link search 6.5 Media search 6.6 Search only in one domain 6.7 Search in categories 6.8 Greek language support 6.9 Block queries 7. Chronological order for result listing 7.1 Text result listing 7.2 Media result listing 8. PDF converter 9. Clean . . .
. . .
of logging data 11. Error messages and Debug mode 12. Delete secondary characters 13. Media search for images, audio streams and videos 13.1 Media indexing 13.2 Not supported media content 13.3 Search for media content 13.4 Statistics for media content 14. Feed support 14.1 XML product feeds 14.2 RDF, RSD, RSS and Atom feeds 15. Result . . .
. . .
Overview 16.2 Definition and configuration 16.3 Activate / disable database 16.4 Backup & Restore of databases 16.5 Copy & and Move 16.6 Enhancing functionality of multiple database support 17. Search in categories 17.1 Hierachical structure 17.2 Parallel structure 18. User suggested sites 19. Vulnerability protection 19.1 Prevent queries . . .
. . .
templates 22.2 Embed the search engine into existing HTML code 22.3 The different style sheet files 23. JSON, XML and RSS result output [ Documentation ] 1. Settings, customizing and statistics If you want to change settings, behavior and design of Sphider-plus, you can do so by means of the Admin interface. There is a wide range of . . .
. . .
interface. There is a wide range of settings foreseen for Sphider-plus. Separated into different submenus like: Sites: - Add Site - Index only the new - Re-index all - Re-index only preferred URLs - Erase Re-index (available also for individual URLs) - Import/export URL list - Approve sites - Banned domains Categories: - Add, edit, delete - . . .
. . .
URL list - Approve sites - Banned domains Categories: - Add, edit, delete - Create new subcategory under Index: - Basic indexing options - Advanced options Clean: - Clean keywords not associated with any link - Clean links not associated with any site - Clean Category table not associated with any site - Clean Media links - Clear Temp table . . .
. . .
- Clear Search log - Clear 'Most Popular Page Links' log - Clear 'Most Popular Media Links' log - Clear Spider log, separate and bulk delete - Clear Thumbnail images, separate and bulk delete - Clear Text cache - Clear Media cache - Clear IDS log file - Clear flood attempts log file - Clear all entries in addurl or banned table - Truncate all . . .
. . .
flood attempts log file - Clear all entries in addurl or banned table - Truncate all tables in database Settings: - General Settings - Index Log Settings - Spider Settings - Search Settings - Order of Result listing - Suggest Options - Page Indexing Weights Database: - Configure up to 5 databases with unlimited number of table sets - Activate . . .
. . .
Database: - Configure up to 5 databases with unlimited number of table sets - Activate separately for 'Admin', 'Search' user and 'Suggest URL user' - Backup / Restore - Copy / Move - Optimize Templates: In order to enable customer's integration of Sphider-plus into existing sites, HTML templates are prepared for Search form Text result . . .
. . .
Search form Text result listing Media result listing Most popular queries etc. Three different designs are offered, which may be selected in submenu 'Settings'. If the layout does not fit the design of your site (which is normal), you may create new designs and modify the appropriate file /templates/My_template/adminstyle.css . . .
. . .
the appropriate file /templates/My_template/adminstyle.css /templates/My_template/userstyle.css Statistics output: - Top keywords (Top 50 with hit counter). - All indexed thumbnails w 3e80 ith ID3 and EXIF info. - Larges pages offering link URL and file size. - Most Popular Searches for text links offering: Link addr., total clicks, last . . .
. . .
Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Searches for media links offering: Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Links (click counter). - Search log offering: Query, Results, Queried at, Time taken, User IP, Country, Host name (Latest100) - Index log offering: . . .
. . .
Country, Host name (Latest100) - Index log offering: File-name, index date and delete option - sitemap log offering: sitemap.xml output sitemap list offering file/page suffixes - IDS log offering: IP, host, query, impact, involved tags, date and time of intrusion. - Flood attempts log offering: IP, query, date and time of flood attempt. - Auto . . .
. . .
attempts log offering: IP, query, date and time of flood attempt. - Auto Re-index log file - Server info offering: Server software, environment, MySQL, PDF-converter, image functions, php.ini file PHP integration, PHP security info. Each item holding lists of details. All text links, media links and thumbnails are active linked. As stated in . . .
. . .
individual Re-indexing, the periodical Re-indexer could be started and aborted in the "Options" menu of each site. 2.5 Preferred Re-indexing Each new URL added to the Admin backend, could be supplied with a priority level. This level will be used by the option 'Re-index only preferred sites'. Level 1 will be interpreted as most important, while . . .
. . .
less all index results will be stored in log files in sub folder /admin/log/ The names of the log files look like: db2_100524-21.47.56_1.html (log file of first thread) db2_100524-21.48.12_2.html (log file of second thread) and is build by the following items: db2 - Number of database. 100524 - Date (May 24, 2010) 21.47.56 - Time when this thread . . .
. . .
spider.php -new1 php spider.php -new2 etc. The IDs will be added to the name of the corresponding log files like: db2_100524-21.47.56_ID1.html (log file of first thread) db2_100524-21.48.12_ID2.html (log file of second thread) IDs could be defined by personal requirements, but the limitations for file names with respect to the OS should be taken . . .
. . .
but will not erase the content of all the other tables. So the check whether the content of a page has changed (MD5sum) is still available for a fast re-index procedure. Once prepared, multithreaded re-indexing could be invoked by starting several threads and adding individual IDs to the option parameter like: php spider.php -erased1 php . . .
. . .
4.4 Ignoring parts of a page Sphider-plus includes an option to exclude parts of pages from being indexed. Thi 775e s can for example be used to prevent search result flooding when certain keywords appear on certain part in most pages (like a header, footer or a menu). Any part of a page between <! sphider_noindex > and <! . . .
. . .
<! sphider_noindex > and <! /sphider_noindex > tags is not indexed, however links in it are followed. 4.5 Ignoring parts of a page by <div id='abc'> Ignoring parts of a page by the <! sphider_noindex > tags requires direct access to the page, because the tags need to be added (edited) to the page. A more flexible method, . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ menu[0-5] / 4.6 Indexing only parts of a page by <div id='abc'> If enabled in Admin settings, the values as defined in the list-file /include/common/divs_use.txt will be used to index only the content between <div . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ table[0-5] / 4.7 Ignore HTML elements defined by <tagname> . . </tagname> This option is foreseen to cooperate with the new HTML5 elements like section, nav, aside, hgroup, article, header, footer, etc. HTML elements . . .
. . .
contain a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */nav[0-5]/ Please keep in mind that element names placed in /include/common/elements_not.txt will be processed case-sensitive. 4.8 Index only HTML elements defined by <tagname> . . </tagname> This is the vice versa . . .
. . .
of all the duplicate content URLs: http://www.example.com/product.php?item=swedish-fish&category=gummy-candy http://www.example.com/product.php?item=swedish-fish&trackingid=1234&sessionid=5678 and Sphider-plus will understand that the duplicates all refer to the canonical URL: http://www.example.com/product.php?item=swedish-fish. The . . .
. . .
charset are supported by the ConvertCharset function and will be used to convert text into UTF-8 Unicode: WINDOWS windows-1250 - Central Europe windows-1251 - Cyrillic windows-1252 - Latin I windows-1253 - Greek windows-1254 - Turkish windows-1255 - Hebrew windows-1256 - Arabic windows-1257 - Baltic windows-1258 - Viet Nam cp874 - Thai - this . . .
. . .
- Baltic windows-1258 - Viet Nam cp874 - Thai - this file is also for DOS DOS cp437 - Latin US cp737 - Greek cp775 - BaltRim cp850 - Latin1 cp852 - Latin2 cp855 - Cyrylic cp857 - Turkish cp860 - Portuguese cp861 - Iceland cp862 - Hebrew cp863 - Canada cp864 - Arabic cp865 - Nordic cp866 - Cyrylic Russian (this is the one, used in IE . . .
. . .
IE Cyrillic (DOS) ) cp869 - Greek2 MAC (Apple) x-mac-cyrillic x-mac-greek x-mac-icelandic x-mac-ce x-mac-roman ISO iso-8859-1 iso-8859-2 iso-8859-3 iso-8859-4 iso-8859-5 iso-8859-6 iso-8859-7 iso-8859-8 iso-8859-9 iso-8859-10 iso-8859-11 iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 . . .
. . .
iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 cp500 cp856 cp875 cp1006 cp1026 koi8-r (Cyrillic) koi8-u (Cyrillic Ukrainian) nextstep us-ascii us-ascii-quotes DSP implementation for NeXT stdenc symbol zdingbat And specially for old Polish programs: mazovia This list is to be read . . .
. . .
Access denied; you need the RELOAD privilege. . . Top 10. Enable real-time output of logging data Up to version 1.5 of Sphider-plus, during index / re-index there was no printout available because: - Several servers, especially on Win32, buffer the output from the script until it terminates before transmitting the results to the browser. - . . .
. . .
is seen. - Some versions of Microsoft Internet Explorer only start to display the page after they have received 256 bytes of output. As progress was not presented during index / re- index procedure, waiting for results became a pain in the neck. Selectable in Admin setting together with the update interval (1 - 10 seconds), AJAX technology was . . .
. . .
characters in front of words Warning: This option should be used with special care and not be activated for non ISO-8859 charsets. Some special characters as part of the word ending might be erased by accidental. Top 13. Media search for images, audio streams and videos 13.1 Media indexing Index of media files is enabled by separated Admin . . .
. . .
and 'Found at' - Total clicks - Last clicked - Query input 'Indexed Image Thumbnails' presenting: - Thumbnail 150 x 100 pixel - Image details like title, filename size of original image, link- and thumb-id - Option to delete the thumbnail In order to open the media files all tables contain active links. Media results are also stored in . . .
. . .
Settings' menu of the admin backend: First option: For results of XML product feeds, present the text extract of 350 characters as part of product attributes. If this option is not activated, all products of the XML product feed will be presented in search results, and the hits of the query string will be highlighted in all involved products. . . .
. . .
content of the database tables (those with the same table prefix) will be destroyed by the restore procedure. 16.5 Copy & Move This section of the Database Management will present only those databases, which are correctly configured, assigned and do have a set of installed tables as described in chapter Definition and configuration. This . . .
. . .
results. Never the less to find these x results, the complete database had to been browsed. Starting with version 2.5, an additional clean option is offered as part of the Admin backend. Main advantage of this option is a significant reduction of the search time for any query, because the content of the db could be limited to offer only x . . .
. . .
<link>http://www.abc.de/images/warp.gif</link> <title>warp.gif</title> <x_size>635</x_size> <y_size>98</y_size> </media_result> <media_result> <num>2</num> <type>audio</type> <url>http://www.abc.de/index.php</url> . . .
. . .
the following content: {"query":"colour","ip":"::1","host_name":"guard_007-hoster","query_time":"2016-03-18 10:56:23 AM","consumed":0.016,"total_results":2,"num_of_results":2,"from":1,"to":2,"text_results":[{"num":1,"weight":"100.0","url":"http:\/\/www.english.le-piaggie.info\/Info_eng.pdf","title":" Info_eng","fulltxt":" . . . . . .
. . .
placed on the ground floor, today's big kitchen with the heating fireplace for open and close mode of . . . ","page_size":"5,661.6kb","domain_name":"www.english.le-piaggie.info"},{"num":2,"weight":"50.0","url":"http:\/\/www.english.le-piaggie.info\/html\/description.html","title":" Description","fulltxt":" . . . cable. Additional active . . .
. . .
D.O.C.G. as far as to the Pratomagno mountains 160 olive-trees Detached house, about 800 m to reach the . . . ","page_size":"26.4 kb","domain_name":"www.english.le-piaggie.info"}]} Top . . .


5a2 All required information. Introduction Release and Legal Info Installation Documentation Change Log [ Documentation Summary ] Preamble: The info presented here is valid only for the latest release of Sphider-plus. At present version 4.2024a published February 12, 2024 is the actual release. 1. Settings, customizing and statistics 2. Indexing . . .
. . .
2. Indexing 2.1 Various options 2.2 Allow other hosts in same domain 2.3 Word stemming 2.4 Periodical Re-indexing 2.5 Preferred indexing 2.6 Multithreaded indexing 2.7 Create thumbnails during index procedure 2.8 Prevent indexing of known malware and pishing pages 2.9 Follow and create sitemap files 2.10 Use private sitemap instead of global . . .
. . .
Sitemap file 3. Using the indexer from command line 3.1 All options 3.2 Multithreaded indexing 4. Keeping pages, words and files from being indexed 4.1 robots.txt 4.2 Must include / must not include string list 4.3 Ignoring links 4.4 Ignoring parts of a page by <! sphider_noindex > 4.5 Ignoring parts of a page by <div id='abc'> . . .
. . .
charset' 6. Search modes 6.1 Search with wildcards * 6.2 Strict search ! 6.3 Tolerant search 6.4 Link search 6.5 Media search 6.6 Search only in one domain 6.7 Search in categories 6.8 Greek language support 6.9 Block queries 7. Chronological order for result listing 7.1 Text result listing 7.2 Media result listing 8. PDF converter 9. Clean . . .
. . .
of logging data 11. Error messages and Debug mode 12. Delete secondary characters 13. Media search for images, audio streams and videos 13.1 Media indexing 13.2 Not supported media content 13.3 Search for media content 13.4 Statistics for media content 14. Feed support 14.1 XML product feeds 14.2 RDF, RSD, RSS and Atom feeds 15. Result . . .
. . .
Overview 16.2 Definition and configuration 16.3 Activate / disable database 16.4 Backup & Restore of databases 16.5 Copy & and Move 16.6 Enhancing functionality of multiple database support 17. Search in categories 17.1 Hierachical structure 17.2 Parallel structure 18. User suggested sites 19. Vulnerability protection 19.1 Prevent queries . . .
. . .
templates 22.2 Embed the search engine into existing HTML code 22.3 The different style sheet files 23. JSON, XML and RSS result output [ Documentation ] 1. Settings, customizing and statistics If you want to change settings, behavior and design of Sphider-plus, you can do so by means of the Admin interface. There is a wide range of . . .
. . .
interface. There is a wide range of settings foreseen for Sphider-plus. Separated into different submenus like: Sites: - Add Site - Index only the new - Re-index all - Re-index only preferred URLs - Erase Re-index (available also for individual URLs) - Import/export URL list - Approve sites - Banned domains Categories: - Add, edit, delete - . . .
. . .
URL list - Approve sites - Banned domains Categories: - Add, edit, delete - Create new subcategory under Index: - Basic indexing options - Advanced options Clean: - Clean keywords not associated with any link - Clean links not associated with any site - Clean Category table not associated with any site - Clean Media links - Clear Temp table . . .
. . .
- Clear Search log - Clear 'Most Popular Page Links' log - Clear 'Most Popular Media Links' log - Clear Spider log, separate and bulk delete - Clear Thumbnail images, separate and bulk delete - Clear Text cache - Clear Media cache - Clear IDS log file - Clear flood attempts log file - Clear all entries in addurl or banned table - Truncate all . . .
. . .
flood attempts log file - Clear all entries in addurl or banned table - Truncate all tables in database Settings: - General Settings - Index Log Settings - Spider Settings - Search Settings - Order of Result listing - Suggest Options - Page Indexing Weights Database: - Configure up to 5 databases with unlimited number of table sets - Activate . . .
. . .
Database: - Configure up to 5 databases with unlimited number of table sets - Activate separately for 'Admin', 'Search' user and 'Suggest URL user' - Backup / Restore - Copy / Move - Optimize Templates: In order to enable customer's integration of Sphider-plus into existing sites, HTML templates are prepared for Search form Text result . . .
. . .
Search form Text result listing Media result listing Most popular queries etc. Three different designs are offered, which may be selected in submenu 'Settings'. If the layout does not fit the design of your site (which is normal), you may create new designs and modify the appropriate file /templates/My_template/adminstyle.css . . .
. . .
the appropriate file /templates/My_template/adminstyle.css /templates/My_template/userstyle.css Statistics output: - Top keywords (Top 50 with hit counter). - All indexed thumbnails w 3e80 ith ID3 and EXIF info. - Larges pages offering link URL and file size. - Most Popular Searches for text links offering: Link addr., total clicks, last . . .
. . .
Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Searches for media links offering: Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Links (click counter). - Search log offering: Query, Results, Queried at, Time taken, User IP, Country, Host name (Latest100) - Index log offering: . . .
. . .
Country, Host name (Latest100) - Index log offering: File-name, index date and delete option - sitemap log offering: sitemap.xml output sitemap list offering file/page suffixes - IDS log offering: IP, host, query, impact, involved tags, date and time of intrusion. - Flood attempts log offering: IP, query, date and time of flood attempt. - Auto . . .
. . .
attempts log offering: IP, query, date and time of flood attempt. - Auto Re-index log file - Server info offering: Server software, environment, MySQL, PDF-converter, image functions, php.ini file PHP integration, PHP security info. Each item holding lists of details. All text links, media links and thumbnails are active linked. As stated in . . .
. . .
individual Re-indexing, the periodical Re-indexer could be started and aborted in the "Options" menu of each site. 2.5 Preferred Re-indexing Each new URL added to the Admin backend, could be supplied with a priority level. This level will be used by the option 'Re-index only preferred sites'. Level 1 will be interpreted as most important, while . . .
. . .
less all index results will be stored in log files in sub folder /admin/log/ The names of the log files look like: db2_100524-21.47.56_1.html (log file of first thread) db2_100524-21.48.12_2.html (log file of second thread) and is build by the following items: db2 - Number of database. 100524 - Date (May 24, 2010) 21.47.56 - Time when this thread . . .
. . .
spider.php -new1 php spider.php -new2 etc. The IDs will be added to the name of the corresponding log files like: db2_100524-21.47.56_ID1.html (log file of first thread) db2_100524-21.48.12_ID2.html (log file of second thread) IDs could be defined by personal requirements, but the limitations for file names with respect to the OS should be taken . . .
. . .
but will not erase the content of all the other tables. So the check whether the content of a page has changed (MD5sum) is still available for a fast re-index procedure. Once prepared, multithreaded re-indexing could be invoked by starting several threads and adding individual IDs to the option parameter like: php spider.php -erased1 php . . .
. . .
4.4 Ignoring parts of a page Sphider-plus includes an option to exclude parts of pages from being indexed. Thi 775e s can for example be used to prevent search result flooding when certain keywords appear on certain part in most pages (like a header, footer or a menu). Any part of a page between <! sphider_noindex > and <! . . .
. . .
<! sphider_noindex > and <! /sphider_noindex > tags is not indexed, however links in it are followed. 4.5 Ignoring parts of a page by <div id='abc'> Ignoring parts of a page by the <! sphider_noindex > tags requires direct access to the page, because the tags need to be added (edited) to the page. A more flexible method, . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ menu[0-5] / 4.6 Indexing only parts of a page by <div id='abc'> If enabled in Admin settings, the values as defined in the list-file /include/common/divs_use.txt will be used to index only the content between <div . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ table[0-5] / 4.7 Ignore HTML elements defined by <tagname> . . </tagname> This option is foreseen to cooperate with the new HTML5 elements like section, nav, aside, hgroup, article, header, footer, etc. HTML elements . . .
. . .
contain a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */nav[0-5]/ Please keep in mind that element names placed in /include/common/elements_not.txt will be processed case-sensitive. 4.8 Index only HTML elements defined by <tagname> . . </tagname> This is the vice versa . . .
. . .
of all the duplicate content URLs: http://www.example.com/product.php?item=swedish-fish&category=gummy-candy http://www.example.com/product.php?item=swedish-fish&trackingid=1234&sessionid=5678 and Sphider-plus will understand that the duplicates all refer to the canonical URL: http://www.example.com/product.php?item=swedish-fish. The . . .
. . .
charset are supported by the ConvertCharset function and will be used to convert text into UTF-8 Unicode: WINDOWS windows-1250 - Central Europe windows-1251 - Cyrillic windows-1252 - Latin I windows-1253 - Greek windows-1254 - Turkish windows-1255 - Hebrew windows-1256 - Arabic windows-1257 - Baltic windows-1258 - Viet Nam cp874 - Thai - this . . .
. . .
- Baltic windows-1258 - Viet Nam cp874 - Thai - this file is also for DOS DOS cp437 - Latin US cp737 - Greek cp775 - BaltRim cp850 - Latin1 cp852 - Latin2 cp855 - Cyrylic cp857 - Turkish cp860 - Portuguese cp861 - Iceland cp862 - Hebrew cp863 - Canada cp864 - Arabic cp865 - Nordic cp866 - Cyrylic Russian (this is the one, used in IE . . .
. . .
IE Cyrillic (DOS) ) cp869 - Greek2 MAC (Apple) x-mac-cyrillic x-mac-greek x-mac-icelandic x-mac-ce x-mac-roman ISO iso-8859-1 iso-8859-2 iso-8859-3 iso-8859-4 iso-8859-5 iso-8859-6 iso-8859-7 iso-8859-8 iso-8859-9 iso-8859-10 iso-8859-11 iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 . . .
. . .
iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 cp500 cp856 cp875 cp1006 cp1026 koi8-r (Cyrillic) koi8-u (Cyrillic Ukrainian) nextstep us-ascii us-ascii-quotes DSP implementation for NeXT stdenc symbol zdingbat And specially for old Polish programs: mazovia This list is to be read . . .
. . .
Access denied; you need the RELOAD privilege. . . Top 10. Enable real-time output of logging data Up to version 1.5 of Sphider-plus, during index / re-index there was no printout available because: - Several servers, especially on Win32, buffer the output from the script until it terminates before transmitting the results to the browser. - . . .
. . .
is seen. - Some versions of Microsoft Internet Explorer only start to display the page after they have received 256 bytes of output. As progress was not presented during index / re- index procedure, waiting for results became a pain in the neck. Selectable in Admin setting together with the update interval (1 - 10 seconds), AJAX technology was . . .
. . .
characters in front of words Warning: This option should be used with special care and not be activated for non ISO-8859 charsets. Some special characters as part of the word ending might be erased by accidental. Top 13. Media search for images, audio streams and videos 13.1 Media indexing Index of media files is enabled by separated Admin . . .
. . .
and 'Found at' - Total clicks - Last clicked - Query input 'Indexed Image Thumbnails' presenting: - Thumbnail 150 x 100 pixel - Image details like title, filename size of original image, link- and thumb-id - Option to delete the thumbnail In order to open the media files all tables contain active links. Media results are also stored in . . .
. . .
Settings' menu of the admin backend: First option: For results of XML product feeds, present the text extract of 350 characters as part of product attributes. If this option is not activated, all products of the XML product feed will be presented in search results, and the hits of the query string will be highlighted in all involved products. . . .
. . .
content of the database tables (those with the same table prefix) will be destroyed by the restore procedure. 16.5 Copy & Move This section of the Database Management will present only those databases, which are correctly configured, assigned and do have a set of installed tables as described in chapter Definition and configuration. This . . .
. . .
results. Never the less to find these x results, the complete database had to been browsed. Starting with version 2.5, an additional clean option is offered as part of the Admin backend. Main advantage of this option is a significant reduction of the search time for any query, because the content of the db could be limited to offer only x . . .
. . .
<link>http://www.abc.de/images/warp.gif</link> <title>warp.gif</title> <x_size>635</x_size> <y_size>98</y_size> </media_result> <media_result> <num>2</num> <type>audio</type> <url>http://www.abc.de/index.php</url> . . .
. . .
the following content: {"query":"colour","ip":"::1","host_name":"guard_007-hoster","query_time":"2016-03-18 10:56:23 AM","consumed":0.016,"total_results":2,"num_of_results":2,"from":1,"to":2,"text_results":[{"num":1,"weight":"100.0","url":"http:\/\/www.english.le-piaggie.info\/Info_eng.pdf","title":" Info_eng","fulltxt":" . . . . . .
. . .
placed on the ground floor, today's big kitchen with the heating fireplace for open and close mode of . . . ","page_size":"5,661.6kb","domain_name":"www.english.le-piaggie.info"},{"num":2,"weight":"50.0","url":"http:\/\/www.english.le-piaggie.info\/html\/description.html","title":" Description","fulltxt":" . . . cable. Additional active . . .
. . .
D.O.C.G. as far as to the Pratomagno mountains 160 olive-trees Detached house, about 800 m to reach the . . . ","page_size":"26.4 kb","domain_name":"www.english.le-piaggie.info"}]} Top . . .


5a2 All required information. Introduction Release and Legal Info Installation Documentation Change Log [ Documentation Summary ] Preamble: The info presented here is valid only for the latest release of Sphider-plus. At present version 4.2024a published February 12, 2024 is the actual release. 1. Settings, customizing and statistics 2. Indexing . . .
. . .
2. Indexing 2.1 Various options 2.2 Allow other hosts in same domain 2.3 Word stemming 2.4 Periodical Re-indexing 2.5 Preferred indexing 2.6 Multithreaded indexing 2.7 Create thumbnails during index procedure 2.8 Prevent indexing of known malware and pishing pages 2.9 Follow and create sitemap files 2.10 Use private sitemap instead of global . . .
. . .
Sitemap file 3. Using the indexer from command line 3.1 All options 3.2 Multithreaded indexing 4. Keeping pages, words and files from being indexed 4.1 robots.txt 4.2 Must include / must not include string list 4.3 Ignoring links 4.4 Ignoring parts of a page by <! sphider_noindex > 4.5 Ignoring parts of a page by <div id='abc'> . . .
. . .
charset' 6. Search modes 6.1 Search with wildcards * 6.2 Strict search ! 6.3 Tolerant search 6.4 Link search 6.5 Media search 6.6 Search only in one domain 6.7 Search in categories 6.8 Greek language support 6.9 Block queries 7. Chronological order for result listing 7.1 Text result listing 7.2 Media result listing 8. PDF converter 9. Clean . . .
. . .
of logging data 11. Error messages and Debug mode 12. Delete secondary characters 13. Media search for images, audio streams and videos 13.1 Media indexing 13.2 Not supported media content 13.3 Search for media content 13.4 Statistics for media content 14. Feed support 14.1 XML product feeds 14.2 RDF, RSD, RSS and Atom feeds 15. Result . . .
. . .
Overview 16.2 Definition and configuration 16.3 Activate / disable database 16.4 Backup & Restore of databases 16.5 Copy & and Move 16.6 Enhancing functionality of multiple database support 17. Search in categories 17.1 Hierachical structure 17.2 Parallel structure 18. User suggested sites 19. Vulnerability protection 19.1 Prevent queries . . .
. . .
templates 22.2 Embed the search engine into existing HTML code 22.3 The different style sheet files 23. JSON, XML and RSS result output [ Documentation ] 1. Settings, customizing and statistics If you want to change settings, behavior and design of Sphider-plus, you can do so by means of the Admin interface. There is a wide range of . . .
. . .
interface. There is a wide range of settings foreseen for Sphider-plus. Separated into different submenus like: Sites: - Add Site - Index only the new - Re-index all - Re-index only preferred URLs - Erase Re-index (available also for individual URLs) - Import/export URL list - Approve sites - Banned domains Categories: - Add, edit, delete - . . .
. . .
URL list - Approve sites - Banned domains Categories: - Add, edit, delete - Create new subcategory under Index: - Basic indexing options - Advanced options Clean: - Clean keywords not associated with any link - Clean links not associated with any site - Clean Category table not associated with any site - Clean Media links - Clear Temp table . . .
. . .
- Clear Search log - Clear 'Most Popular Page Links' log - Clear 'Most Popular Media Links' log - Clear Spider log, separate and bulk delete - Clear Thumbnail images, separate and bulk delete - Clear Text cache - Clear Media cache - Clear IDS log file - Clear flood attempts log file - Clear all entries in addurl or banned table - Truncate all . . .
. . .
flood attempts log file - Clear all entries in addurl or banned table - Truncate all tables in database Settings: - General Settings - Index Log Settings - Spider Settings - Search Settings - Order of Result listing - Suggest Options - Page Indexing Weights Database: - Configure up to 5 databases with unlimited number of table sets - Activate . . .
. . .
Database: - Configure up to 5 databases with unlimited number of table sets - Activate separately for 'Admin', 'Search' user and 'Suggest URL user' - Backup / Restore - Copy / Move - Optimize Templates: In order to enable customer's integration of Sphider-plus into existing sites, HTML templates are prepared for Search form Text result . . .
. . .
Search form Text result listing Media result listing Most popular queries etc. Three different designs are offered, which may be selected in submenu 'Settings'. If the layout does not fit the design of your site (which is normal), you may create new designs and modify the appropriate file /templates/My_template/adminstyle.css . . .
. . .
the appropriate file /templates/My_template/adminstyle.css /templates/My_template/userstyle.css Statistics output: - Top keywords (Top 50 with hit counter). - All indexed thumbnails w 3e80 ith ID3 and EXIF info. - Larges pages offering link URL and file size. - Most Popular Searches for text links offering: Link addr., total clicks, last . . .
. . .
Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Searches for media links offering: Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Links (click counter). - Search log offering: Query, Results, Queried at, Time taken, User IP, Country, Host name (Latest100) - Index log offering: . . .
. . .
Country, Host name (Latest100) - Index log offering: File-name, index date and delete option - sitemap log offering: sitemap.xml output sitemap list offering file/page suffixes - IDS log offering: IP, host, query, impact, involved tags, date and time of intrusion. - Flood attempts log offering: IP, query, date and time of flood attempt. - Auto . . .
. . .
attempts log offering: IP, query, date and time of flood attempt. - Auto Re-index log file - Server info offering: Server software, environment, MySQL, PDF-converter, image functions, php.ini file PHP integration, PHP security info. Each item holding lists of details. All text links, media links and thumbnails are active linked. As stated in . . .
. . .
individual Re-indexing, the periodical Re-indexer could be started and aborted in the "Options" menu of each site. 2.5 Preferred Re-indexing Each new URL added to the Admin backend, could be supplied with a priority level. This level will be used by the option 'Re-index only preferred sites'. Level 1 will be interpreted as most important, while . . .
. . .
less all index results will be stored in log files in sub folder /admin/log/ The names of the log files look like: db2_100524-21.47.56_1.html (log file of first thread) db2_100524-21.48.12_2.html (log file of second thread) and is build by the following items: db2 - Number of database. 100524 - Date (May 24, 2010) 21.47.56 - Time when this thread . . .
. . .
spider.php -new1 php spider.php -new2 etc. The IDs will be added to the name of the corresponding log files like: db2_100524-21.47.56_ID1.html (log file of first thread) db2_100524-21.48.12_ID2.html (log file of second thread) IDs could be defined by personal requirements, but the limitations for file names with respect to the OS should be taken . . .
. . .
but will not erase the content of all the other tables. So the check whether the content of a page has changed (MD5sum) is still available for a fast re-index procedure. Once prepared, multithreaded re-indexing could be invoked by starting several threads and adding individual IDs to the option parameter like: php spider.php -erased1 php . . .
. . .
4.4 Ignoring parts of a page Sphider-plus includes an option to exclude parts of pages from being indexed. Thi 775e s can for example be used to prevent search result flooding when certain keywords appear on certain part in most pages (like a header, footer or a menu). Any part of a page between <! sphider_noindex > and <! . . .
. . .
<! sphider_noindex > and <! /sphider_noindex > tags is not indexed, however links in it are followed. 4.5 Ignoring parts of a page by <div id='abc'> Ignoring parts of a page by the <! sphider_noindex > tags requires direct access to the page, because the tags need to be added (edited) to the page. A more flexible method, . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ menu[0-5] / 4.6 Indexing only parts of a page by <div id='abc'> If enabled in Admin settings, the values as defined in the list-file /include/common/divs_use.txt will be used to index only the content between <div . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ table[0-5] / 4.7 Ignore HTML elements defined by <tagname> . . </tagname> This option is foreseen to cooperate with the new HTML5 elements like section, nav, aside, hgroup, article, header, footer, etc. HTML elements . . .
. . .
contain a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */nav[0-5]/ Please keep in mind that element names placed in /include/common/elements_not.txt will be processed case-sensitive. 4.8 Index only HTML elements defined by <tagname> . . </tagname> This is the vice versa . . .
. . .
of all the duplicate content URLs: http://www.example.com/product.php?item=swedish-fish&category=gummy-candy http://www.example.com/product.php?item=swedish-fish&trackingid=1234&sessionid=5678 and Sphider-plus will understand that the duplicates all refer to the canonical URL: http://www.example.com/product.php?item=swedish-fish. The . . .
. . .
charset are supported by the ConvertCharset function and will be used to convert text into UTF-8 Unicode: WINDOWS windows-1250 - Central Europe windows-1251 - Cyrillic windows-1252 - Latin I windows-1253 - Greek windows-1254 - Turkish windows-1255 - Hebrew windows-1256 - Arabic windows-1257 - Baltic windows-1258 - Viet Nam cp874 - Thai - this . . .
. . .
- Baltic windows-1258 - Viet Nam cp874 - Thai - this file is also for DOS DOS cp437 - Latin US cp737 - Greek cp775 - BaltRim cp850 - Latin1 cp852 - Latin2 cp855 - Cyrylic cp857 - Turkish cp860 - Portuguese cp861 - Iceland cp862 - Hebrew cp863 - Canada cp864 - Arabic cp865 - Nordic cp866 - Cyrylic Russian (this is the one, used in IE . . .
. . .
IE Cyrillic (DOS) ) cp869 - Greek2 MAC (Apple) x-mac-cyrillic x-mac-greek x-mac-icelandic x-mac-ce x-mac-roman ISO iso-8859-1 iso-8859-2 iso-8859-3 iso-8859-4 iso-8859-5 iso-8859-6 iso-8859-7 iso-8859-8 iso-8859-9 iso-8859-10 iso-8859-11 iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 . . .
. . .
iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 cp500 cp856 cp875 cp1006 cp1026 koi8-r (Cyrillic) koi8-u (Cyrillic Ukrainian) nextstep us-ascii us-ascii-quotes DSP implementation for NeXT stdenc symbol zdingbat And specially for old Polish programs: mazovia This list is to be read . . .
. . .
Access denied; you need the RELOAD privilege. . . Top 10. Enable real-time output of logging data Up to version 1.5 of Sphider-plus, during index / re-index there was no printout available because: - Several servers, especially on Win32, buffer the output from the script until it terminates before transmitting the results to the browser. - . . .
. . .
is seen. - Some versions of Microsoft Internet Explorer only start to display the page after they have received 256 bytes of output. As progress was not presented during index / re- index procedure, waiting for results became a pain in the neck. Selectable in Admin setting together with the update interval (1 - 10 seconds), AJAX technology was . . .
. . .
characters in front of words Warning: This option should be used with special care and not be activated for non ISO-8859 charsets. Some special characters as part of the word ending might be erased by accidental. Top 13. Media search for images, audio streams and videos 13.1 Media indexing Index of media files is enabled by separated Admin . . .
. . .
and 'Found at' - Total clicks - Last clicked - Query input 'Indexed Image Thumbnails' presenting: - Thumbnail 150 x 100 pixel - Image details like title, filename size of original image, link- and thumb-id - Option to delete the thumbnail In order to open the media files all tables contain active links. Media results are also stored in . . .
. . .
Settings' menu of the admin backend: First option: For results of XML product feeds, present the text extract of 350 characters as part of product attributes. If this option is not activated, all products of the XML product feed will be presented in search results, and the hits of the query string will be highlighted in all involved products. . . .
. . .
content of the database tables (those with the same table prefix) will be destroyed by the restore procedure. 16.5 Copy & Move This section of the Database Management will present only those databases, which are correctly configured, assigned and do have a set of installed tables as described in chapter Definition and configuration. This . . .
. . .
results. Never the less to find these x results, the complete database had to been browsed. Starting with version 2.5, an additional clean option is offered as part of the Admin backend. Main advantage of this option is a significant reduction of the search time for any query, because the content of the db could be limited to offer only x . . .
. . .
<link>http://www.abc.de/images/warp.gif</link> <title>warp.gif</title> <x_size>635</x_size> <y_size>98</y_size> </media_result> <media_result> <num>2</num> <type>audio</type> <url>http://www.abc.de/index.php</url> . . .
. . .
the following content: {"query":"colour","ip":"::1","host_name":"guard_007-hoster","query_time":"2016-03-18 10:56:23 AM","consumed":0.016,"total_results":2,"num_of_results":2,"from":1,"to":2,"text_results":[{"num":1,"weight":"100.0","url":"http:\/\/www.english.le-piaggie.info\/Info_eng.pdf","title":" Info_eng","fulltxt":" . . . . . .
. . .
placed on the ground floor, today's big kitchen with the heating fireplace for open and close mode of . . . ","page_size":"5,661.6kb","domain_name":"www.english.le-piaggie.info"},{"num":2,"weight":"50.0","url":"http:\/\/www.english.le-piaggie.info\/html\/description.html","title":" Description","fulltxt":" . . . cable. Additional active . . .
. . .
D.O.C.G. as far as to the Pratomagno mountains 160 olive-trees Detached house, about 800 m to reach the . . . ","page_size":"26.4 kb","domain_name":"www.english.le-piaggie.info"}]} Top . . .


5a2 All required information. Introduction Release and Legal Info Installation Documentation Change Log [ Documentation Summary ] Preamble: The info presented here is valid only for the latest release of Sphider-plus. At present version 4.2024a published February 12, 2024 is the actual release. 1. Settings, customizing and statistics 2. Indexing . . .
. . .
2. Indexing 2.1 Various options 2.2 Allow other hosts in same domain 2.3 Word stemming 2.4 Periodical Re-indexing 2.5 Preferred indexing 2.6 Multithreaded indexing 2.7 Create thumbnails during index procedure 2.8 Prevent indexing of known malware and pishing pages 2.9 Follow and create sitemap files 2.10 Use private sitemap instead of global . . .
. . .
Sitemap file 3. Using the indexer from command line 3.1 All options 3.2 Multithreaded indexing 4. Keeping pages, words and files from being indexed 4.1 robots.txt 4.2 Must include / must not include string list 4.3 Ignoring links 4.4 Ignoring parts of a page by <! sphider_noindex > 4.5 Ignoring parts of a page by <div id='abc'> . . .
. . .
charset' 6. Search modes 6.1 Search with wildcards * 6.2 Strict search ! 6.3 Tolerant search 6.4 Link search 6.5 Media search 6.6 Search only in one domain 6.7 Search in categories 6.8 Greek language support 6.9 Block queries 7. Chronological order for result listing 7.1 Text result listing 7.2 Media result listing 8. PDF converter 9. Clean . . .
. . .
of logging data 11. Error messages and Debug mode 12. Delete secondary characters 13. Media search for images, audio streams and videos 13.1 Media indexing 13.2 Not supported media content 13.3 Search for media content 13.4 Statistics for media content 14. Feed support 14.1 XML product feeds 14.2 RDF, RSD, RSS and Atom feeds 15. Result . . .
. . .
Overview 16.2 Definition and configuration 16.3 Activate / disable database 16.4 Backup & Restore of databases 16.5 Copy & and Move 16.6 Enhancing functionality of multiple database support 17. Search in categories 17.1 Hierachical structure 17.2 Parallel structure 18. User suggested sites 19. Vulnerability protection 19.1 Prevent queries . . .
. . .
templates 22.2 Embed the search engine into existing HTML code 22.3 The different style sheet files 23. JSON, XML and RSS result output [ Documentation ] 1. Settings, customizing and statistics If you want to change settings, behavior and design of Sphider-plus, you can do so by means of the Admin interface. There is a wide range of . . .
. . .
interface. There is a wide range of settings foreseen for Sphider-plus. Separated into different submenus like: Sites: - Add Site - Index only the new - Re-index all - Re-index only preferred URLs - Erase Re-index (available also for individual URLs) - Import/export URL list - Approve sites - Banned domains Categories: - Add, edit, delete - . . .
. . .
URL list - Approve sites - Banned domains Categories: - Add, edit, delete - Create new subcategory under Index: - Basic indexing options - Advanced options Clean: - Clean keywords not associated with any link - Clean links not associated with any site - Clean Category table not associated with any site - Clean Media links - Clear Temp table . . .
. . .
- Clear Search log - Clear 'Most Popular Page Links' log - Clear 'Most Popular Media Links' log - Clear Spider log, separate and bulk delete - Clear Thumbnail images, separate and bulk delete - Clear Text cache - Clear Media cache - Clear IDS log file - Clear flood attempts log file - Clear all entries in addurl or banned table - Truncate all . . .
. . .
flood attempts log file - Clear all entries in addurl or banned table - Truncate all tables in database Settings: - General Settings - Index Log Settings - Spider Settings - Search Settings - Order of Result listing - Suggest Options - Page Indexing Weights Database: - Configure up to 5 databases with unlimited number of table sets - Activate . . .
. . .
Database: - Configure up to 5 databases with unlimited number of table sets - Activate separately for 'Admin', 'Search' user and 'Suggest URL user' - Backup / Restore - Copy / Move - Optimize Templates: In order to enable customer's integration of Sphider-plus into existing sites, HTML templates are prepared for Search form Text result . . .
. . .
Search form Text result listing Media result listing Most popular queries etc. Three different designs are offered, which may be selected in submenu 'Settings'. If the layout does not fit the design of your site (which is normal), you may create new designs and modify the appropriate file /templates/My_template/adminstyle.css . . .
. . .
the appropriate file /templates/My_template/adminstyle.css /templates/My_template/userstyle.css Statistics output: - Top keywords (Top 50 with hit counter). - All indexed thumbnails w 3e80 ith ID3 and EXIF info. - Larges pages offering link URL and file size. - Most Popular Searches for text links offering: Link addr., total clicks, last . . .
. . .
Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Searches for media links offering: Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Links (click counter). - Search log offering: Query, Results, Queried at, Time taken, User IP, Country, Host name (Latest100) - Index log offering: . . .
. . .
Country, Host name (Latest100) - Index log offering: File-name, index date and delete option - sitemap log offering: sitemap.xml output sitemap list offering file/page suffixes - IDS log offering: IP, host, query, impact, involved tags, date and time of intrusion. - Flood attempts log offering: IP, query, date and time of flood attempt. - Auto . . .
. . .
attempts log offering: IP, query, date and time of flood attempt. - Auto Re-index log file - Server info offering: Server software, environment, MySQL, PDF-converter, image functions, php.ini file PHP integration, PHP security info. Each item holding lists of details. All text links, media links and thumbnails are active linked. As stated in . . .
. . .
individual Re-indexing, the periodical Re-indexer could be started and aborted in the "Options" menu of each site. 2.5 Preferred Re-indexing Each new URL added to the Admin backend, could be supplied with a priority level. This level will be used by the option 'Re-index only preferred sites'. Level 1 will be interpreted as most important, while . . .
. . .
less all index results will be stored in log files in sub folder /admin/log/ The names of the log files look like: db2_100524-21.47.56_1.html (log file of first thread) db2_100524-21.48.12_2.html (log file of second thread) and is build by the following items: db2 - Number of database. 100524 - Date (May 24, 2010) 21.47.56 - Time when this thread . . .
. . .
spider.php -new1 php spider.php -new2 etc. The IDs will be added to the name of the corresponding log files like: db2_100524-21.47.56_ID1.html (log file of first thread) db2_100524-21.48.12_ID2.html (log file of second thread) IDs could be defined by personal requirements, but the limitations for file names with respect to the OS should be taken . . .
. . .
but will not erase the content of all the other tables. So the check whether the content of a page has changed (MD5sum) is still available for a fast re-index procedure. Once prepared, multithreaded re-indexing could be invoked by starting several threads and adding individual IDs to the option parameter like: php spider.php -erased1 php . . .
. . .
4.4 Ignoring parts of a page Sphider-plus includes an option to exclude parts of pages from being indexed. Thi 775e s can for example be used to prevent search result flooding when certain keywords appear on certain part in most pages (like a header, footer or a menu). Any part of a page between <! sphider_noindex > and <! . . .
. . .
<! sphider_noindex > and <! /sphider_noindex > tags is not indexed, however links in it are followed. 4.5 Ignoring parts of a page by <div id='abc'> Ignoring parts of a page by the <! sphider_noindex > tags requires direct access to the page, because the tags need to be added (edited) to the page. A more flexible method, . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ menu[0-5] / 4.6 Indexing only parts of a page by <div id='abc'> If enabled in Admin settings, the values as defined in the list-file /include/common/divs_use.txt will be used to index only the content between <div . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ table[0-5] / 4.7 Ignore HTML elements defined by <tagname> . . </tagname> This option is foreseen to cooperate with the new HTML5 elements like section, nav, aside, hgroup, article, header, footer, etc. HTML elements . . .
. . .
contain a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */nav[0-5]/ Please keep in mind that element names placed in /include/common/elements_not.txt will be processed case-sensitive. 4.8 Index only HTML elements defined by <tagname> . . </tagname> This is the vice versa . . .
. . .
of all the duplicate content URLs: http://www.example.com/product.php?item=swedish-fish&category=gummy-candy http://www.example.com/product.php?item=swedish-fish&trackingid=1234&sessionid=5678 and Sphider-plus will understand that the duplicates all refer to the canonical URL: http://www.example.com/product.php?item=swedish-fish. The . . .
. . .
charset are supported by the ConvertCharset function and will be used to convert text into UTF-8 Unicode: WINDOWS windows-1250 - Central Europe windows-1251 - Cyrillic windows-1252 - Latin I windows-1253 - Greek windows-1254 - Turkish windows-1255 - Hebrew windows-1256 - Arabic windows-1257 - Baltic windows-1258 - Viet Nam cp874 - Thai - this . . .
. . .
- Baltic windows-1258 - Viet Nam cp874 - Thai - this file is also for DOS DOS cp437 - Latin US cp737 - Greek cp775 - BaltRim cp850 - Latin1 cp852 - Latin2 cp855 - Cyrylic cp857 - Turkish cp860 - Portuguese cp861 - Iceland cp862 - Hebrew cp863 - Canada cp864 - Arabic cp865 - Nordic cp866 - Cyrylic Russian (this is the one, used in IE . . .
. . .
IE Cyrillic (DOS) ) cp869 - Greek2 MAC (Apple) x-mac-cyrillic x-mac-greek x-mac-icelandic x-mac-ce x-mac-roman ISO iso-8859-1 iso-8859-2 iso-8859-3 iso-8859-4 iso-8859-5 iso-8859-6 iso-8859-7 iso-8859-8 iso-8859-9 iso-8859-10 iso-8859-11 iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 . . .
. . .
iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 cp500 cp856 cp875 cp1006 cp1026 koi8-r (Cyrillic) koi8-u (Cyrillic Ukrainian) nextstep us-ascii us-ascii-quotes DSP implementation for NeXT stdenc symbol zdingbat And specially for old Polish programs: mazovia This list is to be read . . .
. . .
Access denied; you need the RELOAD privilege. . . Top 10. Enable real-time output of logging data Up to version 1.5 of Sphider-plus, during index / re-index there was no printout available because: - Several servers, especially on Win32, buffer the output from the script until it terminates before transmitting the results to the browser. - . . .
. . .
is seen. - Some versions of Microsoft Internet Explorer only start to display the page after they have received 256 bytes of output. As progress was not presented during index / re- index procedure, waiting for results became a pain in the neck. Selectable in Admin setting together with the update interval (1 - 10 seconds), AJAX technology was . . .
. . .
characters in front of words Warning: This option should be used with special care and not be activated for non ISO-8859 charsets. Some special characters as part of the word ending might be erased by accidental. Top 13. Media search for images, audio streams and videos 13.1 Media indexing Index of media files is enabled by separated Admin . . .
. . .
and 'Found at' - Total clicks - Last clicked - Query input 'Indexed Image Thumbnails' presenting: - Thumbnail 150 x 100 pixel - Image details like title, filename size of original image, link- and thumb-id - Option to delete the thumbnail In order to open the media files all tables contain active links. Media results are also stored in . . .
. . .
Settings' menu of the admin backend: First option: For results of XML product feeds, present the text extract of 350 characters as part of product attributes. If this option is not activated, all products of the XML product feed will be presented in search results, and the hits of the query string will be highlighted in all involved products. . . .
. . .
content of the database tables (those with the same table prefix) will be destroyed by the restore procedure. 16.5 Copy & Move This section of the Database Management will present only those databases, which are correctly configured, assigned and do have a set of installed tables as described in chapter Definition and configuration. This . . .
. . .
results. Never the less to find these x results, the complete database had to been browsed. Starting with version 2.5, an additional clean option is offered as part of the Admin backend. Main advantage of this option is a significant reduction of the search time for any query, because the content of the db could be limited to offer only x . . .
. . .
<link>http://www.abc.de/images/warp.gif</link> <title>warp.gif</title> <x_size>635</x_size> <y_size>98</y_size> </media_result> <media_result> <num>2</num> <type>audio</type> <url>http://www.abc.de/index.php</url> . . .
. . .
the following content: {"query":"colour","ip":"::1","host_name":"guard_007-hoster","query_time":"2016-03-18 10:56:23 AM","consumed":0.016,"total_results":2,"num_of_results":2,"from":1,"to":2,"text_results":[{"num":1,"weight":"100.0","url":"http:\/\/www.english.le-piaggie.info\/Info_eng.pdf","title":" Info_eng","fulltxt":" . . . . . .
. . .
placed on the ground floor, today's big kitchen with the heating fireplace for open and close mode of . . . ","page_size":"5,661.6kb","domain_name":"www.english.le-piaggie.info"},{"num":2,"weight":"50.0","url":"http:\/\/www.english.le-piaggie.info\/html\/description.html","title":" Description","fulltxt":" . . . cable. Additional active . . .
. . .
D.O.C.G. as far as to the Pratomagno mountains 160 olive-trees Detached house, about 800 m to reach the . . . ","page_size":"26.4 kb","domain_name":"www.english.le-piaggie.info"}]} Top . . .


5a2 All required information. Introduction Release and Legal Info Installation Documentation Change Log [ Documentation Summary ] Preamble: The info presented here is valid only for the latest release of Sphider-plus. At present version 4.2024a published February 12, 2024 is the actual release. 1. Settings, customizing and statistics 2. Indexing . . .
. . .
2. Indexing 2.1 Various options 2.2 Allow other hosts in same domain 2.3 Word stemming 2.4 Periodical Re-indexing 2.5 Preferred indexing 2.6 Multithreaded indexing 2.7 Create thumbnails during index procedure 2.8 Prevent indexing of known malware and pishing pages 2.9 Follow and create sitemap files 2.10 Use private sitemap instead of global . . .
. . .
Sitemap file 3. Using the indexer from command line 3.1 All options 3.2 Multithreaded indexing 4. Keeping pages, words and files from being indexed 4.1 robots.txt 4.2 Must include / must not include string list 4.3 Ignoring links 4.4 Ignoring parts of a page by <! sphider_noindex > 4.5 Ignoring parts of a page by <div id='abc'> . . .
. . .
charset' 6. Search modes 6.1 Search with wildcards * 6.2 Strict search ! 6.3 Tolerant search 6.4 Link search 6.5 Media search 6.6 Search only in one domain 6.7 Search in categories 6.8 Greek language support 6.9 Block queries 7. Chronological order for result listing 7.1 Text result listing 7.2 Media result listing 8. PDF converter 9. Clean . . .
. . .
of logging data 11. Error messages and Debug mode 12. Delete secondary characters 13. Media search for images, audio streams and videos 13.1 Media indexing 13.2 Not supported media content 13.3 Search for media content 13.4 Statistics for media content 14. Feed support 14.1 XML product feeds 14.2 RDF, RSD, RSS and Atom feeds 15. Result . . .
. . .
Overview 16.2 Definition and configuration 16.3 Activate / disable database 16.4 Backup & Restore of databases 16.5 Copy & and Move 16.6 Enhancing functionality of multiple database support 17. Search in categories 17.1 Hierachical structure 17.2 Parallel structure 18. User suggested sites 19. Vulnerability protection 19.1 Prevent queries . . .
. . .
templates 22.2 Embed the search engine into existing HTML code 22.3 The different style sheet files 23. JSON, XML and RSS result output [ Documentation ] 1. Settings, customizing and statistics If you want to change settings, behavior and design of Sphider-plus, you can do so by means of the Admin interface. There is a wide range of . . .
. . .
interface. There is a wide range of settings foreseen for Sphider-plus. Separated into different submenus like: Sites: - Add Site - Index only the new - Re-index all - Re-index only preferred URLs - Erase Re-index (available also for individual URLs) - Import/export URL list - Approve sites - Banned domains Categories: - Add, edit, delete - . . .
. . .
URL list - Approve sites - Banned domains Categories: - Add, edit, delete - Create new subcategory under Index: - Basic indexing options - Advanced options Clean: - Clean keywords not associated with any link - Clean links not associated with any site - Clean Category table not associated with any site - Clean Media links - Clear Temp table . . .
. . .
- Clear Search log - Clear 'Most Popular Page Links' log - Clear 'Most Popular Media Links' log - Clear Spider log, separate and bulk delete - Clear Thumbnail images, separate and bulk delete - Clear Text cache - Clear Media cache - Clear IDS log file - Clear flood attempts log file - Clear all entries in addurl or banned table - Truncate all . . .
. . .
flood attempts log file - Clear all entries in addurl or banned table - Truncate all tables in database Settings: - General Settings - Index Log Settings - Spider Settings - Search Settings - Order of Result listing - Suggest Options - Page Indexing Weights Database: - Configure up to 5 databases with unlimited number of table sets - Activate . . .
. . .
Database: - Configure up to 5 databases with unlimited number of table sets - Activate separately for 'Admin', 'Search' user and 'Suggest URL user' - Backup / Restore - Copy / Move - Optimize Templates: In order to enable customer's integration of Sphider-plus into existing sites, HTML templates are prepared for Search form Text result . . .
. . .
Search form Text result listing Media result listing Most popular queries etc. Three different designs are offered, which may be selected in submenu 'Settings'. If the layout does not fit the design of your site (which is normal), you may create new designs and modify the appropriate file /templates/My_template/adminstyle.css . . .
. . .
the appropriate file /templates/My_template/adminstyle.css /templates/My_template/userstyle.css Statistics output: - Top keywords (Top 50 with hit counter). - All indexed thumbnails w 3e80 ith ID3 and EXIF info. - Larges pages offering link URL and file size. - Most Popular Searches for text links offering: Link addr., total clicks, last . . .
. . .
Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Searches for media links offering: Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Links (click counter). - Search log offering: Query, Results, Queried at, Time taken, User IP, Country, Host name (Latest100) - Index log offering: . . .
. . .
Country, Host name (Latest100) - Index log offering: File-name, index date and delete option - sitemap log offering: sitemap.xml output sitemap list offering file/page suffixes - IDS log offering: IP, host, query, impact, involved tags, date and time of intrusion. - Flood attempts log offering: IP, query, date and time of flood attempt. - Auto . . .
. . .
attempts log offering: IP, query, date and time of flood attempt. - Auto Re-index log file - Server info offering: Server software, environment, MySQL, PDF-converter, image functions, php.ini file PHP integration, PHP security info. Each item holding lists of details. All text links, media links and thumbnails are active linked. As stated in . . .
. . .
individual Re-indexing, the periodical Re-indexer could be started and aborted in the "Options" menu of each site. 2.5 Preferred Re-indexing Each new URL added to the Admin backend, could be supplied with a priority level. This level will be used by the option 'Re-index only preferred sites'. Level 1 will be interpreted as most important, while . . .
. . .
less all index results will be stored in log files in sub folder /admin/log/ The names of the log files look like: db2_100524-21.47.56_1.html (log file of first thread) db2_100524-21.48.12_2.html (log file of second thread) and is build by the following items: db2 - Number of database. 100524 - Date (May 24, 2010) 21.47.56 - Time when this thread . . .
. . .
spider.php -new1 php spider.php -new2 etc. The IDs will be added to the name of the corresponding log files like: db2_100524-21.47.56_ID1.html (log file of first thread) db2_100524-21.48.12_ID2.html (log file of second thread) IDs could be defined by personal requirements, but the limitations for file names with respect to the OS should be taken . . .
. . .
but will not erase the content of all the other tables. So the check whether the content of a page has changed (MD5sum) is still available for a fast re-index procedure. Once prepared, multithreaded re-indexing could be invoked by starting several threads and adding individual IDs to the option parameter like: php spider.php -erased1 php . . .
. . .
4.4 Ignoring parts of a page Sphider-plus includes an option to exclude parts of pages from being indexed. Thi 775e s can for example be used to prevent search result flooding when certain keywords appear on certain part in most pages (like a header, footer or a menu). Any part of a page between <! sphider_noindex > and <! . . .
. . .
<! sphider_noindex > and <! /sphider_noindex > tags is not indexed, however links in it are followed. 4.5 Ignoring parts of a page by <div id='abc'> Ignoring parts of a page by the <! sphider_noindex > tags requires direct access to the page, because the tags need to be added (edited) to the page. A more flexible method, . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ menu[0-5] / 4.6 Indexing only parts of a page by <div id='abc'> If enabled in Admin settings, the values as defined in the list-file /include/common/divs_use.txt will be used to index only the content between <div . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ table[0-5] / 4.7 Ignore HTML elements defined by <tagname> . . </tagname> This option is foreseen to cooperate with the new HTML5 elements like section, nav, aside, hgroup, article, header, footer, etc. HTML elements . . .
. . .
contain a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */nav[0-5]/ Please keep in mind that element names placed in /include/common/elements_not.txt will be processed case-sensitive. 4.8 Index only HTML elements defined by <tagname> . . </tagname> This is the vice versa . . .
. . .
of all the duplicate content URLs: http://www.example.com/product.php?item=swedish-fish&category=gummy-candy http://www.example.com/product.php?item=swedish-fish&trackingid=1234&sessionid=5678 and Sphider-plus will understand that the duplicates all refer to the canonical URL: http://www.example.com/product.php?item=swedish-fish. The . . .
. . .
charset are supported by the ConvertCharset function and will be used to convert text into UTF-8 Unicode: WINDOWS windows-1250 - Central Europe windows-1251 - Cyrillic windows-1252 - Latin I windows-1253 - Greek windows-1254 - Turkish windows-1255 - Hebrew windows-1256 - Arabic windows-1257 - Baltic windows-1258 - Viet Nam cp874 - Thai - this . . .
. . .
- Baltic windows-1258 - Viet Nam cp874 - Thai - this file is also for DOS DOS cp437 - Latin US cp737 - Greek cp775 - BaltRim cp850 - Latin1 cp852 - Latin2 cp855 - Cyrylic cp857 - Turkish cp860 - Portuguese cp861 - Iceland cp862 - Hebrew cp863 - Canada cp864 - Arabic cp865 - Nordic cp866 - Cyrylic Russian (this is the one, used in IE . . .
. . .
IE Cyrillic (DOS) ) cp869 - Greek2 MAC (Apple) x-mac-cyrillic x-mac-greek x-mac-icelandic x-mac-ce x-mac-roman ISO iso-8859-1 iso-8859-2 iso-8859-3 iso-8859-4 iso-8859-5 iso-8859-6 iso-8859-7 iso-8859-8 iso-8859-9 iso-8859-10 iso-8859-11 iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 . . .
. . .
iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 cp500 cp856 cp875 cp1006 cp1026 koi8-r (Cyrillic) koi8-u (Cyrillic Ukrainian) nextstep us-ascii us-ascii-quotes DSP implementation for NeXT stdenc symbol zdingbat And specially for old Polish programs: mazovia This list is to be read . . .
. . .
Access denied; you need the RELOAD privilege. . . Top 10. Enable real-time output of logging data Up to version 1.5 of Sphider-plus, during index / re-index there was no printout available because: - Several servers, especially on Win32, buffer the output from the script until it terminates before transmitting the results to the browser. - . . .
. . .
is seen. - Some versions of Microsoft Internet Explorer only start to display the page after they have received 256 bytes of output. As progress was not presented during index / re- index procedure, waiting for results became a pain in the neck. Selectable in Admin setting together with the update interval (1 - 10 seconds), AJAX technology was . . .
. . .
characters in front of words Warning: This option should be used with special care and not be activated for non ISO-8859 charsets. Some special characters as part of the word ending might be erased by accidental. Top 13. Media search for images, audio streams and videos 13.1 Media indexing Index of media files is enabled by separated Admin . . .
. . .
and 'Found at' - Total clicks - Last clicked - Query input 'Indexed Image Thumbnails' presenting: - Thumbnail 150 x 100 pixel - Image details like title, filename size of original image, link- and thumb-id - Option to delete the thumbnail In order to open the media files all tables contain active links. Media results are also stored in . . .
. . .
Settings' menu of the admin backend: First option: For results of XML product feeds, present the text extract of 350 characters as part of product attributes. If this option is not activated, all products of the XML product feed will be presented in search results, and the hits of the query string will be highlighted in all involved products. . . .
. . .
content of the database tables (those with the same table prefix) will be destroyed by the restore procedure. 16.5 Copy & Move This section of the Database Management will present only those databases, which are correctly configured, assigned and do have a set of installed tables as described in chapter Definition and configuration. This . . .
. . .
results. Never the less to find these x results, the complete database had to been browsed. Starting with version 2.5, an additional clean option is offered as part of the Admin backend. Main advantage of this option is a significant reduction of the search time for any query, because the content of the db could be limited to offer only x . . .
. . .
<link>http://www.abc.de/images/warp.gif</link> <title>warp.gif</title> <x_size>635</x_size> <y_size>98</y_size> </media_result> <media_result> <num>2</num> <type>audio</type> <url>http://www.abc.de/index.php</url> . . .
. . .
the following content: {"query":"colour","ip":"::1","host_name":"guard_007-hoster","query_time":"2016-03-18 10:56:23 AM","consumed":0.016,"total_results":2,"num_of_results":2,"from":1,"to":2,"text_results":[{"num":1,"weight":"100.0","url":"http:\/\/www.english.le-piaggie.info\/Info_eng.pdf","title":" Info_eng","fulltxt":" . . . . . .
. . .
placed on the ground floor, today's big kitchen with the heating fireplace for open and close mode of . . . ","page_size":"5,661.6kb","domain_name":"www.english.le-piaggie.info"},{"num":2,"weight":"50.0","url":"http:\/\/www.english.le-piaggie.info\/html\/description.html","title":" Description","fulltxt":" . . . cable. Additional active . . .
. . .
D.O.C.G. as far as to the Pratomagno mountains 160 olive-trees Detached house, about 800 m to reach the . . . ","page_size":"26.4 kb","domain_name":"www.english.le-piaggie.info"}]} Top . . .


5a2 All required information. Introduction Release and Legal Info Installation Documentation Change Log [ Documentation Summary ] Preamble: The info presented here is valid only for the latest release of Sphider-plus. At present version 4.2024a published February 12, 2024 is the actual release. 1. Settings, customizing and statistics 2. Indexing . . .
. . .
2. Indexing 2.1 Various options 2.2 Allow other hosts in same domain 2.3 Word stemming 2.4 Periodical Re-indexing 2.5 Preferred indexing 2.6 Multithreaded indexing 2.7 Create thumbnails during index procedure 2.8 Prevent indexing of known malware and pishing pages 2.9 Follow and create sitemap files 2.10 Use private sitemap instead of global . . .
. . .
Sitemap file 3. Using the indexer from command line 3.1 All options 3.2 Multithreaded indexing 4. Keeping pages, words and files from being indexed 4.1 robots.txt 4.2 Must include / must not include string list 4.3 Ignoring links 4.4 Ignoring parts of a page by <! sphider_noindex > 4.5 Ignoring parts of a page by <div id='abc'> . . .
. . .
charset' 6. Search modes 6.1 Search with wildcards * 6.2 Strict search ! 6.3 Tolerant search 6.4 Link search 6.5 Media search 6.6 Search only in one domain 6.7 Search in categories 6.8 Greek language support 6.9 Block queries 7. Chronological order for result listing 7.1 Text result listing 7.2 Media result listing 8. PDF converter 9. Clean . . .
. . .
of logging data 11. Error messages and Debug mode 12. Delete secondary characters 13. Media search for images, audio streams and videos 13.1 Media indexing 13.2 Not supported media content 13.3 Search for media content 13.4 Statistics for media content 14. Feed support 14.1 XML product feeds 14.2 RDF, RSD, RSS and Atom feeds 15. Result . . .
. . .
Overview 16.2 Definition and configuration 16.3 Activate / disable database 16.4 Backup & Restore of databases 16.5 Copy & and Move 16.6 Enhancing functionality of multiple database support 17. Search in categories 17.1 Hierachical structure 17.2 Parallel structure 18. User suggested sites 19. Vulnerability protection 19.1 Prevent queries . . .
. . .
templates 22.2 Embed the search engine into existing HTML code 22.3 The different style sheet files 23. JSON, XML and RSS result output [ Documentation ] 1. Settings, customizing and statistics If you want to change settings, behavior and design of Sphider-plus, you can do so by means of the Admin interface. There is a wide range of . . .
. . .
interface. There is a wide range of settings foreseen for Sphider-plus. Separated into different submenus like: Sites: - Add Site - Index only the new - Re-index all - Re-index only preferred URLs - Erase Re-index (available also for individual URLs) - Import/export URL list - Approve sites - Banned domains Categories: - Add, edit, delete - . . .
. . .
URL list - Approve sites - Banned domains Categories: - Add, edit, delete - Create new subcategory under Index: - Basic indexing options - Advanced options Clean: - Clean keywords not associated with any link - Clean links not associated with any site - Clean Category table not associated with any site - Clean Media links - Clear Temp table . . .
. . .
- Clear Search log - Clear 'Most Popular Page Links' log - Clear 'Most Popular Media Links' log - Clear Spider log, separate and bulk delete - Clear Thumbnail images, separate and bulk delete - Clear Text cache - Clear Media cache - Clear IDS log file - Clear flood attempts log file - Clear all entries in addurl or banned table - Truncate all . . .
. . .
flood attempts log file - Clear all entries in addurl or banned table - Truncate all tables in database Settings: - General Settings - Index Log Settings - Spider Settings - Search Settings - Order of Result listing - Suggest Options - Page Indexing Weights Database: - Configure up to 5 databases with unlimited number of table sets - Activate . . .
. . .
Database: - Configure up to 5 databases with unlimited number of table sets - Activate separately for 'Admin', 'Search' user and 'Suggest URL user' - Backup / Restore - Copy / Move - Optimize Templates: In order to enable customer's integration of Sphider-plus into existing sites, HTML templates are prepared for Search form Text result . . .
. . .
Search form Text result listing Media result listing Most popular queries etc. Three different designs are offered, which may be selected in submenu 'Settings'. If the layout does not fit the design of your site (which is normal), you may create new designs and modify the appropriate file /templates/My_template/adminstyle.css . . .
. . .
the appropriate file /templates/My_template/adminstyle.css /templates/My_template/userstyle.css Statistics output: - Top keywords (Top 50 with hit counter). - All indexed thumbnails w 3e80 ith ID3 and EXIF info. - Larges pages offering link URL and file size. - Most Popular Searches for text links offering: Link addr., total clicks, last . . .
. . .
Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Searches for media links offering: Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Links (click counter). - Search log offering: Query, Results, Queried at, Time taken, User IP, Country, Host name (Latest100) - Index log offering: . . .
. . .
Country, Host name (Latest100) - Index log offering: File-name, index date and delete option - sitemap log offering: sitemap.xml output sitemap list offering file/page suffixes - IDS log offering: IP, host, query, impact, involved tags, date and time of intrusion. - Flood attempts log offering: IP, query, date and time of flood attempt. - Auto . . .
. . .
attempts log offering: IP, query, date and time of flood attempt. - Auto Re-index log file - Server info offering: Server software, environment, MySQL, PDF-converter, image functions, php.ini file PHP integration, PHP security info. Each item holding lists of details. All text links, media links and thumbnails are active linked. As stated in . . .
. . .
individual Re-indexing, the periodical Re-indexer could be started and aborted in the "Options" menu of each site. 2.5 Preferred Re-indexing Each new URL added to the Admin backend, could be supplied with a priority level. This level will be used by the option 'Re-index only preferred sites'. Level 1 will be interpreted as most important, while . . .
. . .
less all index results will be stored in log files in sub folder /admin/log/ The names of the log files look like: db2_100524-21.47.56_1.html (log file of first thread) db2_100524-21.48.12_2.html (log file of second thread) and is build by the following items: db2 - Number of database. 100524 - Date (May 24, 2010) 21.47.56 - Time when this thread . . .
. . .
spider.php -new1 php spider.php -new2 etc. The IDs will be added to the name of the corresponding log files like: db2_100524-21.47.56_ID1.html (log file of first thread) db2_100524-21.48.12_ID2.html (log file of second thread) IDs could be defined by personal requirements, but the limitations for file names with respect to the OS should be taken . . .
. . .
but will not erase the content of all the other tables. So the check whether the content of a page has changed (MD5sum) is still available for a fast re-index procedure. Once prepared, multithreaded re-indexing could be invoked by starting several threads and adding individual IDs to the option parameter like: php spider.php -erased1 php . . .
. . .
4.4 Ignoring parts of a page Sphider-plus includes an option to exclude parts of pages from being indexed. Thi 775e s can for example be used to prevent search result flooding when certain keywords appear on certain part in most pages (like a header, footer or a menu). Any part of a page between <! sphider_noindex > and <! . . .
. . .
<! sphider_noindex > and <! /sphider_noindex > tags is not indexed, however links in it are followed. 4.5 Ignoring parts of a page by <div id='abc'> Ignoring parts of a page by the <! sphider_noindex > tags requires direct access to the page, because the tags need to be added (edited) to the page. A more flexible method, . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ menu[0-5] / 4.6 Indexing only parts of a page by <div id='abc'> If enabled in Admin settings, the values as defined in the list-file /include/common/divs_use.txt will be used to index only the content between <div . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ table[0-5] / 4.7 Ignore HTML elements defined by <tagname> . . </tagname> This option is foreseen to cooperate with the new HTML5 elements like section, nav, aside, hgroup, article, header, footer, etc. HTML elements . . .
. . .
contain a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */nav[0-5]/ Please keep in mind that element names placed in /include/common/elements_not.txt will be processed case-sensitive. 4.8 Index only HTML elements defined by <tagname> . . </tagname> This is the vice versa . . .
. . .
of all the duplicate content URLs: http://www.example.com/product.php?item=swedish-fish&category=gummy-candy http://www.example.com/product.php?item=swedish-fish&trackingid=1234&sessionid=5678 and Sphider-plus will understand that the duplicates all refer to the canonical URL: http://www.example.com/product.php?item=swedish-fish. The . . .
. . .
charset are supported by the ConvertCharset function and will be used to convert text into UTF-8 Unicode: WINDOWS windows-1250 - Central Europe windows-1251 - Cyrillic windows-1252 - Latin I windows-1253 - Greek windows-1254 - Turkish windows-1255 - Hebrew windows-1256 - Arabic windows-1257 - Baltic windows-1258 - Viet Nam cp874 - Thai - this . . .
. . .
- Baltic windows-1258 - Viet Nam cp874 - Thai - this file is also for DOS DOS cp437 - Latin US cp737 - Greek cp775 - BaltRim cp850 - Latin1 cp852 - Latin2 cp855 - Cyrylic cp857 - Turkish cp860 - Portuguese cp861 - Iceland cp862 - Hebrew cp863 - Canada cp864 - Arabic cp865 - Nordic cp866 - Cyrylic Russian (this is the one, used in IE . . .
. . .
IE Cyrillic (DOS) ) cp869 - Greek2 MAC (Apple) x-mac-cyrillic x-mac-greek x-mac-icelandic x-mac-ce x-mac-roman ISO iso-8859-1 iso-8859-2 iso-8859-3 iso-8859-4 iso-8859-5 iso-8859-6 iso-8859-7 iso-8859-8 iso-8859-9 iso-8859-10 iso-8859-11 iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 . . .
. . .
iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 cp500 cp856 cp875 cp1006 cp1026 koi8-r (Cyrillic) koi8-u (Cyrillic Ukrainian) nextstep us-ascii us-ascii-quotes DSP implementation for NeXT stdenc symbol zdingbat And specially for old Polish programs: mazovia This list is to be read . . .
. . .
Access denied; you need the RELOAD privilege. . . Top 10. Enable real-time output of logging data Up to version 1.5 of Sphider-plus, during index / re-index there was no printout available because: - Several servers, especially on Win32, buffer the output from the script until it terminates before transmitting the results to the browser. - . . .
. . .
is seen. - Some versions of Microsoft Internet Explorer only start to display the page after they have received 256 bytes of output. As progress was not presented during index / re- index procedure, waiting for results became a pain in the neck. Selectable in Admin setting together with the update interval (1 - 10 seconds), AJAX technology was . . .
. . .
characters in front of words Warning: This option should be used with special care and not be activated for non ISO-8859 charsets. Some special characters as part of the word ending might be erased by accidental. Top 13. Media search for images, audio streams and videos 13.1 Media indexing Index of media files is enabled by separated Admin . . .
. . .
and 'Found at' - Total clicks - Last clicked - Query input 'Indexed Image Thumbnails' presenting: - Thumbnail 150 x 100 pixel - Image details like title, filename size of original image, link- and thumb-id - Option to delete the thumbnail In order to open the media files all tables contain active links. Media results are also stored in . . .
. . .
Settings' menu of the admin backend: First option: For results of XML product feeds, present the text extract of 350 characters as part of product attributes. If this option is not activated, all products of the XML product feed will be presented in search results, and the hits of the query string will be highlighted in all involved products. . . .
. . .
content of the database tables (those with the same table prefix) will be destroyed by the restore procedure. 16.5 Copy & Move This section of the Database Management will present only those databases, which are correctly configured, assigned and do have a set of installed tables as described in chapter Definition and configuration. This . . .
. . .
results. Never the less to find these x results, the complete database had to been browsed. Starting with version 2.5, an additional clean option is offered as part of the Admin backend. Main advantage of this option is a significant reduction of the search time for any query, because the content of the db could be limited to offer only x . . .
. . .
<link>http://www.abc.de/images/warp.gif</link> <title>warp.gif</title> <x_size>635</x_size> <y_size>98</y_size> </media_result> <media_result> <num>2</num> <type>audio</type> <url>http://www.abc.de/index.php</url> . . .
. . .
the following content: {"query":"colour","ip":"::1","host_name":"guard_007-hoster","query_time":"2016-03-18 10:56:23 AM","consumed":0.016,"total_results":2,"num_of_results":2,"from":1,"to":2,"text_results":[{"num":1,"weight":"100.0","url":"http:\/\/www.english.le-piaggie.info\/Info_eng.pdf","title":" Info_eng","fulltxt":" . . . . . .
. . .
placed on the ground floor, today's big kitchen with the heating fireplace for open and close mode of . . . ","page_size":"5,661.6kb","domain_name":"www.english.le-piaggie.info"},{"num":2,"weight":"50.0","url":"http:\/\/www.english.le-piaggie.info\/html\/description.html","title":" Description","fulltxt":" . . . cable. Additional active . . .
. . .
D.O.C.G. as far as to the Pratomagno mountains 160 olive-trees Detached house, about 800 m to reach the . . . ","page_size":"26.4 kb","domain_name":"www.english.le-piaggie.info"}]} Top . . .


5a2 All required information. Introduction Release and Legal Info Installation Documentation Change Log [ Documentation Summary ] Preamble: The info presented here is valid only for the latest release of Sphider-plus. At present version 4.2024a published February 12, 2024 is the actual release. 1. Settings, customizing and statistics 2. Indexing . . .
. . .
2. Indexing 2.1 Various options 2.2 Allow other hosts in same domain 2.3 Word stemming 2.4 Periodical Re-indexing 2.5 Preferred indexing 2.6 Multithreaded indexing 2.7 Create thumbnails during index procedure 2.8 Prevent indexing of known malware and pishing pages 2.9 Follow and create sitemap files 2.10 Use private sitemap instead of global . . .
. . .
Sitemap file 3. Using the indexer from command line 3.1 All options 3.2 Multithreaded indexing 4. Keeping pages, words and files from being indexed 4.1 robots.txt 4.2 Must include / must not include string list 4.3 Ignoring links 4.4 Ignoring parts of a page by <! sphider_noindex > 4.5 Ignoring parts of a page by <div id='abc'> . . .
. . .
charset' 6. Search modes 6.1 Search with wildcards * 6.2 Strict search ! 6.3 Tolerant search 6.4 Link search 6.5 Media search 6.6 Search only in one domain 6.7 Search in categories 6.8 Greek language support 6.9 Block queries 7. Chronological order for result listing 7.1 Text result listing 7.2 Media result listing 8. PDF converter 9. Clean . . .
. . .
of logging data 11. Error messages and Debug mode 12. Delete secondary characters 13. Media search for images, audio streams and videos 13.1 Media indexing 13.2 Not supported media content 13.3 Search for media content 13.4 Statistics for media content 14. Feed support 14.1 XML product feeds 14.2 RDF, RSD, RSS and Atom feeds 15. Result . . .
. . .
Overview 16.2 Definition and configuration 16.3 Activate / disable database 16.4 Backup & Restore of databases 16.5 Copy & and Move 16.6 Enhancing functionality of multiple database support 17. Search in categories 17.1 Hierachical structure 17.2 Parallel structure 18. User suggested sites 19. Vulnerability protection 19.1 Prevent queries . . .
. . .
templates 22.2 Embed the search engine into existing HTML code 22.3 The different style sheet files 23. JSON, XML and RSS result output [ Documentation ] 1. Settings, customizing and statistics If you want to change settings, behavior and design of Sphider-plus, you can do so by means of the Admin interface. There is a wide range of . . .
. . .
interface. There is a wide range of settings foreseen for Sphider-plus. Separated into different submenus like: Sites: - Add Site - Index only the new - Re-index all - Re-index only preferred URLs - Erase Re-index (available also for individual URLs) - Import/export URL list - Approve sites - Banned domains Categories: - Add, edit, delete - . . .
. . .
URL list - Approve sites - Banned domains Categories: - Add, edit, delete - Create new subcategory under Index: - Basic indexing options - Advanced options Clean: - Clean keywords not associated with any link - Clean links not associated with any site - Clean Category table not associated with any site - Clean Media links - Clear Temp table . . .
. . .
- Clear Search log - Clear 'Most Popular Page Links' log - Clear 'Most Popular Media Links' log - Clear Spider log, separate and bulk delete - Clear Thumbnail images, separate and bulk delete - Clear Text cache - Clear Media cache - Clear IDS log file - Clear flood attempts log file - Clear all entries in addurl or banned table - Truncate all . . .
. . .
flood attempts log file - Clear all entries in addurl or banned table - Truncate all tables in database Settings: - General Settings - Index Log Settings - Spider Settings - Search Settings - Order of Result listing - Suggest Options - Page Indexing Weights Database: - Configure up to 5 databases with unlimited number of table sets - Activate . . .
. . .
Database: - Configure up to 5 databases with unlimited number of table sets - Activate separately for 'Admin', 'Search' user and 'Suggest URL user' - Backup / Restore - Copy / Move - Optimize Templates: In order to enable customer's integration of Sphider-plus into existing sites, HTML templates are prepared for Search form Text result . . .
. . .
Search form Text result listing Media result listing Most popular queries etc. Three different designs are offered, which may be selected in submenu 'Settings'. If the layout does not fit the design of your site (which is normal), you may create new designs and modify the appropriate file /templates/My_template/adminstyle.css . . .
. . .
the appropriate file /templates/My_template/adminstyle.css /templates/My_template/userstyle.css Statistics output: - Top keywords (Top 50 with hit counter). - All indexed thumbnails w 3e80 ith ID3 and EXIF info. - Larges pages offering link URL and file size. - Most Popular Searches for text links offering: Link addr., total clicks, last . . .
. . .
Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Searches for media links offering: Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Links (click counter). - Search log offering: Query, Results, Queried at, Time taken, User IP, Country, Host name (Latest100) - Index log offering: . . .
. . .
Country, Host name (Latest100) - Index log offering: File-name, index date and delete option - sitemap log offering: sitemap.xml output sitemap list offering file/page suffixes - IDS log offering: IP, host, query, impact, involved tags, date and time of intrusion. - Flood attempts log offering: IP, query, date and time of flood attempt. - Auto . . .
. . .
attempts log offering: IP, query, date and time of flood attempt. - Auto Re-index log file - Server info offering: Server software, environment, MySQL, PDF-converter, image functions, php.ini file PHP integration, PHP security info. Each item holding lists of details. All text links, media links and thumbnails are active linked. As stated in . . .
. . .
individual Re-indexing, the periodical Re-indexer could be started and aborted in the "Options" menu of each site. 2.5 Preferred Re-indexing Each new URL added to the Admin backend, could be supplied with a priority level. This level will be used by the option 'Re-index only preferred sites'. Level 1 will be interpreted as most important, while . . .
. . .
less all index results will be stored in log files in sub folder /admin/log/ The names of the log files look like: db2_100524-21.47.56_1.html (log file of first thread) db2_100524-21.48.12_2.html (log file of second thread) and is build by the following items: db2 - Number of database. 100524 - Date (May 24, 2010) 21.47.56 - Time when this thread . . .
. . .
spider.php -new1 php spider.php -new2 etc. The IDs will be added to the name of the corresponding log files like: db2_100524-21.47.56_ID1.html (log file of first thread) db2_100524-21.48.12_ID2.html (log file of second thread) IDs could be defined by personal requirements, but the limitations for file names with respect to the OS should be taken . . .
. . .
but will not erase the content of all the other tables. So the check whether the content of a page has changed (MD5sum)) is still available for a fast re-index procedure. Once prepared, multithreaded re-indexing could be invoked by starting several threads and adding individual IDs to the option parameter like: php spider.php -erased1 php . . .
. . .
4.4 Ignoring parts of a page Sphider-plus includes an option to exclude parts of pages from being indexed. Thi 775e s can for example be used to prevent search result flooding when certain keywords appear on certain part in most pages (like a header, footer or a menu). Any part of a page between <! sphider_noindex > and <! . . .
. . .
<! sphider_noindex > and <! /sphider_noindex > tags is not indexed, however links in it are followed. 4.5 Ignoring parts of a page by <div id='abc'> Ignoring parts of a page by the <! sphider_noindex > tags requires direct access to the page, because the tags need to be added (edited) to the page. A more flexible method, . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ menu[0-5] / 4.6 Indexing only parts of a page by <div id='abc'> If enabled in Admin settings, the values as defined in the list-file /include/common/divs_use.txt will be used to index only the content between <div . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ table[0-5] / 4.7 Ignore HTML elements defined by <tagname> . . </tagname> This option is foreseen to cooperate with the new HTML5 elements like section, nav, aside, hgroup, article, header, footer, etc. HTML elements . . .
. . .
contain a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */nav[0-5]/ Please keep in mind that element names placed in /include/common/elements_not.txt will be processed case-sensitive. 4.8 Index only HTML elements defined by <tagname> . . </tagname> This is the vice versa . . .
. . .
of all the duplicate content URLs: http://www.example.com/product.php?item=swedish-fish&category=gummy-candy http://www.example.com/product.php?item=swedish-fish&trackingid=1234&sessionid=5678 and Sphider-plus will understand that the duplicates all refer to the canonical URL: http://www.example.com/product.php?item=swedish-fish. The . . .
. . .
charset are supported by the ConvertCharset function and will be used to convert text into UTF-8 Unicode: WINDOWS windows-1250 - Central Europe windows-1251 - Cyrillic windows-1252 - Latin I windows-1253 - Greek windows-1254 - Turkish windows-1255 - Hebrew windows-1256 - Arabic windows-1257 - Baltic windows-1258 - Viet Nam cp874 - Thai - this . . .
. . .
- Baltic windows-1258 - Viet Nam cp874 - Thai - this file is also for DOS DOS cp437 - Latin US cp737 - Greek cp775 - BaltRim cp850 - Latin1 cp852 - Latin2 cp855 - Cyrylic cp857 - Turkish cp860 - Portuguese cp861 - Iceland cp862 - Hebrew cp863 - Canada cp864 - Arabic cp865 - Nordic cp866 - Cyrylic Russian (this is the one, used in IE . . .
. . .
IE Cyrillic (DOS) ) cp869 - Greek2 MAC (Apple) x-mac-cyrillic x-mac-greek x-mac-icelandic x-mac-ce x-mac-roman ISO iso-8859-1 iso-8859-2 iso-8859-3 iso-8859-4 iso-8859-5 iso-8859-6 iso-8859-7 iso-8859-8 iso-8859-9 iso-8859-10 iso-8859-11 iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 . . .
. . .
iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 cp500 cp856 cp875 cp1006 cp1026 koi8-r (Cyrillic) koi8-u (Cyrillic Ukrainian) nextstep us-ascii us-ascii-quotes DSP implementation for NeXT stdenc symbol zdingbat And specially for old Polish programs: mazovia This list is to be read . . .
. . .
Access denied; you need the RELOAD privilege. . . Top 10. Enable real-time output of logging data Up to version 1.5 of Sphider-plus, during index / re-index there was no printout available because: - Several servers, especially on Win32, buffer the output from the script until it terminates before transmitting the results to the browser. - . . .
. . .
is seen. - Some versions of Microsoft Internet Explorer only start to display the page after they have received 256 bytes of output. As progress was not presented during index / re- index procedure, waiting for results became a pain in the neck. Selectable in Admin setting together with the update interval (1 - 10 seconds), AJAX technology was . . .
. . .
characters in front of words Warning: This option should be used with special care and not be activated for non ISO-8859 charsets. Some special characters as part of the word ending might be erased by accidental. Top 13. Media search for images, audio streams and videos 13.1 Media indexing Index of media files is enabled by separated Admin . . .
. . .
and 'Found at' - Total clicks - Last clicked - Query input 'Indexed Image Thumbnails' presenting: - Thumbnail 150 x 100 pixel - Image details like title, filename size of original image, link- and thumb-id - Option to delete the thumbnail In order to open the media files all tables contain active links. Media results are also stored in . . .
. . .
Settings' menu of the admin backend: First option: For results of XML product feeds, present the text extract of 350 characters as part of product attributes. If this option is not activated, all products of the XML product feed will be presented in search results, and the hits of the query string will be highlighted in all involved products. . . .
. . .
content of the database tables (those with the same table prefix) will be destroyed by the restore procedure. 16.5 Copy & Move This section of the Database Management will present only those databases, which are correctly configured, assigned and do have a set of installed tables as described in chapter Definition and configuration. This . . .
. . .
results. Never the less to find these x results, the complete database had to been browsed. Starting with version 2.5, an additional clean option is offered as part of the Admin backend. Main advantage of this option is a significant reduction of the search time for any query, because the content of the db could be limited to offer only x . . .
. . .
<link>http://www.abc.de/images/warp.gif</link> <title>warp.gif</title> <x_size>635</x_size> <y_size>98</y_size> </media_result> <media_result> <num>2</num> <type>audio</type> <url>http://www.abc.de/index.php</url> . . .
. . .
the following content: {"query":"colour","ip":"::1","host_name":"guard_007-hoster","query_time":"2016-03-18 10:56:23 AM","consumed":0.016,"total_results":2,"num_of_results":2,"from":1,"to":2,"text_results":[{"num":1,"weight":"100.0","url":"http:\/\/www.english.le-piaggie.info\/Info_eng.pdf","title":" Info_eng","fulltxt":" . . . . . .
. . .
placed on the ground floor, today's big kitchen with the heating fireplace for open and close mode of . . . ","page_size":"5,661.6kb","domain_name":"www.english.le-piaggie.info"},{"num":2,"weight":"50.0","url":"http:\/\/www.english.le-piaggie.info\/html\/description.html","title":" Description","fulltxt":" . . . cable. Additional active . . .
. . .
D.O.C.G. as far as to the Pratomagno mountains 160 olive-trees Detached house, about 800 m to reach the . . . ","page_size":"26.4 kb","domain_name":"www.english.le-piaggie.info"}]} Top . . .


5a2 All required information. Introduction Release and Legal Info Installation Documentation Change Log [ Documentation Summary ] Preamble: The info presented here is valid only for the latest release of Sphider-plus. At present version 4.2024a published February 12, 2024 is the actual release. 1. Settings, customizing and statistics 2. Indexing . . .
. . .
2. Indexing 2.1 Various options 2.2 Allow other hosts in same domain 2.3 Word stemming 2.4 Periodical Re-indexing 2.5 Preferred indexing 2.6 Multithreaded indexing 2.7 Create thumbnails during index procedure 2.8 Prevent indexing of known malware and pishing pages 2.9 Follow and create sitemap files 2.10 Use private sitemap instead of global . . .
. . .
Sitemap file 3. Using the indexer from command line 3.1 All options 3.2 Multithreaded indexing 4. Keeping pages, words and files from being indexed 4.1 robots.txt 4.2 Must include / must not include string list 4.3 Ignoring links 4.4 Ignoring parts of a page by <! sphider_noindex > 4.5 Ignoring parts of a page by <div id='abc'> . . .
. . .
charset' 6. Search modes 6.1 Search with wildcards * 6.2 Strict search ! 6.3 Tolerant search 6.4 Link search 6.5 Media search 6.6 Search only in one domain 6.7 Search in categories 6.8 Greek language support 6.9 Block queries 7. Chronological order for result listing 7.1 Text result listing 7.2 Media result listing 8. PDF converter 9. Clean . . .
. . .
of logging data 11. Error messages and Debug mode 12. Delete secondary characters 13. Media search for images, audio streams and videos 13.1 Media indexing 13.2 Not supported media content 13.3 Search for media content 13.4 Statistics for media content 14. Feed support 14.1 XML product feeds 14.2 RDF, RSD, RSS and Atom feeds 15. Result . . .
. . .
Overview 16.2 Definition and configuration 16.3 Activate / disable database 16.4 Backup & Restore of databases 16.5 Copy & and Move 16.6 Enhancing functionality of multiple database support 17. Search in categories 17.1 Hierachical structure 17.2 Parallel structure 18. User suggested sites 19. Vulnerability protection 19.1 Prevent queries . . .
. . .
templates 22.2 Embed the search engine into existing HTML code 22.3 The different style sheet files 23. JSON, XML and RSS result output [ Documentation ] 1. Settings, customizing and statistics If you want to change settings, behavior and design of Sphider-plus, you can do so by means of the Admin interface. There is a wide range of . . .
. . .
interface. There is a wide range of settings foreseen for Sphider-plus. Separated into different submenus like: Sites: - Add Site - Index only the new - Re-index all - Re-index only preferred URLs - Erase Re-index (available also for individual URLs) - Import/export URL list - Approve sites - Banned domains Categories: - Add, edit, delete - . . .
. . .
URL list - Approve sites - Banned domains Categories: - Add, edit, delete - Create new subcategory under Index: - Basic indexing options - Advanced options Clean: - Clean keywords not associated with any link - Clean links not associated with any site - Clean Category table not associated with any site - Clean Media links - Clear Temp table . . .
. . .
- Clear Search log - Clear 'Most Popular Page Links' log - Clear 'Most Popular Media Links' log - Clear Spider log, separate and bulk delete - Clear Thumbnail images, separate and bulk delete - Clear Text cache - Clear Media cache - Clear IDS log file - Clear flood attempts log file - Clear all entries in addurl or banned table - Truncate all . . .
. . .
flood attempts log file - Clear all entries in addurl or banned table - Truncate all tables in database Settings: - General Settings - Index Log Settings - Spider Settings - Search Settings - Order of Result listing - Suggest Options - Page Indexing Weights Database: - Configure up to 5 databases with unlimited number of table sets - Activate . . .
. . .
Database: - Configure up to 5 databases with unlimited number of table sets - Activate separately for 'Admin', 'Search' user and 'Suggest URL user' - Backup / Restore - Copy / Move - Optimize Templates: In order to enable customer's integration of Sphider-plus into existing sites, HTML templates are prepared for Search form Text result . . .
. . .
Search form Text result listing Media result listing Most popular queries etc. Three different designs are offered, which may be selected in submenu 'Settings'. If the layout does not fit the design of your site (which is normal), you may create new designs and modify the appropriate file /templates/My_template/adminstyle.css . . .
. . .
the appropriate file /templates/My_template/adminstyle.css /templates/My_template/userstyle.css Statistics output: - Top keywords (Top 50 with hit counter). - All indexed thumbnails w 3e80 ith ID3 and EXIF info. - Larges pages offering link URL and file size. - Most Popular Searches for text links offering: Link addr., total clicks, last . . .
. . .
Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Searches for media links offering: Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Links (click counter). - Search log offering: Query, Results, Queried at, Time taken, User IP, Country, Host name (Latest100) - Index log offering: . . .
. . .
Country, Host name (Latest100) - Index log offering: File-name, index date and delete option - sitemap log offering: sitemap.xml output sitemap list offering file/page suffixes - IDS log offering: IP, host, query, impact, involved tags, date and time of intrusion. - Flood attempts log offering: IP, query, date and time of flood attempt. - Auto . . .
. . .
attempts log offering: IP, query, date and time of flood attempt. - Auto Re-index log file - Server info offering: Server software, environment, MySQL, PDF-converter, image functions, php.ini file PHP integration, PHP security info. Each item holding lists of details. All text links, media links and thumbnails are active linked. As stated in . . .
. . .
individual Re-indexing, the periodical Re-indexer could be started and aborted in the "Options" menu of each site. 2.5 Preferred Re-indexing Each new URL added to the Admin backend, could be supplied with a priority level. This level will be used by the option 'Re-index only preferred sites'. Level 1 will be interpreted as most important, while . . .
. . .
less all index results will be stored in log files in sub folder /admin/log/ The names of the log files look like: db2_100524-21.47.56_1.html (log file of first thread) db2_100524-21.48.12_2.html (log file of second thread) and is build by the following items: db2 - Number of database. 100524 - Date (May 24, 2010) 21.47.56 - Time when this thread . . .
. . .
spider.php -new1 php spider.php -new2 etc. The IDs will be added to the name of the corresponding log files like: db2_100524-21.47.56_ID1.html (log file of first thread) db2_100524-21.48.12_ID2.html (log file of second thread) IDs could be defined by personal requirements, but the limitations for file names with respect to the OS should be taken . . .
. . .
but will not erase the content of all the other tables. So the check whether the content of a page has changed (MD5sum) is still available for a fast re-index procedure. Once prepared, multithreaded re-indexing could be invoked by starting several threads and adding individual IDs to the option parameter like: php spider.php -erased1 php . . .
. . .
4.4 Ignoring parts of a page Sphider-plus includes an option to exclude parts of pages from being indexed. Thi 775e s can for example be used to prevent search result flooding when certain keywords appear on certain part in most pages (like a header, footer or a menu). Any part of a page between <! sphider_noindex > and <! . . .
. . .
<! sphider_noindex > and <! /sphider_noindex > tags is not indexed, however links in it are followed. 4.5 Ignoring parts of a page by <div id='abc'> Ignoring parts of a page by the <! sphider_noindex > tags requires direct access to the page, because the tags need to be added (edited) to the page. A more flexible method, . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ menu[0-5]] / 4.6 Indexing only parts of a page by <div id='abc'> If enabled in Admin settings, the values as defined in the list-file /include/common/divs_use.txt will be used to index only the content between <div . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ table[0-5] / 4.7 Ignore HTML elements defined by <tagname> . . </tagname> This option is foreseen to cooperate with the new HTML5 elements like section, nav, aside, hgroup, article, header, footer, etc. HTML elements . . .
. . .
contain a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */nav[0-5]/ Please keep in mind that element names placed in /include/common/elements_not.txt will be processed case-sensitive. 4.8 Index only HTML elements defined by <tagname> . . </tagname> This is the vice versa . . .
. . .
of all the duplicate content URLs: http://www.example.com/product.php?item=swedish-fish&category=gummy-candy http://www.example.com/product.php?item=swedish-fish&trackingid=1234&sessionid=5678 and Sphider-plus will understand that the duplicates all refer to the canonical URL: http://www.example.com/product.php?item=swedish-fish. The . . .
. . .
charset are supported by the ConvertCharset function and will be used to convert text into UTF-8 Unicode: WINDOWS windows-1250 - Central Europe windows-1251 - Cyrillic windows-1252 - Latin I windows-1253 - Greek windows-1254 - Turkish windows-1255 - Hebrew windows-1256 - Arabic windows-1257 - Baltic windows-1258 - Viet Nam cp874 - Thai - this . . .
. . .
- Baltic windows-1258 - Viet Nam cp874 - Thai - this file is also for DOS DOS cp437 - Latin US cp737 - Greek cp775 - BaltRim cp850 - Latin1 cp852 - Latin2 cp855 - Cyrylic cp857 - Turkish cp860 - Portuguese cp861 - Iceland cp862 - Hebrew cp863 - Canada cp864 - Arabic cp865 - Nordic cp866 - Cyrylic Russian (this is the one, used in IE . . .
. . .
IE Cyrillic (DOS) ) cp869 - Greek2 MAC (Apple) x-mac-cyrillic x-mac-greek x-mac-icelandic x-mac-ce x-mac-roman ISO iso-8859-1 iso-8859-2 iso-8859-3 iso-8859-4 iso-8859-5 iso-8859-6 iso-8859-7 iso-8859-8 iso-8859-9 iso-8859-10 iso-8859-11 iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 . . .
. . .
iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 cp500 cp856 cp875 cp1006 cp1026 koi8-r (Cyrillic) koi8-u (Cyrillic Ukrainian) nextstep us-ascii us-ascii-quotes DSP implementation for NeXT stdenc symbol zdingbat And specially for old Polish programs: mazovia This list is to be read . . .
. . .
Access denied; you need the RELOAD privilege. . . Top 10. Enable real-time output of logging data Up to version 1.5 of Sphider-plus, during index / re-index there was no printout available because: - Several servers, especially on Win32, buffer the output from the script until it terminates before transmitting the results to the browser. - . . .
. . .
is seen. - Some versions of Microsoft Internet Explorer only start to display the page after they have received 256 bytes of output. As progress was not presented during index / re- index procedure, waiting for results became a pain in the neck. Selectable in Admin setting together with the update interval (1 - 10 seconds), AJAX technology was . . .
. . .
characters in front of words Warning: This option should be used with special care and not be activated for non ISO-8859 charsets. Some special characters as part of the word ending might be erased by accidental. Top 13. Media search for images, audio streams and videos 13.1 Media indexing Index of media files is enabled by separated Admin . . .
. . .
and 'Found at' - Total clicks - Last clicked - Query input 'Indexed Image Thumbnails' presenting: - Thumbnail 150 x 100 pixel - Image details like title, filename size of original image, link- and thumb-id - Option to delete the thumbnail In order to open the media files all tables contain active links. Media results are also stored in . . .
. . .
Settings' menu of the admin backend: First option: For results of XML product feeds, present the text extract of 350 characters as part of product attributes. If this option is not activated, all products of the XML product feed will be presented in search results, and the hits of the query string will be highlighted in all involved products. . . .
. . .
content of the database tables (those with the same table prefix) will be destroyed by the restore procedure. 16.5 Copy & Move This section of the Database Management will present only those databases, which are correctly configured, assigned and do have a set of installed tables as described in chapter Definition and configuration. This . . .
. . .
results. Never the less to find these x results, the complete database had to been browsed. Starting with version 2.5, an additional clean option is offered as part of the Admin backend. Main advantage of this option is a significant reduction of the search time for any query, because the content of the db could be limited to offer only x . . .
. . .
<link>http://www.abc.de/images/warp.gif</link> <title>warp.gif</title> <x_size>635</x_size> <y_size>98</y_size> </media_result> <media_result> <num>2</num> <type>audio</type> <url>http://www.abc.de/index.php</url> . . .
. . .
the following content: {"query":"colour","ip":"::1","host_name":"guard_007-hoster","query_time":"2016-03-18 10:56:23 AM","consumed":0.016,"total_results":2,"num_of_results":2,"from":1,"to":2,"text_results":[{"num":1,"weight":"100.0","url":"http:\/\/www.english.le-piaggie.info\/Info_eng.pdf","title":" Info_eng","fulltxt":" . . . . . .
. . .
placed on the ground floor, today's big kitchen with the heating fireplace for open and close mode of . . . ","page_size":"5,661.6kb","domain_name":"www.english.le-piaggie.info"},{"num":2,"weight":"50.0","url":"http:\/\/www.english.le-piaggie.info\/html\/description.html","title":" Description","fulltxt":" . . . cable. Additional active . . .
. . .
D.O.C.G. as far as to the Pratomagno mountains 160 olive-trees Detached house, about 800 m to reach the . . . ","page_size":"26.4 kb","domain_name":"www.english.le-piaggie.info"}]} Top . . .


5a2 All required information. Introduction Release and Legal Info Installation Documentation Change Log [ Documentation Summary ] Preamble: The info presented here is valid only for the latest release of Sphider-plus. At present version 4.2024a published February 12, 2024 is the actual release. 1. Settings, customizing and statistics 2. Indexing . . .
. . .
2. Indexing 2.1 Various options 2.2 Allow other hosts in same domain 2.3 Word stemming 2.4 Periodical Re-indexing 2.5 Preferred indexing 2.6 Multithreaded indexing 2.7 Create thumbnails during index procedure 2.8 Prevent indexing of known malware and pishing pages 2.9 Follow and create sitemap files 2.10 Use private sitemap instead of global . . .
. . .
Sitemap file 3. Using the indexer from command line 3.1 All options 3.2 Multithreaded indexing 4. Keeping pages, words and files from being indexed 4.1 robots.txt 4.2 Must include / must not include string list 4.3 Ignoring links 4.4 Ignoring parts of a page by <! sphider_noindex > 4.5 Ignoring parts of a page by <div id='abc'> . . .
. . .
charset' 6. Search modes 6.1 Search with wildcards * 6.2 Strict search ! 6.3 Tolerant search 6.4 Link search 6.5 Media search 6.6 Search only in one domain 6.7 Search in categories 6.8 Greek language support 6.9 Block queries 7. Chronological order for result listing 7.1 Text result listing 7.2 Media result listing 8. PDF converter 9. Clean . . .
. . .
of logging data 11. Error messages and Debug mode 12. Delete secondary characters 13. Media search for images, audio streams and videos 13.1 Media indexing 13.2 Not supported media content 13.3 Search for media content 13.4 Statistics for media content 14. Feed support 14.1 XML product feeds 14.2 RDF, RSD, RSS and Atom feeds 15. Result . . .
. . .
Overview 16.2 Definition and configuration 16.3 Activate / disable database 16.4 Backup & Restore of databases 16.5 Copy & and Move 16.6 Enhancing functionality of multiple database support 17. Search in categories 17.1 Hierachical structure 17.2 Parallel structure 18. User suggested sites 19. Vulnerability protection 19.1 Prevent queries . . .
. . .
templates 22.2 Embed the search engine into existing HTML code 22.3 The different style sheet files 23. JSON, XML and RSS result output [ Documentation ] 1. Settings, customizing and statistics If you want to change settings, behavior and design of Sphider-plus, you can do so by means of the Admin interface. There is a wide range of . . .
. . .
interface. There is a wide range of settings foreseen for Sphider-plus. Separated into different submenus like: Sites: - Add Site - Index only the new - Re-index all - Re-index only preferred URLs - Erase Re-index (available also for individual URLs) - Import/export URL list - Approve sites - Banned domains Categories: - Add, edit, delete - . . .
. . .
URL list - Approve sites - Banned domains Categories: - Add, edit, delete - Create new subcategory under Index: - Basic indexing options - Advanced options Clean: - Clean keywords not associated with any link - Clean links not associated with any site - Clean Category table not associated with any site - Clean Media links - Clear Temp table . . .
. . .
- Clear Search log - Clear 'Most Popular Page Links' log - Clear 'Most Popular Media Links' log - Clear Spider log, separate and bulk delete - Clear Thumbnail images, separate and bulk delete - Clear Text cache - Clear Media cache - Clear IDS log file - Clear flood attempts log file - Clear all entries in addurl or banned table - Truncate all . . .
. . .
flood attempts log file - Clear all entries in addurl or banned table - Truncate all tables in database Settings: - General Settings - Index Log Settings - Spider Settings - Search Settings - Order of Result listing - Suggest Options - Page Indexing Weights Database: - Configure up to 5 databases with unlimited number of table sets - Activate . . .
. . .
Database: - Configure up to 5 databases with unlimited number of table sets - Activate separately for 'Admin', 'Search' user and 'Suggest URL user' - Backup / Restore - Copy / Move - Optimize Templates: In order to enable customer's integration of Sphider-plus into existing sites, HTML templates are prepared for Search form Text result . . .
. . .
Search form Text result listing Media result listing Most popular queries etc. Three different designs are offered, which may be selected in submenu 'Settings'. If the layout does not fit the design of your site (which is normal), you may create new designs and modify the appropriate file /templates/My_template/adminstyle.css . . .
. . .
the appropriate file /templates/My_template/adminstyle.css /templates/My_template/userstyle.css Statistics output: - Top keywords (Top 50 with hit counter). - All indexed thumbnails w 3e80 ith ID3 and EXIF info. - Larges pages offering link URL and file size. - Most Popular Searches for text links offering: Link addr., total clicks, last . . .
. . .
Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Searches for media links offering: Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Links (click counter). - Search log offering: Query, Results, Queried at, Time taken, User IP, Country, Host name (Latest100) - Index log offering: . . .
. . .
Country, Host name (Latest100) - Index log offering: File-name, index date and delete option - sitemap log offering: sitemap.xml output sitemap list offering file/page suffixes - IDS log offering: IP, host, query, impact, involved tags, date and time of intrusion. - Flood attempts log offering: IP, query, date and time of flood attempt. - Auto . . .
. . .
attempts log offering: IP, query, date and time of flood attempt. - Auto Re-index log file - Server info offering: Server software, environment, MySQL, PDF-converter, image functions, php.ini file PHP integration, PHP security info. Each item holding lists of details. All text links, media links and thumbnails are active linked. As stated in . . .
. . .
individual Re-indexing, the periodical Re-indexer could be started and aborted in the "Options" menu of each site. 2.5 Preferred Re-indexing Each new URL added to the Admin backend, could be supplied with a priority level. This level will be used by the option 'Re-index only preferred sites'. Level 1 will be interpreted as most important, while . . .
. . .
less all index results will be stored in log files in sub folder /admin/log/ The names of the log files look like: db2_100524-21.47.56_1.html (log file of first thread) db2_100524-21.48.12_2.html (log file of second thread) and is build by the following items: db2 - Number of database. 100524 - Date (May 24, 2010) 21.47.56 - Time when this thread . . .
. . .
spider.php -new1 php spider.php -new2 etc. The IDs will be added to the name of the corresponding log files like: db2_100524-21.47.56_ID1.html (log file of first thread) db2_100524-21.48.12_ID2.html (log file of second thread) IDs could be defined by personal requirements, but the limitations for file names with respect to the OS should be taken . . .
. . .
but will not erase the content of all the other tables. So the check whether the content of a page has changed (MD5sum) is still available for a fast re-index procedure. Once prepared, multithreaded re-indexing could be invoked by starting several threads and adding individual IDs to the option parameter like: php spider.php -erased1 php . . .
. . .
4.4 Ignoring parts of a page Sphider-plus includes an option to exclude parts of pages from being indexed. Thi 775e s can for example be used to prevent search result flooding when certain keywords appear on certain part in most pages (like a header, footer or a menu). Any part of a page between <! sphider_noindex > and <! . . .
. . .
<! sphider_noindex > and <! /sphider_noindex > tags is not indexed, however links in it are followed. 4.5 Ignoring parts of a page by <div id='abc'> Ignoring parts of a page by the <! sphider_noindex > tags requires direct access to the page, because the tags need to be added (edited) to the page. A more flexible method, . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ menu[0-5] / 4.6 Indexing only parts of a page by <div id='abc'> If enabled in Admin settings, the values as defined in the list-file /include/common/divs_use.txt will be used to index only the content between <div . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ table[0-5] / 4.7 Ignore HTML elements defined by <tagname> . . </tagname> This option is foreseen to cooperate with the new HTML5 elements like section, nav, aside, hgroup, article, header, footer, etc. HTML elements . . .
. . .
contain a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */nav[0-5]/]/ Please keep in mind that element names placed in /include/common/elements_not.txt will be processed case-sensitive. 4.8 Index only HTML elements defined by <tagname> . . </tagname> This is the vice versa . . .
. . .
of all the duplicate content URLs: http://www.example.com/product.php?item=swedish-fish&category=gummy-candy http://www.example.com/product.php?item=swedish-fish&trackingid=1234&sessionid=5678 and Sphider-plus will understand that the duplicates all refer to the canonical URL: http://www.example.com/product.php?item=swedish-fish. The . . .
. . .
charset are supported by the ConvertCharset function and will be used to convert text into UTF-8 Unicode: WINDOWS windows-1250 - Central Europe windows-1251 - Cyrillic windows-1252 - Latin I windows-1253 - Greek windows-1254 - Turkish windows-1255 - Hebrew windows-1256 - Arabic windows-1257 - Baltic windows-1258 - Viet Nam cp874 - Thai - this . . .
. . .
- Baltic windows-1258 - Viet Nam cp874 - Thai - this file is also for DOS DOS cp437 - Latin US cp737 - Greek cp775 - BaltRim cp850 - Latin1 cp852 - Latin2 cp855 - Cyrylic cp857 - Turkish cp860 - Portuguese cp861 - Iceland cp862 - Hebrew cp863 - Canada cp864 - Arabic cp865 - Nordic cp866 - Cyrylic Russian (this is the one, used in IE . . .
. . .
IE Cyrillic (DOS) ) cp869 - Greek2 MAC (Apple) x-mac-cyrillic x-mac-greek x-mac-icelandic x-mac-ce x-mac-roman ISO iso-8859-1 iso-8859-2 iso-8859-3 iso-8859-4 iso-8859-5 iso-8859-6 iso-8859-7 iso-8859-8 iso-8859-9 iso-8859-10 iso-8859-11 iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 . . .
. . .
iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 cp500 cp856 cp875 cp1006 cp1026 koi8-r (Cyrillic) koi8-u (Cyrillic Ukrainian) nextstep us-ascii us-ascii-quotes DSP implementation for NeXT stdenc symbol zdingbat And specially for old Polish programs: mazovia This list is to be read . . .
. . .
Access denied; you need the RELOAD privilege. . . Top 10. Enable real-time output of logging data Up to version 1.5 of Sphider-plus, during index / re-index there was no printout available because: - Several servers, especially on Win32, buffer the output from the script until it terminates before transmitting the results to the browser. - . . .
. . .
is seen. - Some versions of Microsoft Internet Explorer only start to display the page after they have received 256 bytes of output. As progress was not presented during index / re- index procedure, waiting for results became a pain in the neck. Selectable in Admin setting together with the update interval (1 - 10 seconds), AJAX technology was . . .
. . .
characters in front of words Warning: This option should be used with special care and not be activated for non ISO-8859 charsets. Some special characters as part of the word ending might be erased by accidental. Top 13. Media search for images, audio streams and videos 13.1 Media indexing Index of media files is enabled by separated Admin . . .
. . .
and 'Found at' - Total clicks - Last clicked - Query input 'Indexed Image Thumbnails' presenting: - Thumbnail 150 x 100 pixel - Image details like title, filename size of original image, link- and thumb-id - Option to delete the thumbnail In order to open the media files all tables contain active links. Media results are also stored in . . .
. . .
Settings' menu of the admin backend: First option: For results of XML product feeds, present the text extract of 350 characters as part of product attributes. If this option is not activated, all products of the XML product feed will be presented in search results, and the hits of the query string will be highlighted in all involved products. . . .
. . .
content of the database tables (those with the same table prefix) will be destroyed by the restore procedure. 16.5 Copy & Move This section of the Database Management will present only those databases, which are correctly configured, assigned and do have a set of installed tables as described in chapter Definition and configuration. This . . .
. . .
results. Never the less to find these x results, the complete database had to been browsed. Starting with version 2.5, an additional clean option is offered as part of the Admin backend. Main advantage of this option is a significant reduction of the search time for any query, because the content of the db could be limited to offer only x . . .
. . .
<link>http://www.abc.de/images/warp.gif</link> <title>warp.gif</title> <x_size>635</x_size> <y_size>98</y_size> </media_result> <media_result> <num>2</num> <type>audio</type> <url>http://www.abc.de/index.php</url> . . .
. . .
the following content: {"query":"colour","ip":"::1","host_name":"guard_007-hoster","query_time":"2016-03-18 10:56:23 AM","consumed":0.016,"total_results":2,"num_of_results":2,"from":1,"to":2,"text_results":[{"num":1,"weight":"100.0","url":"http:\/\/www.english.le-piaggie.info\/Info_eng.pdf","title":" Info_eng","fulltxt":" . . . . . .
. . .
placed on the ground floor, today's big kitchen with the heating fireplace for open and close mode of . . . ","page_size":"5,661.6kb","domain_name":"www.english.le-piaggie.info"},{"num":2,"weight":"50.0","url":"http:\/\/www.english.le-piaggie.info\/html\/description.html","title":" Description","fulltxt":" . . . cable. Additional active . . .
. . .
D.O.C.G. as far as to the Pratomagno mountains 160 olive-trees Detached house, about 800 m to reach the . . . ","page_size":"26.4 kb","domain_name":"www.english.le-piaggie.info"}]} Top . . .


5a2 All required information. Introduction Release and Legal Info Installation Documentation Change Log [ Documentation Summary ] Preamble: The info presented here is valid only for the latest release of Sphider-plus. At present version 4.2024a published February 12, 2024 is the actual release. 1. Settings, customizing and statistics 2. Indexing . . .
. . .
2. Indexing 2.1 Various options 2.2 Allow other hosts in same domain 2.3 Word stemming 2.4 Periodical Re-indexing 2.5 Preferred indexing 2.6 Multithreaded indexing 2.7 Create thumbnails during index procedure 2.8 Prevent indexing of known malware and pishing pages 2.9 Follow and create sitemap files 2.10 Use private sitemap instead of global . . .
. . .
Sitemap file 3. Using the indexer from command line 3.1 All options 3.2 Multithreaded indexing 4. Keeping pages, words and files from being indexed 4.1 robots.txt 4.2 Must include / must not include string list 4.3 Ignoring links 4.4 Ignoring parts of a page by <! sphider_noindex > 4.5 Ignoring parts of a page by <div id='abc'> . . .
. . .
charset' 6. Search modes 6.1 Search with wildcards * 6.2 Strict search ! 6.3 Tolerant search 6.4 Link search 6.5 Media search 6.6 Search only in one domain 6.7 Search in categories 6.8 Greek language support 6.9 Block queries 7. Chronological order for result listing 7.1 Text result listing 7.2 Media result listing 8. PDF converter 9. Clean . . .
. . .
of logging data 11. Error messages and Debug mode 12. Delete secondary characters 13. Media search for images, audio streams and videos 13.1 Media indexing 13.2 Not supported media content 13.3 Search for media content 13.4 Statistics for media content 14. Feed support 14.1 XML product feeds 14.2 RDF, RSD, RSS and Atom feeds 15. Result . . .
. . .
Overview 16.2 Definition and configuration 16.3 Activate / disable database 16.4 Backup & Restore of databases 16.5 Copy & and Move 16.6 Enhancing functionality of multiple database support 17. Search in categories 17.1 Hierachical structure 17.2 Parallel structure 18. User suggested sites 19. Vulnerability protection 19.1 Prevent queries . . .
. . .
templates 22.2 Embed the search engine into existing HTML code 22.3 The different style sheet files 23. JSON, XML and RSS result output [ Documentation ] 1. Settings, customizing and statistics If you want to change settings, behavior and design of Sphider-plus, you can do so by means of the Admin interface. There is a wide range of . . .
. . .
interface. There is a wide range of settings foreseen for Sphider-plus. Separated into different submenus like: Sites: - Add Site - Index only the new - Re-index all - Re-index only preferred URLs - Erase Re-index (available also for individual URLs) - Import/export URL list - Approve sites - Banned domains Categories: - Add, edit, delete - . . .
. . .
URL list - Approve sites - Banned domains Categories: - Add, edit, delete - Create new subcategory under Index: - Basic indexing options - Advanced options Clean: - Clean keywords not associated with any link - Clean links not associated with any site - Clean Category table not associated with any site - Clean Media links - Clear Temp table . . .
. . .
- Clear Search log - Clear 'Most Popular Page Links' log - Clear 'Most Popular Media Links' log - Clear Spider log, separate and bulk delete - Clear Thumbnail images, separate and bulk delete - Clear Text cache - Clear Media cache - Clear IDS log file - Clear flood attempts log file - Clear all entries in addurl or banned table - Truncate all . . .
. . .
flood attempts log file - Clear all entries in addurl or banned table - Truncate all tables in database Settings: - General Settings - Index Log Settings - Spider Settings - Search Settings - Order of Result listing - Suggest Options - Page Indexing Weights Database: - Configure up to 5 databases with unlimited number of table sets - Activate . . .
. . .
Database: - Configure up to 5 databases with unlimited number of table sets - Activate separately for 'Admin', 'Search' user and 'Suggest URL user' - Backup / Restore - Copy / Move - Optimize Templates: In order to enable customer's integration of Sphider-plus into existing sites, HTML templates are prepared for Search form Text result . . .
. . .
Search form Text result listing Media result listing Most popular queries etc. Three different designs are offered, which may be selected in submenu 'Settings'. If the layout does not fit the design of your site (which is normal), you may create new designs and modify the appropriate file /templates/My_template/adminstyle.css . . .
. . .
the appropriate file /templates/My_template/adminstyle.css /templates/My_template/userstyle.css Statistics output: - Top keywords (Top 50 with hit counter). - All indexed thumbnails w 3e80 ith ID3 and EXIF info. - Larges pages offering link URL and file size. - Most Popular Searches for text links offering: Link addr., total clicks, last . . .
. . .
Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Searches for media links offering: Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Links (click counter). - Search log offering: Query, Results, Queried at, Time taken, User IP, Country, Host name (Latest100) - Index log offering: . . .
. . .
Country, Host name (Latest100) - Index log offering: File-name, index date and delete option - sitemap log offering: sitemap.xml output sitemap list offering file/page suffixes - IDS log offering: IP, host, query, impact, involved tags, date and time of intrusion. - Flood attempts log offering: IP, query, date and time of flood attempt. - Auto . . .
. . .
attempts log offering: IP, query, date and time of flood attempt. - Auto Re-index log file - Server info offering: Server software, environment, MySQL, PDF-converter, image functions, php.ini file PHP integration, PHP security info. Each item holding lists of details. All text links, media links and thumbnails are active linked. As stated in . . .
. . .
individual Re-indexing, the periodical Re-indexer could be started and aborted in the "Options" menu of each site. 2.5 Preferred Re-indexing Each new URL added to the Admin backend, could be supplied with a priority level. This level will be used by the option 'Re-index only preferred sites'. Level 1 will be interpreted as most important, while . . .
. . .
less all index results will be stored in log files in sub folder /admin/log/ The names of the log files look like: db2_100524-21.47.56_1.html (log file of first thread) db2_100524-21.48.12_2.html (log file of second thread) and is build by the following items: db2 - Number of database. 100524 - Date (May 24, 2010) 21.47.56 - Time when this thread . . .
. . .
spider.php -new1 php spider.php -new2 etc. The IDs will be added to the name of the corresponding log files like: db2_100524-21.47.56_ID1.html (log file of first thread) db2_100524-21.48.12_ID2.html (log file of second thread) IDs could be defined by personal requirements, but the limitations for file names with respect to the OS should be taken . . .
. . .
but will not erase the content of all the other tables. So the check whether the content of a page has changed (MD5sum) is still available for a fast re-index procedure. Once prepared, multithreaded re-indexing could be invoked by starting several threads and adding individual IDs to the option parameter like: php spider.php -erased1 php . . .
. . .
4.4 Ignoring parts of a page Sphider-plus includes an option to exclude parts of pages from being indexed. Thi 775e s can for example be used to prevent search result flooding when certain keywords appear on certain part in most pages (like a header, footer or a menu). Any part of a page between <! sphider_noindex > and <! . . .
. . .
<! sphider_noindex > and <! /sphider_noindex > tags is not indexed, however links in it are followed. 4.5 Ignoring parts of a page by <div id='abc'> Ignoring parts of a page by the <! sphider_noindex > tags requires direct access to the page, because the tags need to be added (edited) to the page. A more flexible method, . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ menu[0-5] / 4.6 Indexing only parts of a page by <div id='abc'> If enabled in Admin settings, the values as defined in the list-file /include/common/divs_use.txt will be used to index only the content between <div . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ table[0-5] / 4.7 Ignore HTML elements defined by <tagname> . . </tagname> This option is foreseen to cooperate with the new HTML5 elements like section, nav, aside, hgroup, article, header, footer, etc. HTML elements . . .
. . .
contain a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */nav[0-5]/ Please keep in mind that element names placed in /include/common/elements_not.txt will be processed case-sensitive. 4.8 Index only HTML elements defined by <tagname> . . </tagname> This is the vice versa . . .
. . .
of all the duplicate content URLs: http://www.example.com/product.php?item=swedish-fish&category=gummy-candy http://www.example.com/product.php?item=swedish-fish&trackingid=1234&sessionid=5678 and Sphider-plus will understand that the duplicates all refer to the canonical URL: http://www.example.com/product.php?item=swedish-fish. The . . .
. . .
charset are supported by the ConvertCharset function and will be used to convert text into UTF-8 Unicode: WINDOWS windows-1250 - Central Europe windows-1251 - Cyrillic windows-1252 - Latin I windows-1253 - Greek windows-1254 - Turkish windows-1255 - Hebrew windows-1256 - Arabic windows-1257 - Baltic windows-1258 - Viet Nam cp874 - Thai - this . . .
. . .
- Baltic windows-1258 - Viet Nam cp874 - Thai - this file is also for DOS DOS cp437 - Latin US cp737 - Greek cp775 - BaltRim cp850 - Latin1 cp852 - Latin2 cp855 - Cyrylic cp857 - Turkish cp860 - Portuguese cp861 - Iceland cp862 - Hebrew cp863 - Canada cp864 - Arabic cp865 - Nordic cp866 - Cyrylic Russian (this is the one, used in IE . . .
. . .
IE Cyrillic (DOS) ) cp869 - Greek2 MAC (Apple) x-mac-cyrillic x-mac-greek x-mac-icelandic x-mac-ce x-mac-roman ISO iso-8859-1 iso-8859-2 iso-8859-3 iso-8859-4 iso-8859-5 iso-8859-6 iso-8859-7 iso-8859-8 iso-8859-9 iso-8859-10 iso-8859-11 iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 . . .
. . .
iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 cp500 cp856 cp875 cp1006 cp1026 koi8-r (Cyrillic) koi8-u (Cyrillic Ukrainian) nextstep us-ascii us-ascii-quotes DSP implementation for NeXT stdenc symbol zdingbat And specially for old Polish programs: mazovia This list is to be read . . .
. . .
Access denied; you need the RELOAD privilege. . . Top 10. Enable real-time output of logging data Up to version 1.5 of Sphider-plus, during index / re-index there was no printout available because: - Several servers, especially on Win32, buffer the output from the script until it terminates before transmitting the results to the browser. - . . .
. . .
is seen. - Some versions of Microsoft Internet Explorer only start to display the page after they have received 256 bytes of output. As progress was not presented during index / re- index procedure, waiting for results became a pain in the neck. Selectable in Admin setting together with the update interval (1 - 10 seconds), AJAX technology was . . .
. . .
characters in front of words Warning: This option should be used with special care and not be activated for non ISO-8859 charsets. Some special characters as part of the word ending might be erased by accidental. Top 13. Media search for images, audio streams and videos 13.1 Media indexing Index of media files is enabled by separated Admin . . .
. . .
and 'Found at' - Total clicks - Last clicked - Query input 'Indexed Image Thumbnails' presenting: - Thumbnail 150 x 100 pixel - Image details like title, filename size of original image, link- and thumb-id - Option to delete the thumbnail In order to open the media files all tables contain active links. Media results are also stored in . . .
. . .
Settings' menu of the admin backend: First option: For results of XML product feeds, present the text extract of 350 characters as part of product attributes. If this option is not activated, all products of the XML product feed will be presented in search results, and the hits of the query string will be highlighted in all involved products. . . .
. . .
content of the database tables (those with the same table prefix) will be destroyed by the restore procedure. 16.5 Copy & Move This section of the Database Management will present only those databases, which are correctly configured, assigned and do have a set of installed tables as described in chapter Definition and configuration. This . . .
. . .
results. Never the less to find these x results, the complete database had to been browsed. Starting with version 2.5, an additional clean option is offered as part of the Admin backend. Main advantage of this option is a significant reduction of the search time for any query, because the content of the db could be limited to offer only x . . .
. . .
<link>http://www.abc.de/images/warp.gif</link> <title>warp.gif</title> <x_size>635</x_size> <y_size>98</y_size> </media_result> <media_result> <num>2</num> <type>audio</type> <url>http://www.abc.de/index.php</url> . . .
. . .
the following content: {"query":"colour","ip":"::1","host_name":"guard_007-hoster","query_time":"2016-03-18 10:56:23 AM","consumed":0.016,"total_results":2,"num_of_results":2,"from":1,"to":2,"text_results":[{"num":1,"weight":"100.0","url":"http:\/\/www.english.le-piaggie.info\/Info_eng.pdf","title":" Info_eng","fulltxt":" . . . . . .
. . .
placed on the ground floor, today's big kitchen with the heating fireplace for open and close mode of . . . ","pagesize56616kbdomainnamewwwenglishle-piaggieinfo}{num2weight500urlhttp\/\/wwwenglishle-piaggieinfo\/html\/descriptionhtmltitle":"5,661.6kb","domain_name":"www.english.le-piaggie.info"},{"num":2,"weight":"50.0","url":"http:\/\/www.english.le-piaggie.info\/html\/description.html","title":" Description","fulltxt":" . . . cable. Additional active . . .
. . .
D.O.C.G. as far as to the Pratomagno mountains 160 olive-trees Detached house, about 800 m to reach the . . . ","pagesize56616kbdomainnamewwwenglishle-piaggieinfo}{num2weight500urlhttp\/\/wwwenglishle-piaggieinfo\/html\/descriptionhtmltitle":"26.4 kb","domain_name":"www.english.le-piaggie.info"}]} Top . . .


5a2 All required information. Introduction Release and Legal Info Installation Documentation Change Log [ Documentation Summary ] Preamble: The info presented here is valid only for the latest release of Sphider-plus. At present version 4.2024a published February 12, 2024 is the actual release. 1. Settings, customizing and statistics 2. Indexing . . .
. . .
2. Indexing 2.1 Various options 2.2 Allow other hosts in same domain 2.3 Word stemming 2.4 Periodical Re-indexing 2.5 Preferred indexing 2.6 Multithreaded indexing 2.7 Create thumbnails during index procedure 2.8 Prevent indexing of known malware and pishing pages 2.9 Follow and create sitemap files 2.10 Use private sitemap instead of global . . .
. . .
Sitemap file 3. Using the indexer from command line 3.1 All options 3.2 Multithreaded indexing 4. Keeping pages, words and files from being indexed 4.1 robots.txt 4.2 Must include / must not include string list 4.3 Ignoring links 4.4 Ignoring parts of a page by <! sphider_noindex > 4.5 Ignoring parts of a page by <div id='abc'> . . .
. . .
charset' 6. Search modes 6.1 Search with wildcards * 6.2 Strict search ! 6.3 Tolerant search 6.4 Link search 6.5 Media search 6.6 Search only in one domain 6.7 Search in categories 6.8 Greek language support 6.9 Block queries 7. Chronological order for result listing 7.1 Text result listing 7.2 Media result listing 8. PDF converter 9. Clean . . .
. . .
of logging data 11. Error messages and Debug mode 12. Delete secondary characters 13. Media search for images, audio streams and videos 13.1 Media indexing 13.2 Not supported media content 13.3 Search for media content 13.4 Statistics for media content 14. Feed support 14.1 XML product feeds 14.2 RDF, RSD, RSS and Atom feeds 15. Result . . .
. . .
Overview 16.2 Definition and configuration 16.3 Activate / disable database 16.4 Backup & Restore of databases 16.5 Copy & and Move 16.6 Enhancing functionality of multiple database support 17. Search in categories 17.1 Hierachical structure 17.2 Parallel structure 18. User suggested sites 19. Vulnerability protection 19.1 Prevent queries . . .
. . .
templates 22.2 Embed the search engine into existing HTML code 22.3 The different style sheet files 23. JSON, XML and RSS result output [ Documentation ] 1. Settings, customizing and statistics If you want to change settings, behavior and design of Sphider-plus, you can do so by means of the Admin interface. There is a wide range of . . .
. . .
interface. There is a wide range of settings foreseen for Sphider-plus. Separated into different submenus like: Sites: - Add Site - Index only the new - Re-index all - Re-index only preferred URLs - Erase Re-index (available also for individual URLs) - Import/export URL list - Approve sites - Banned domains Categories: - Add, edit, delete - . . .
. . .
URL list - Approve sites - Banned domains Categories: - Add, edit, delete - Create new subcategory under Index: - Basic indexing options - Advanced options Clean: - Clean keywords not associated with any link - Clean links not associated with any site - Clean Category table not associated with any site - Clean Media links - Clear Temp table . . .
. . .
- Clear Search log - Clear 'Most Popular Page Links' log - Clear 'Most Popular Media Links' log - Clear Spider log, separate and bulk delete - Clear Thumbnail images, separate and bulk delete - Clear Text cache - Clear Media cache - Clear IDS log file - Clear flood attempts log file - Clear all entries in addurl or banned table - Truncate all . . .
. . .
flood attempts log file - Clear all entries in addurl or banned table - Truncate all tables in database Settings: - General Settings - Index Log Settings - Spider Settings - Search Settings - Order of Result listing - Suggest Options - Page Indexing Weights Database: - Configure up to 5 databases with unlimited number of table sets - Activate . . .
. . .
Database: - Configure up to 5 databases with unlimited number of table sets - Activate separately for 'Admin', 'Search' user and 'Suggest URL user' - Backup / Restore - Copy / Move - Optimize Templates: In order to enable customer's integration of Sphider-plus into existing sites, HTML templates are prepared for Search form Text result . . .
. . .
Search form Text result listing Media result listing Most popular queries etc. Three different designs are offered, which may be selected in submenu 'Settings'. If the layout does not fit the design of your site (which is normal), you may create new designs and modify the appropriate file /templates/My_template/adminstyle.css . . .
. . .
the appropriate file /templates/My_template/adminstyle.css /templates/My_template/userstyle.css Statistics output: - Top keywords (Top 50 with hit counter). - All indexed thumbnails w 3e80 ith ID3 and EXIF info. - Larges pages offering link URL and file size. - Most Popular Searches for text links offering: Link addr., total clicks, last . . .
. . .
Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Searches for media links offering: Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Links (click counter). - Search log offering: Query, Results, Queried at, Time taken, User IP, Country, Host name (Latest100) - Index log offering: . . .
. . .
Country, Host name (Latest100) - Index log offering: File-name, index date and delete option - sitemap log offering: sitemap.xml output sitemap list offering file/page suffixes - IDS log offering: IP, host, query, impact, involved tags, date and time of intrusion. - Flood attempts log offering: IP, query, date and time of flood attempt. - Auto . . .
. . .
attempts log offering: IP, query, date and time of flood attempt. - Auto Re-index log file - Server info offering: Server software, environment, MySQL, PDF-converter, image functions, php.ini file PHP integration, PHP security info. Each item holding lists of details. All text links, media links and thumbnails are active linked. As stated in . . .
. . .
individual Re-indexing, the periodical Re-indexer could be started and aborted in the "Options" menu of each site. 2.5 Preferred Re-indexing Each new URL added to the Admin backend, could be supplied with a priority level. This level will be used by the option 'Re-index only preferred sites'. Level 1 will be interpreted as most important, while . . .
. . .
less all index results will be stored in log files in sub folder /admin/log/ The names of the log files look like: db2_100524-21.47.56_1.html (log file of first thread) db2_100524-21.48.12_2.html (log file of second thread) and is build by the following items: db2 - Number of database. 100524 - Date (May 24, 2010) 21.47.56 - Time when this thread . . .
. . .
spider.php -new1 php spider.php -new2 etc. The IDs will be added to the name of the corresponding log files like: db2_100524-21.47.56_ID1.html (log file of first thread) db2_100524-21.48.12_ID2.html (log file of second thread) IDs could be defined by personal requirements, but the limitations for file names with respect to the OS should be taken . . .
. . .
but will not erase the content of all the other tables. So the check whether the content of a page has changed (MD5sum) is still available for a fast re-index procedure. Once prepared, multithreaded re-indexing could be invoked by starting several threads and adding individual IDs to the option parameter like: php spider.php -erased1 php . . .
. . .
4.4 Ignoring parts of a page Sphider-plus includes an option to exclude parts of pages from being indexed. Thi 775e s can for example be used to prevent search result flooding when certain keywords appear on certain part in most pages (like a header, footer or a menu). Any part of a page between <! sphider_noindex > and <! . . .
. . .
<! sphider_noindex > and <! /sphider_noindex > tags is not indexed, however links in it are followed. 4.5 Ignoring parts of a page by <div id='abc'> Ignoring parts of a page by the <! sphider_noindex > tags requires direct access to the page, because the tags need to be added (edited) to the page. A more flexible method, . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ menu[0-5] / 4.6 Indexing only parts of a page by <div id='abc'> If enabled in Admin settings, the values as defined in the list-file /include/common/divs_use.txt will be used to index only the content between <div . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ table[0-5]] / 4.7 Ignore HTML elements defined by <tagname> . . </tagname> This option is foreseen to cooperate with the new HTML5 elements like section, nav, aside, hgroup, article, header, footer, etc. HTML elements . . .
. . .
contain a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */nav[0-5]/ Please keep in mind that element names placed in /include/common/elements_not.txt will be processed case-sensitive. 4.8 Index only HTML elements defined by <tagname> . . </tagname> This is the vice versa . . .
. . .
of all the duplicate content URLs: http://www.example.com/product.php?item=swedish-fish&category=gummy-candy http://www.example.com/product.php?item=swedish-fish&trackingid=1234&sessionid=5678 and Sphider-plus will understand that the duplicates all refer to the canonical URL: http://www.example.com/product.php?item=swedish-fish. The . . .
. . .
charset are supported by the ConvertCharset function and will be used to convert text into UTF-8 Unicode: WINDOWS windows-1250 - Central Europe windows-1251 - Cyrillic windows-1252 - Latin I windows-1253 - Greek windows-1254 - Turkish windows-1255 - Hebrew windows-1256 - Arabic windows-1257 - Baltic windows-1258 - Viet Nam cp874 - Thai - this . . .
. . .
- Baltic windows-1258 - Viet Nam cp874 - Thai - this file is also for DOS DOS cp437 - Latin US cp737 - Greek cp775 - BaltRim cp850 - Latin1 cp852 - Latin2 cp855 - Cyrylic cp857 - Turkish cp860 - Portuguese cp861 - Iceland cp862 - Hebrew cp863 - Canada cp864 - Arabic cp865 - Nordic cp866 - Cyrylic Russian (this is the one, used in IE . . .
. . .
IE Cyrillic (DOS) ) cp869 - Greek2 MAC (Apple) x-mac-cyrillic x-mac-greek x-mac-icelandic x-mac-ce x-mac-roman ISO iso-8859-1 iso-8859-2 iso-8859-3 iso-8859-4 iso-8859-5 iso-8859-6 iso-8859-7 iso-8859-8 iso-8859-9 iso-8859-10 iso-8859-11 iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 . . .
. . .
iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 cp500 cp856 cp875 cp1006 cp1026 koi8-r (Cyrillic) koi8-u (Cyrillic Ukrainian) nextstep us-ascii us-ascii-quotes DSP implementation for NeXT stdenc symbol zdingbat And specially for old Polish programs: mazovia This list is to be read . . .
. . .
Access denied; you need the RELOAD privilege. . . Top 10. Enable real-time output of logging data Up to version 1.5 of Sphider-plus, during index / re-index there was no printout available because: - Several servers, especially on Win32, buffer the output from the script until it terminates before transmitting the results to the browser. - . . .
. . .
is seen. - Some versions of Microsoft Internet Explorer only start to display the page after they have received 256 bytes of output. As progress was not presented during index / re- index procedure, waiting for results became a pain in the neck. Selectable in Admin setting together with the update interval (1 - 10 seconds), AJAX technology was . . .
. . .
characters in front of words Warning: This option should be used with special care and not be activated for non ISO-8859 charsets. Some special characters as part of the word ending might be erased by accidental. Top 13. Media search for images, audio streams and videos 13.1 Media indexing Index of media files is enabled by separated Admin . . .
. . .
and 'Found at' - Total clicks - Last clicked - Query input 'Indexed Image Thumbnails' presenting: - Thumbnail 150 x 100 pixel - Image details like title, filename size of original image, link- and thumb-id - Option to delete the thumbnail In order to open the media files all tables contain active links. Media results are also stored in . . .
. . .
Settings' menu of the admin backend: First option: For results of XML product feeds, present the text extract of 350 characters as part of product attributes. If this option is not activated, all products of the XML product feed will be presented in search results, and the hits of the query string will be highlighted in all involved products. . . .
. . .
content of the database tables (those with the same table prefix) will be destroyed by the restore procedure. 16.5 Copy & Move This section of the Database Management will present only those databases, which are correctly configured, assigned and do have a set of installed tables as described in chapter Definition and configuration. This . . .
. . .
results. Never the less to find these x results, the complete database had to been browsed. Starting with version 2.5, an additional clean option is offered as part of the Admin backend. Main advantage of this option is a significant reduction of the search time for any query, because the content of the db could be limited to offer only x . . .
. . .
<link>http://www.abc.de/images/warp.gif</link> <title>warp.gif</title> <x_size>635</x_size> <y_size>98</y_size> </media_result> <media_result> <num>2</num> <type>audio</type> <url>http://www.abc.de/index.php</url> . . .
. . .
the following content: {"query":"colour","ip":"::1","host_name":"guard_007-hoster","query_time":"2016-03-18 10:56:23 AM","consumed":0.016,"total_results":2,"num_of_results":2,"from":1,"to":2,"text_results":[{"num":1,"weight":"100.0","url":"http:\/\/www.english.le-piaggie.info\/Info_eng.pdf","title":" Info_eng","fulltxt":" . . . . . .
. . .
placed on the ground floor, today's big kitchen with the heating fireplace for open and close mode of . . . ","page_size":"5,661.6kb","domain_name":"www.english.le-piaggie.info"},{"num":2,"weight":"50.0","url":"http:\/\/www.english.le-piaggie.info\/html\/description.html","title":" Description","fulltxt":" . . . cable. Additional active . . .
. . .
D.O.C.G. as far as to the Pratomagno mountains 160 olive-trees Detached house, about 800 m to reach the . . . ","page_size":"26.4 kb","domain_name":"www.english.le-piaggie.info"}]} Top . . .


5a2 All required information. Introduction Release and Legal Info Installation Documentation Change Log [ Documentation Summary ] Preamble: The info presented here is valid only for the latest release of Sphider-plus. At present version 4.2024a published February 12, 2024 is the actual release. 1. Settings, customizing and statistics 2. Indexing . . .
. . .
2. Indexing 2.1 Various options 2.2 Allow other hosts in same domain 2.3 Word stemming 2.4 Periodical Re-indexing 2.5 Preferred indexing 2.6 Multithreaded indexing 2.7 Create thumbnails during index procedure 2.8 Prevent indexing of known malware and pishing pages 2.9 Follow and create sitemap files 2.10 Use private sitemap instead of global . . .
. . .
Sitemap file 3. Using the indexer from command line 3.1 All options 3.2 Multithreaded indexing 4. Keeping pages, words and files from being indexed 4.1 robots.txt 4.2 Must include / must not include string list 4.3 Ignoring links 4.4 Ignoring parts of a page by <! sphider_noindex > 4.5 Ignoring parts of a page by <div id='abc'> . . .
. . .
charset' 6. Search modes 6.1 Search with wildcards * 6.2 Strict search ! 6.3 Tolerant search 6.4 Link search 6.5 Media search 6.6 Search only in one domain 6.7 Search in categories 6.8 Greek language support 6.9 Block queries 7. Chronological order for result listing 7.1 Text result listing 7.2 Media result listing 8. PDF converter 9. Clean . . .
. . .
of logging data 11. Error messages and Debug mode 12. Delete secondary characters 13. Media search for images, audio streams and videos 13.1 Media indexing 13.2 Not supported media content 13.3 Search for media content 13.4 Statistics for media content 14. Feed support 14.1 XML product feeds 14.2 RDF, RSD, RSS and Atom feeds 15. Result . . .
. . .
Overview 16.2 Definition and configuration 16.3 Activate / disable database 16.4 Backup & Restore of databases 16.5 Copy & and Move 16.6 Enhancing functionality of multiple database support 17. Search in categories 17.1 Hierachical structure 17.2 Parallel structure 18. User suggested sites 19. Vulnerability protection 19.1 Prevent queries . . .
. . .
templates 22.2 Embed the search engine into existing HTML code 22.3 The different style sheet files 23. JSON, XML and RSS result output [ Documentation ] 1. Settings, customizing and statistics If you want to change settings, behavior and design of Sphider-plus, you can do so by means of the Admin interface. There is a wide range of . . .
. . .
interface. There is a wide range of settings foreseen for Sphider-plus. Separated into different submenus like: Sites: - Add Site - Index only the new - Re-index all - Re-index only preferred URLs - Erase Re-index (available also for individual URLs) - Import/export URL list - Approve sites - Banned domains Categories: - Add, edit, delete - . . .
. . .
URL list - Approve sites - Banned domains Categories: - Add, edit, delete - Create new subcategory under Index: - Basic indexing options - Advanced options Clean: - Clean keywords not associated with any link - Clean links not associated with any site - Clean Category table not associated with any site - Clean Media links - Clear Temp table . . .
. . .
- Clear Search log - Clear 'Most Popular Page Links' log - Clear 'Most Popular Media Links' log - Clear Spider log, separate and bulk delete - Clear Thumbnail images, separate and bulk delete - Clear Text cache - Clear Media cache - Clear IDS log file - Clear flood attempts log file - Clear all entries in addurl or banned table - Truncate all . . .
. . .
flood attempts log file - Clear all entries in addurl or banned table - Truncate all tables in database Settings: - General Settings - Index Log Settings - Spider Settings - Search Settings - Order of Result listing - Suggest Options - Page Indexing Weights Database: - Configure up to 5 databases with unlimited number of table sets - Activate . . .
. . .
Database: - Configure up to 5 databases with unlimited number of table sets - Activate separately for 'Admin', 'Search' user and 'Suggest URL user' - Backup / Restore - Copy / Move - Optimize Templates: In order to enable customer's integration of Sphider-plus into existing sites, HTML templates are prepared for Search form Text result . . .
. . .
Search form Text result listing Media result listing Most popular queries etc. Three different designs are offered, which may be selected in submenu 'Settings'. If the layout does not fit the design of your site (which is normal), you may create new designs and modify the appropriate file /templates/My_template/adminstyle.css . . .
. . .
the appropriate file /templates/My_template/adminstyle.css /templates/My_template/userstyle.css Statistics output: - Top keywords (Top 50 with hit counter). - All indexed thumbnails w 3e80 ith ID3 and EXIF info. - Larges pages offering link URL and file size. - Most Popular Searches for text links offering: Link addr., total clicks, last . . .
. . .
Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Searches for media links offering: Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Links (click counter). - Search log offering: Query, Results, Queried at, Time taken, User IP, Country, Host name (Latest100) - Index log offering: . . .
. . .
Country, Host name (Latest100) - Index log offering: File-name, index date and delete option - sitemap log offering: sitemap.xml output sitemap list offering file/page suffixes - IDS log offering: IP, host, query, impact, involved tags, date and time of intrusion. - Flood attempts log offering: IP, query, date and time of flood attempt. - Auto . . .
. . .
attempts log offering: IP, query, date and time of flood attempt. - Auto Re-index log file - Server info offering: Server software, environment, MySQL, PDF-converter, image functions, php.ini file PHP integration, PHP security info. Each item holding lists of details. All text links, media links and thumbnails are active linked. As stated in . . .
. . .
individual Re-indexing, the periodical Re-indexer could be started and aborted in the "Options" menu of each site. 2.5 Preferred Re-indexing Each new URL added to the Admin backend, could be supplied with a priority level. This level will be used by the option 'Re-index only preferred sites'. Level 1 will be interpreted as most important, while . . .
. . .
less all index results will be stored in log files in sub folder /admin/log/ The names of the log files look like: db2_100524-21.47.56_1.html (log file of first thread) db2_100524-21.48.12_2.html (log file of second thread) and is build by the following items: db2 - Number of database. 100524 - Date (May 24, 2010) 21.47.56 - Time when this thread . . .
. . .
spider.php -new1 php spider.php -new2 etc. The IDs will be added to the name of the corresponding log files like: db2_100524-21.47.56_ID1.html (log file of first thread) db2_100524-21.48.12_ID2.html (log file of second thread) IDs could be defined by personal requirements, but the limitations for file names with respect to the OS should be taken . . .
. . .
but will not erase the content of all the other tables. So the check whether the content of a page has changed (MD5sum) is still available for a fast re-index procedure. Once prepared, multithreaded re-indexing could be invoked by starting several threads and adding individual IDs to the option parameter like: php spider.php -erased1 php . . .
. . .
4.4 Ignoring parts of a page Sphider-plus includes an option to exclude parts of pages from being indexed. Thi 775e s can for example be used to prevent search result flooding when certain keywords appear on certain part in most pages (like a header, footer or a menu). Any part of a page between <! sphider_noindex > and <! . . .
. . .
<! sphider_noindex > and <! /sphider_noindex > tags is not indexed, however links in it are followed. 4.5 Ignoring parts of a page by <div id='abc'> Ignoring parts of a page by the <! sphider_noindex > tags requires direct access to the page, because the tags need to be added (edited) to the page. A more flexible method, . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ menu[0-5] / 4.6 Indexing only parts of a page by <div id='abc'> If enabled in Admin settings, the values as defined in the list-file /include/common/divs_use.txt will be used to index only the content between <div . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ table[0-5] / 4.7 Ignore HTML elements defined by <tagname> . . </tagname> This option is foreseen to cooperate with the new HTML5 elements like section, nav, aside, hgroup, article, header, footer, etc. HTML elements . . .
. . .
contain a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */nav[0-5]/ Please keep in mind that element names placed in /include/common/elements_not.txt will be processed case-sensitive. 4.8 Index only HTML elements defined by <tagname> . . </tagname> This is the vice versa . . .
. . .
of all the duplicate content URLs: http://www.example.com/product.php?item=swedish-fish&category=gummy-candy http://www.example.com/product.php?item=swedish-fish&trackingid=1234&sessionid=5678 and Sphider-plus will understand that the duplicates all refer to the canonical URL: http://www.example.com/product.php?item=swedish-fish. The . . .
. . .
charset are supported by the ConvertCharset function and will be used to convert text into UTF-8 Unicode: WINDOWS windows-1250 - Central Europe windows-1251 - Cyrillic windows-1252 - Latin I windows-1253 - Greek windows-1254 - Turkish windows-1255 - Hebrew windows-1256 - Arabic windows-1257 - Baltic windows-1258 - Viet Nam cp874 - Thai - this . . .
. . .
- Baltic windows-1258 - Viet Nam cp874 - Thai - this file is also for DOS DOS cp437 - Latin US cp737 - Greek cp775 - BaltRim cp850 - Latin1 cp852 - Latin2 cp855 - Cyrylic cp857 - Turkish cp860 - Portuguese cp861 - Iceland cp862 - Hebrew cp863 - Canada cp864 - Arabic cp865 - Nordic cp866 - Cyrylic Russian (this is the one, used in IE . . .
. . .
IE Cyrillic (DOS) ) cp869 - Greek2 MAC (Apple) x-mac-cyrillic x-mac-greek x-mac-icelandic x-mac-ce x-mac-roman ISO iso-8859-1 iso-8859-2 iso-8859-3 iso-8859-4 iso-8859-5 iso-8859-6 iso-8859-7 iso-8859-8 iso-8859-9 iso-8859-10 iso-8859-11 iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 . . .
. . .
iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 cp500 cp856 cp875 cp1006 cp1026 koi8-r (Cyrillic) koi8-u (Cyrillic Ukrainian) nextstep us-ascii us-ascii-quotes DSP implementation for NeXT stdenc symbol zdingbat And specially for old Polish programs: mazovia This list is to be read . . .
. . .
Access denied; you need the RELOAD privilege. . . Top 10. Enable real-time output of logging data Up to version 1.5 of Sphider-plus, during index / re-index there was no printout available because: - Several servers, especially on Win32, buffer the output from the script until it terminates before transmitting the results to the browser. - . . .
. . .
is seen. - Some versions of Microsoft Internet Explorer only start to display the page after they have received 256 bytes of output. As progress was not presented during index / re- index procedure, waiting for results became a pain in the neck. Selectable in Admin setting together with the update interval (1 - 10 seconds), AJAX technology was . . .
. . .
characters in front of words Warning: This option should be used with special care and not be activated for non ISO-8859 charsets. Some special characters as part of the word ending might be erased by accidental. Top 13. Media search for images, audio streams and videos 13.1 Media indexing Index of media files is enabled by separated Admin . . .
. . .
and 'Found at' - Total clicks - Last clicked - Query input 'Indexed Image Thumbnails' presenting: - Thumbnail 150 x 100 pixel - Image details like title, filename size of original image, link- and thumb-id - Option to delete the thumbnail In order to open the media files all tables contain active links. Media results are also stored in . . .
. . .
Settings' menu of the admin backend: First option: For results of XML product feeds, present the text extract of 350 characters as part of product attributes. If this option is not activated, all products of the XML product feed will be presented in search results, and the hits of the query string will be highlighted in all involved products. . . .
. . .
content of the database tables (those with the same table prefix) will be destroyed by the restore procedure. 16.5 Copy & Move This section of the Database Management will present only those databases, which are correctly configured, assigned and do have a set of installed tables as described in chapter Definition and configuration. This . . .
. . .
results. Never the less to find these x results, the complete database had to been browsed. Starting with version 2.5, an additional clean option is offered as part of the Admin backend. Main advantage of this option is a significant reduction of the search time for any query, because the content of the db could be limited to offer only x . . .
. . .
<link>http://www.abc.de/images/warp.gif</link> <title>warp.gif</title> <x_size>635</x_size> <y_size>98</y_size> </media_result> <media_result> <num>2</num> <type>audio</type> <url>http://www.abc.de/index.php</url> . . .
. . .
the following content: {"query":"colour","ip":"::1","host_name":"guard_007-hoster","query_time":"2016-03-18 10:56:23 AM","consumed":0.016,"total_results":2,"num_of_results":2,"from":1,"to":2,"text_results":[{"num":1,"weight":"100.0","url":"http:\/\/www.english.le-piaggie.info\/Info_eng.pdf","title":" Info_eng","fulltxt":" . . . . . .
. . .
placed on the ground floor, today's big kitchen with the heating fireplace for open and close mode of . . . ","page_size":"5,661.6kb","domain_name":"www.english.le-piaggie.info"},{"num":2,"weight":"50.0","url":"http:\/\/www.english.le-piaggie.info\/html\/description.html","title":" Description","fulltxt":" . . . cable. Additional active . . .
. . .
D.O.C.G. as far as to the Pratomagno mountains 160 olive-trees Detached house, about 800 m to reach the . . . ","page_size":"26.4 kb","domain_name":"www.english.le-piaggie.info"}]} Top . . .


5a2 All required information. Introduction Release and Legal Info Installation Documentation Change Log [ Documentation Summary ] Preamble: The info presented here is valid only for the latest release of Sphider-plus. At present version 4.2024a published February 12, 2024 is the actual release. 1. Settings, customizing and statistics 2. Indexing . . .
. . .
2. Indexing 2.1 Various options 2.2 Allow other hosts in same domain 2.3 Word stemming 2.4 Periodical Re-indexing 2.5 Preferred indexing 2.6 Multithreaded indexing 2.7 Create thumbnails during index procedure 2.8 Prevent indexing of known malware and pishing pages 2.9 Follow and create sitemap files 2.10 Use private sitemap instead of global . . .
. . .
Sitemap file 3. Using the indexer from command line 3.1 All options 3.2 Multithreaded indexing 4. Keeping pages, words and files from being indexed 4.1 robots.txt 4.2 Must include / must not include string list 4.3 Ignoring links 4.4 Ignoring parts of a page by <! sphider_noindex > 4.5 Ignoring parts of a page by <div id='abc'> . . .
. . .
charset' 6. Search modes 6.1 Search with wildcards * 6.2 Strict search ! 6.3 Tolerant search 6.4 Link search 6.5 Media search 6.6 Search only in one domain 6.7 Search in categories 6.8 Greek language support 6.9 Block queries 7. Chronological order for result listing 7.1 Text result listing 7.2 Media result listing 8. PDF converter 9. Clean . . .
. . .
of logging data 11. Error messages and Debug mode 12. Delete secondary characters 13. Media search for images, audio streams and videos 13.1 Media indexing 13.2 Not supported media content 13.3 Search for media content 13.4 Statistics for media content 14. Feed support 14.1 XML product feeds 14.2 RDF, RSD, RSS and Atom feeds 15. Result . . .
. . .
Overview 16.2 Definition and configuration 16.3 Activate / disable database 16.4 Backup & Restore of databases 16.5 Copy & and Move 16.6 Enhancing functionality of multiple database support 17. Search in categories 17.1 Hierachical structure 17.2 Parallel structure 18. User suggested sites 19. Vulnerability protection 19.1 Prevent queries . . .
. . .
templates 22.2 Embed the search engine into existing HTML code 22.3 The different style sheet files 23. JSON, XML and RSS result output [ Documentation ] 1. Settings, customizing and statistics If you want to change settings, behavior and design of Sphider-plus, you can do so by means of the Admin interface. There is a wide range of . . .
. . .
interface. There is a wide range of settings foreseen for Sphider-plus. Separated into different submenus like: Sites: - Add Site - Index only the new - Re-index all - Re-index only preferred URLs - Erase Re-index (available also for individual URLs) - Import/export URL list - Approve sites - Banned domains Categories: - Add, edit, delete - . . .
. . .
URL list - Approve sites - Banned domains Categories: - Add, edit, delete - Create new subcategory under Index: - Basic indexing options - Advanced options Clean: - Clean keywords not associated with any link - Clean links not associated with any site - Clean Category table not associated with any site - Clean Media links - Clear Temp table . . .
. . .
- Clear Search log - Clear 'Most Popular Page Links' log - Clear 'Most Popular Media Links' log - Clear Spider log, separate and bulk delete - Clear Thumbnail images, separate and bulk delete - Clear Text cache - Clear Media cache - Clear IDS log file - Clear flood attempts log file - Clear all entries in addurl or banned table - Truncate all . . .
. . .
flood attempts log file - Clear all entries in addurl or banned table - Truncate all tables in database Settings: - General Settings - Index Log Settings - Spider Settings - Search Settings - Order of Result listing - Suggest Options - Page Indexing Weights Database: - Configure up to 5 databases with unlimited number of table sets - Activate . . .
. . .
Database: - Configure up to 5 databases with unlimited number of table sets - Activate separately for 'Admin', 'Search' user and 'Suggest URL user' - Backup / Restore - Copy / Move - Optimize Templates: In order to enable customer's integration of Sphider-plus into existing sites, HTML templates are prepared for Search form Text result . . .
. . .
Search form Text result listing Media result listing Most popular queries etc. Three different designs are offered, which may be selected in submenu 'Settings'. If the layout does not fit the design of your site (which is normal), you may create new designs and modify the appropriate file /templates/My_template/adminstyle.css . . .
. . .
the appropriate file /templates/My_template/adminstyle.css /templates/My_template/userstyle.css Statistics output: - Top keywords (Top 50 with hit counter). - All indexed thumbnails w 3e80 ith ID3 and EXIF info. - Larges pages offering link URL and file size. - Most Popular Searches for text links offering: Link addr., total clicks, last . . .
. . .
Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Searches for media links offering: Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Links (click counter). - Search log offering: Query, Results, Queried at, Time taken, User IP, Country, Host name (Latest100) - Index log offering: . . .
. . .
Country, Host name (Latest100) - Index log offering: File-name, index date and delete option - sitemap log offering: sitemap.xml output sitemap list offering file/page suffixes - IDS log offering: IP, host, query, impact, involved tags, date and time of intrusion. - Flood attempts log offering: IP, query, date and time of flood attempt. - Auto . . .
. . .
attempts log offering: IP, query, date and time of flood attempt. - Auto Re-index log file - Server info offering: Server software, environment, MySQL, PDF-converter, image functions, php.ini file PHP integration, PHP security info. Each item holding lists of details. All text links, media links and thumbnails are active linked. As stated in . . .
. . .
individual Re-indexing, the periodical Re-indexer could be started and aborted in the "Options" menu of each site. 2.5 Preferred Re-indexing Each new URL added to the Admin backend, could be supplied with a priority level. This level will be used by the option 'Re-index only preferred sites'. Level 1 will be interpreted as most important, while . . .
. . .
less all index results will be stored in log files in sub folder /admin/log/ The names of the log files look like: db2_100524-21.47.56_1.html (log file of first thread) db2_100524-21.48.12_2.html (log file of second thread) and is build by the following items: db2 - Number of database. 100524 - Date (May 24, 2010) 21.47.56 - Time when this thread . . .
. . .
spider.php -new1 php spider.php -new2 etc. The IDs will be added to the name of the corresponding log files like: db2_100524-21.47.56_ID1.html (log file of first thread) db2_100524-21.48.12_ID2.html (log file of second thread) IDs could be defined by personal requirements, but the limitations for file names with respect to the OS should be taken . . .
. . .
but will not erase the content of all the other tables. So the check whether the content of a page has changed (MD5sum) is still available for a fast re-index procedure. Once prepared, multithreaded re-indexing could be invoked by starting several threads and adding individual IDs to the option parameter like: php spider.php -erased1 php . . .
. . .
4.4 Ignoring parts of a page Sphider-plus includes an option to exclude parts of pages from being indexed. Thi 775e s can for example be used to prevent search result flooding when certain keywords appear on certain part in most pages (like a header, footer or a menu). Any part of a page between <! sphider_noindex > and <! . . .
. . .
<! sphider_noindex > and <! /sphider_noindex > tags is not indexed, however links in it are followed. 4.5 Ignoring parts of a page by <div id='abc'> Ignoring parts of a page by the <! sphider_noindex > tags requires direct access to the page, because the tags need to be added (edited) to the page. A more flexible method, . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ menu[0-5] / 4.6 Indexing only parts of a page by <div id='abc'> If enabled in Admin settings, the values as defined in the list-file /include/common/divs_use.txt will be used to index only the content between <div . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ table[0-5] / 4.7 Ignore HTML elements defined by <tagname> . . </tagname> This option is foreseen to cooperate with the new HTML5 elements like section, nav, aside, hgroup, article, header, footer, etc. HTML elements . . .
. . .
contain a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */nav[0-5]/ Please keep in mind that element names placed in /include/common/elements_not.txt will be processed case-sensitive. 4.8 Index only HTML elements defined by <tagname> . . </tagname> This is the vice versa . . .
. . .
of all the duplicate content URLs: http://www.example.com/product.php?item=swedish-fish&category=gummy-candy http://www.example.com/product.php?item=swedish-fish&trackingid=1234&sessionid=5678 and Sphider-plus will understand that the duplicates all refer to the canonical URL: http://www.example.com/product.php?item=swedish-fish. The . . .
. . .
charset are supported by the ConvertCharset function and will be used to convert text into UTF-8 Unicode: WINDOWS windows-1250 - Central Europe windows-1251 - Cyrillic windows-1252 - Latin I windows-1253 - Greek windows-1254 - Turkish windows-1255 - Hebrew windows-1256 - Arabic windows-1257 - Baltic windows-1258 - Viet Nam cp874 - Thai - this . . .
. . .
- Baltic windows-1258 - Viet Nam cp874 - Thai - this file is also for DOS DOS cp437 - Latin US cp737 - Greek cp775 - BaltRim cp850 - Latin1 cp852 - Latin2 cp855 - Cyrylic cp857 - Turkish cp860 - Portuguese cp861 - Iceland cp862 - Hebrew cp863 - Canada cp864 - Arabic cp865 - Nordic cp866 - Cyrylic Russian (this is the one, used in IE . . .
. . .
IE Cyrillic (DOS) ) cp869 - Greek2 MAC (Apple) x-mac-cyrillic x-mac-greek x-mac-icelandic x-mac-ce x-mac-roman ISO iso-8859-1 iso-8859-2 iso-8859-3 iso-8859-4 iso-8859-5 iso-8859-6 iso-8859-7 iso-8859-8 iso-8859-9 iso-8859-10 iso-8859-11 iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 . . .
. . .
iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 cp500 cp856 cp875 cp1006 cp1026 koi8-r (Cyrillic) koi8-u (Cyrillic Ukrainian) nextstep us-ascii us-ascii-quotes DSP implementation for NeXT stdenc symbol zdingbat And specially for old Polish programs: mazovia This list is to be read . . .
. . .
Access denied; you need the RELOAD privilege. . . Top 10. Enable real-time output of logging data Up to version 1.5 of Sphider-plus, during index / re-index there was no printout available because: - Several servers, especially on Win32, buffer the output from the script until it terminates before transmitting the results to the browser. - . . .
. . .
is seen. - Some versions of Microsoft Internet Explorer only start to display the page after they have received 256 bytes of output. As progress was not presented during index / re- index procedure, waiting for results became a pain in the neck. Selectable in Admin setting together with the update interval (1 - 10 seconds), AJAX technology was . . .
. . .
characters in front of words Warning: This option should be used with special care and not be activated for non ISO-8859 charsets. Some special characters as part of the word ending might be erased by accidental. Top 13. Media search for images, audio streams and videos 13.1 Media indexing Index of media files is enabled by separated Admin . . .
. . .
and 'Found at' - Total clicks - Last clicked - Query input 'Indexed Image Thumbnails' presenting: - Thumbnail 150 x 100 pixel - Image details like title, filename size of original image, link- and thumb-id - Option to delete the thumbnail In order to open the media files all tables contain active links. Media results are also stored in . . .
. . .
Settings' menu of the admin backend: First option: For results of XML product feeds, present the text extract of 350 characters as part of product attributes. If this option is not activated, all products of the XML product feed will be presented in search results, and the hits of the query string will be highlighted in all involved products. . . .
. . .
content of the database tables (those with the same table prefix) will be destroyed by the restore procedure. 16.5 Copy & Move This section of the Database Management will present only those databases, which are correctly configured, assigned and do have a set of installed tables as described in chapter Definition and configuration. This . . .
. . .
results. Never the less to find these x results, the complete database had to been browsed. Starting with version 2.5, an additional clean option is offered as part of the Admin backend. Main advantage of this option is a significant reduction of the search time for any query, because the content of the db could be limited to offer only x . . .
. . .
<link>http://www.abc.de/images/warp.gif</link> <title>warp.gif</title> <x_size>635</x_size> <y_size>98</y_size> </media_result> <media_result> <num>2</num> <type>audio</type> <url>http://www.abc.de/index.php</url> . . .
. . .
the following content: {"query":"colour","ip":"::1","host_name":"guard_007-hoster","query_time":"2016-03-18 10:56:23 AM","consumed":0.016,"total_results":2,"num_of_results":2,"from":1,"to":2,"text_results":[{"num":1,"weight":"100.0","url":"http:\/\/www.english.le-piaggie.info\/Info_eng.pdf","title":" Info_eng","fulltxt":" . . . . . .
. . .
placed on the ground floor, today's big kitchen with the heating fireplace for open and close mode of . . . ","page_size":"5,661.6kb","domain_name":"www.english.le-piaggie.info"},{"num":2,"weight":"50.0","url":"http:\/\/www.english.le-piaggie.info\/html\/description.html","title":" Description","fulltxt":" . . . cable. Additional active . . .
. . .
D.O.C.G. as far as to the Pratomagno mountains 160 olive-trees Detached house, about 800 m to reach the . . . ","page_size":"26.4 kb","domain_name":"www.english.le-piaggie.info"}]} Top . . .


5a2 All required information. Introduction Release and Legal Info Installation Documentation Change Log [ Documentation Summary ] Preamble: The info presented here is valid only for the latest release of Sphider-plus. At present version 4.2024a published February 12, 2024 is the actual release. 1. Settings, customizing and statistics 2. Indexing . . .
. . .
2. Indexing 2.1 Various options 2.2 Allow other hosts in same domain 2.3 Word stemming 2.4 Periodical Re-indexing 2.5 Preferred indexing 2.6 Multithreaded indexing 2.7 Create thumbnails during index procedure 2.8 Prevent indexing of known malware and pishing pages 2.9 Follow and create sitemap files 2.10 Use private sitemap instead of global . . .
. . .
Sitemap file 3. Using the indexer from command line 3.1 All options 3.2 Multithreaded indexing 4. Keeping pages, words and files from being indexed 4.1 robots.txt 4.2 Must include / must not include string list 4.3 Ignoring links 4.4 Ignoring parts of a page by <! sphider_noindex > 4.5 Ignoring parts of a page by <div id='abc'> . . .
. . .
charset' 6. Search modes 6.1 Search with wildcards * 6.2 Strict search ! 6.3 Tolerant search 6.4 Link search 6.5 Media search 6.6 Search only in one domain 6.7 Search in categories 6.8 Greek language support 6.9 Block queries 7. Chronological order for result listing 7.1 Text result listing 7.2 Media result listing 8. PDF converter 9. Clean . . .
. . .
of logging data 11. Error messages and Debug mode 12. Delete secondary characters 13. Media search for images, audio streams and videos 13.1 Media indexing 13.2 Not supported media content 13.3 Search for media content 13.4 Statistics for media content 14. Feed support 14.1 XML product feeds 14.2 RDF, RSD, RSS and Atom feeds 15. Result . . .
. . .
Overview 16.2 Definition and configuration 16.3 Activate / disable database 16.4 Backup & Restore of databases 16.5 Copy & and Move 16.6 Enhancing functionality of multiple database support 17. Search in categories 17.1 Hierachical structure 17.2 Parallel structure 18. User suggested sites 19. Vulnerability protection 19.1 Prevent queries . . .
. . .
templates 22.2 Embed the search engine into existing HTML code 22.3 The different style sheet files 23. JSON, XML and RSS result output [ Documentation ] 1. Settings, customizing and statistics If you want to change settings, behavior and design of Sphider-plus, you can do so by means of the Admin interface. There is a wide range of . . .
. . .
interface. There is a wide range of settings foreseen for Sphider-plus. Separated into different submenus like: Sites: - Add Site - Index only the new - Re-index all - Re-index only preferred URLs - Erase Re-index (available also for individual URLs) - Import/export URL list - Approve sites - Banned domains Categories: - Add, edit, delete - . . .
. . .
URL list - Approve sites - Banned domains Categories: - Add, edit, delete - Create new subcategory under Index: - Basic indexing options - Advanced options Clean: - Clean keywords not associated with any link - Clean links not associated with any site - Clean Category table not associated with any site - Clean Media links - Clear Temp table . . .
. . .
- Clear Search log - Clear 'Most Popular Page Links' log - Clear 'Most Popular Media Links' log - Clear Spider log, separate and bulk delete - Clear Thumbnail images, separate and bulk delete - Clear Text cache - Clear Media cache - Clear IDS log file - Clear flood attempts log file - Clear all entries in addurl or banned table - Truncate all . . .
. . .
flood attempts log file - Clear all entries in addurl or banned table - Truncate all tables in database Settings: - General Settings - Index Log Settings - Spider Settings - Search Settings - Order of Result listing - Suggest Options - Page Indexing Weights Database: - Configure up to 5 databases with unlimited number of table sets - Activate . . .
. . .
Database: - Configure up to 5 databases with unlimited number of table sets - Activate separately for 'Admin', 'Search' user and 'Suggest URL user' - Backup / Restore - Copy / Move - Optimize Templates: In order to enable customer's integration of Sphider-plus into existing sites, HTML templates are prepared for Search form Text result . . .
. . .
Search form Text result listing Media result listing Most popular queries etc. Three different designs are offered, which may be selected in submenu 'Settings'. If the layout does not fit the design of your site (which is normal), you may create new designs and modify the appropriate file /templates/My_template/adminstyle.css . . .
. . .
the appropriate file /templates/My_template/adminstyle.css /templates/My_template/userstyle.css Statistics output: - Top keywords (Top 50 with hit counter). - All indexed thumbnails w 3e80 ith ID3 and EXIF info. - Larges pages offering link URL and file size. - Most Popular Searches for text links offering: Link addr., total clicks, last . . .
. . .
Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Searches for media links offering: Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Links (click counter). - Search log offering: Query, Results, Queried at, Time taken, User IP, Country, Host name (Latest100) - Index log offering: . . .
. . .
Country, Host name (Latest100) - Index log offering: File-name, index date and delete option - sitemap log offering: sitemap.xml output sitemap list offering file/page suffixes - IDS log offering: IP, host, query, impact, involved tags, date and time of intrusion. - Flood attempts log offering: IP, query, date and time of flood attempt. - Auto . . .
. . .
attempts log offering: IP, query, date and time of flood attempt. - Auto Re-index log file - Server info offering: Server software, environment, MySQL, PDF-converter, image functions, php.ini file PHP integration, PHP security info. Each item holding lists of details. All text links, media links and thumbnails are active linked. As stated in . . .
. . .
individual Re-indexing, the periodical Re-indexer could be started and aborted in the "Options" menu of each site. 2.5 Preferred Re-indexing Each new URL added to the Admin backend, could be supplied with a priority level. This level will be used by the option 'Re-index only preferred sites'. Level 1 will be interpreted as most important, while . . .
. . .
less all index results will be stored in log files in sub folder /admin/log/ The names of the log files look like: db2_100524-21.47.56_1.html (log file of first thread) db2_100524-21.48.12_2.html (log file of second thread) and is build by the following items: db2 - Number of database. 100524 - Date (May 24, 2010) 21.47.56 - Time when this thread . . .
. . .
spider.php -new1 php spider.php -new2 etc. The IDs will be added to the name of the corresponding log files like: db2_100524-21.47.56_ID1.html (log file of first thread) db2_100524-21.48.12_ID2.html (log file of second thread) IDs could be defined by personal requirements, but the limitations for file names with respect to the OS should be taken . . .
. . .
but will not erase the content of all the other tables. So the check whether the content of a page has changed (MD5sum) is still available for a fast re-index procedure. Once prepared, multithreaded re-indexing could be invoked by starting several threads and adding individual IDs to the option parameter like: php spider.php -erased1 php . . .
. . .
4.4 Ignoring parts of a page Sphider-plus includes an option to exclude parts of pages from being indexed. Thi 775e s can for example be used to prevent search result flooding when certain keywords appear on certain part in most pages (like a header, footer or a menu). Any part of a page between <! sphider_noindex > and <! . . .
. . .
<! sphider_noindex > and <! /sphider_noindex > tags is not indexed, however links in it are followed. 4.5 Ignoring parts of a page by <div id='abc'> Ignoring parts of a page by the <! sphider_noindex > tags requires direct access to the page, because the tags need to be added (edited) to the page. A more flexible method, . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ menu[0-5] / 4.6 Indexing only parts of a page by <div id='abc'> If enabled in Admin settings, the values as defined in the list-file /include/common/divs_use.txt will be used to index only the content between <div . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ table[0-5] / 4.7 Ignore HTML elements defined by <tagname> . . </tagname> This option is foreseen to cooperate with the new HTML5 elements like section, nav, aside, hgroup, article, header, footer, etc. HTML elements . . .
. . .
contain a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */nav[0-5]/ Please keep in mind that element names placed in /include/common/elements_not.txt will be processed case-sensitive. 4.8 Index only HTML elements defined by <tagname> . . </tagname> This is the vice versa . . .
. . .
of all the duplicate content URLs: http://www.example.com/product.php?item=swedish-fish&category=gummy-candy http://www.example.com/product.php?item=swedish-fish&trackingid=1234&sessionid=5678 and Sphider-plus will understand that the duplicates all refer to the canonical URL: http://www.example.com/product.php?item=swedish-fish. The . . .
. . .
charset are supported by the ConvertCharset function and will be used to convert text into UTF-8 Unicode: WINDOWS windows-1250 - Central Europe windows-1251 - Cyrillic windows-1252 - Latin I windows-1253 - Greek windows-1254 - Turkish windows-1255 - Hebrew windows-1256 - Arabic windows-1257 - Baltic windows-1258 - Viet Nam cp874 - Thai - this . . .
. . .
- Baltic windows-1258 - Viet Nam cp874 - Thai - this file is also for DOS DOS cp437 - Latin US cp737 - Greek cp775 - BaltRim cp850 - Latin1 cp852 - Latin2 cp855 - Cyrylic cp857 - Turkish cp860 - Portuguese cp861 - Iceland cp862 - Hebrew cp863 - Canada cp864 - Arabic cp865 - Nordic cp866 - Cyrylic Russian (this is the one, used in IE . . .
. . .
IE Cyrillic (DOS) ) cp869 - Greek2 MAC (Apple) x-mac-cyrillic x-mac-greek x-mac-icelandic x-mac-ce x-mac-roman ISO iso-8859-1 iso-8859-2 iso-8859-3 iso-8859-4 iso-8859-5 iso-8859-6 iso-8859-7 iso-8859-8 iso-8859-9 iso-8859-10 iso-8859-11 iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 . . .
. . .
iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 cp500 cp856 cp875 cp1006 cp1026 koi8-r (Cyrillic) koi8-u (Cyrillic Ukrainian) nextstep us-ascii us-ascii-quotes DSP implementation for NeXT stdenc symbol zdingbat And specially for old Polish programs: mazovia This list is to be read . . .
. . .
Access denied; you need the RELOAD privilege. . . Top 10. Enable real-time output of logging data Up to version 1.5 of Sphider-plus, during index / re-index there was no printout available because: - Several servers, especially on Win32, buffer the output from the script until it terminates before transmitting the results to the browser. - . . .
. . .
is seen. - Some versions of Microsoft Internet Explorer only start to display the page after they have received 256 bytes of output. As progress was not presented during index / re- index procedure, waiting for results became a pain in the neck. Selectable in Admin setting together with the update interval (1 - 10 seconds), AJAX technology was . . .
. . .
characters in front of words Warning: This option should be used with special care and not be activated for non ISO-8859 charsets. Some special characters as part of the word ending might be erased by accidental. Top 13. Media search for images, audio streams and videos 13.1 Media indexing Index of media files is enabled by separated Admin . . .
. . .
and 'Found at' - Total clicks - Last clicked - Query input 'Indexed Image Thumbnails' presenting: - Thumbnail 150 x 100 pixel - Image details like title, filename size of original image, link- and thumb-id - Option to delete the thumbnail In order to open the media files all tables contain active links. Media results are also stored in . . .
. . .
Settings' menu of the admin backend: First option: For results of XML product feeds, present the text extract of 350 characters as part of product attributes. If this option is not activated, all products of the XML product feed will be presented in search results, and the hits of the query string will be highlighted in all involved products. . . .
. . .
content of the database tables (those with the same table prefix) will be destroyed by the restore procedure. 16.5 Copy & Move This section of the Database Management will present only those databases, which are correctly configured, assigned and do have a set of installed tables as described in chapter Definition and configuration. This . . .
. . .
results. Never the less to find these x results, the complete database had to been browsed. Starting with version 2.5, an additional clean option is offered as part of the Admin backend. Main advantage of this option is a significant reduction of the search time for any query, because the content of the db could be limited to offer only x . . .
. . .
<link>http://www.abc.de/images/warp.gif</link> <title>warp.gif</title> <x_size>635</x_size> <y_size>98</y_size> </media_result> <media_result> <num>2</num> <type>audio</type> <url>http://www.abc.de/index.php</url> . . .
. . .
the following content: {"query":"colour","ip":"::1","host_name":"guard_007-hoster","query_time":"2016-03-18 10:56:23 AM","consumed":0.016,"total_results":2,"num_of_results":2,"from":1,"to":2,"text_results":[{"num":1,"weight":"100.0","url":"http:\/\/www.english.le-piaggie.info\/Info_eng.pdf","title":" Info_eng","fulltxt":" . . . . . .
. . .
placed on the ground floor, today's big kitchen with the heating fireplace for open and close mode of . . . ","page_size":"5,661.6kb","domain_name":"www.english.le-piaggie.info"},{"num":2,"weight":"50.0","url":"http:\/\/www.english.le-piaggie.info\/html\/description.html","title":" Description","fulltxt":" . . . cable. Additional active . . .
. . .
D.O.C.G. as far as to the Pratomagno mountains 160 olive-trees Detached house, about 800 m to reach the . . . ","page_size":"26.4 kb","domain_name":"www.english.le-piaggie.info"}]} Top . . .


5a2 All required information. Introduction Release and Legal Info Installation Documentation Change Log [ Documentation Summary ] Preamble: The info presented here is valid only for the latest release of Sphider-plus. At present version 4.2024a published February 12, 2024 is the actual release. 1. Settings, customizing and statistics 2. Indexing . . .
. . .
2. Indexing 2.1 Various options 2.2 Allow other hosts in same domain 2.3 Word stemming 2.4 Periodical Re-indexing 2.5 Preferred indexing 2.6 Multithreaded indexing 2.7 Create thumbnails during index procedure 2.8 Prevent indexing of known malware and pishing pages 2.9 Follow and create sitemap files 2.10 Use private sitemap instead of global . . .
. . .
Sitemap file 3. Using the indexer from command line 3.1 All options 3.2 Multithreaded indexing 4. Keeping pages, words and files from being indexed 4.1 robots.txt 4.2 Must include / must not include string list 4.3 Ignoring links 4.4 Ignoring parts of a page by <! sphider_noindex > 4.5 Ignoring parts of a page by <div id='abc'> . . .
. . .
charset' 6. Search modes 6.1 Search with wildcards * 6.2 Strict search ! 6.3 Tolerant search 6.4 Link search 6.5 Media search 6.6 Search only in one domain 6.7 Search in categories 6.8 Greek language support 6.9 Block queries 7. Chronological order for result listing 7.1 Text result listing 7.2 Media result listing 8. PDF converter 9. Clean . . .
. . .
of logging data 11. Error messages and Debug mode 12. Delete secondary characters 13. Media search for images, audio streams and videos 13.1 Media indexing 13.2 Not supported media content 13.3 Search for media content 13.4 Statistics for media content 14. Feed support 14.1 XML product feeds 14.2 RDF, RSD, RSS and Atom feeds 15. Result . . .
. . .
Overview 16.2 Definition and configuration 16.3 Activate / disable database 16.4 Backup & Restore of databases 16.5 Copy & and Move 16.6 Enhancing functionality of multiple database support 17. Search in categories 17.1 Hierachical structure 17.2 Parallel structure 18. User suggested sites 19. Vulnerability protection 19.1 Prevent queries . . .
. . .
templates 22.2 Embed the search engine into existing HTML code 22.3 The different style sheet files 23. JSON, XML and RSS result output [ Documentation ] 1. Settings, customizing and statistics If you want to change settings, behavior and design of Sphider-plus, you can do so by means of the Admin interface. There is a wide range of . . .
. . .
interface. There is a wide range of settings foreseen for Sphider-plus. Separated into different submenus like: Sites: - Add Site - Index only the new - Re-index all - Re-index only preferred URLs - Erase Re-index (available also for individual URLs) - Import/export URL list - Approve sites - Banned domains Categories: - Add, edit, delete - . . .
. . .
URL list - Approve sites - Banned domains Categories: - Add, edit, delete - Create new subcategory under Index: - Basic indexing options - Advanced options Clean: - Clean keywords not associated with any link - Clean links not associated with any site - Clean Category table not associated with any site - Clean Media links - Clear Temp table . . .
. . .
- Clear Search log - Clear 'Most Popular Page Links' log - Clear 'Most Popular Media Links' log - Clear Spider log, separate and bulk delete - Clear Thumbnail images, separate and bulk delete - Clear Text cache - Clear Media cache - Clear IDS log file - Clear flood attempts log file - Clear all entries in addurl or banned table - Truncate all . . .
. . .
flood attempts log file - Clear all entries in addurl or banned table - Truncate all tables in database Settings: - General Settings - Index Log Settings - Spider Settings - Search Settings - Order of Result listing - Suggest Options - Page Indexing Weights Database: - Configure up to 5 databases with unlimited number of table sets - Activate . . .
. . .
Database: - Configure up to 5 databases with unlimited number of table sets - Activate separately for 'Admin', 'Search' user and 'Suggest URL user' - Backup / Restore - Copy / Move - Optimize Templates: In order to enable customer's integration of Sphider-plus into existing sites, HTML templates are prepared for Search form Text result . . .
. . .
Search form Text result listing Media result listing Most popular queries etc. Three different designs are offered, which may be selected in submenu 'Settings'. If the layout does not fit the design of your site (which is normal), you may create new designs and modify the appropriate file /templates/My_template/adminstyle.css . . .
. . .
the appropriate file /templates/My_template/adminstyle.css /templates/My_template/userstyle.css Statistics output: - Top keywords (Top 50 with hit counter). - All indexed thumbnails w 3e80 ith ID3 and EXIF info. - Larges pages offering link URL and file size. - Most Popular Searches for text links offering: Link addr., total clicks, last . . .
. . .
Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Searches for media links offering: Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Links (click counter). - Search log offering: Query, Results, Queried at, Time taken, User IP, Country, Host name (Latest100) - Index log offering: . . .
. . .
Country, Host name (Latest100) - Index log offering: File-name, index date and delete option - sitemap log offering: sitemap.xml output sitemap list offering file/page suffixes - IDS log offering: IP, host, query, impact, involved tags, date and time of intrusion. - Flood attempts log offering: IP, query, date and time of flood attempt. - Auto . . .
. . .
attempts log offering: IP, query, date and time of flood attempt. - Auto Re-index log file - Server info offering: Server software, environment, MySQL, PDF-converter, image functions, php.ini file PHP integration, PHP security info. Each item holding lists of details. All text links, media links and thumbnails are active linked. As stated in . . .
. . .
individual Re-indexing, the periodical Re-indexer could be started and aborted in the "Options" menu of each site. 2.5 Preferred Re-indexing Each new URL added to the Admin backend, could be supplied with a priority level. This level will be used by the option 'Re-index only preferred sites'. Level 1 will be interpreted as most important, while . . .
. . .
less all index results will be stored in log files in sub folder /admin/log/ The names of the log files look like: db2_100524-21.47.56_1.html (log file of first thread) db2_100524-21.48.12_2.html (log file of second thread) and is build by the following items: db2 - Number of database. 100524 - Date (May 24, 2010) 21.47.56 - Time when this thread . . .
. . .
spider.php -new1 php spider.php -new2 etc. The IDs will be added to the name of the corresponding log files like: db2_100524-21.47.56_ID1.html (log file of first thread) db2_100524-21.48.12_ID2.html (log file of second thread) IDs could be defined by personal requirements, but the limitations for file names with respect to the OS should be taken . . .
. . .
but will not erase the content of all the other tables. So the check whether the content of a page has changed (MD5sum) is still available for a fast re-index procedure. Once prepared, multithreaded re-indexing could be invoked by starting several threads and adding individual IDs to the option parameter like: php spider.php -erased1 php . . .
. . .
4.4 Ignoring parts of a page Sphider-plus includes an option to exclude parts of pages from being indexed. Thi 775e s can for example be used to prevent search result flooding when certain keywords appear on certain part in most pages (like a header, footer or a menu). Any part of a page between <! sphider_noindex > and <! . . .
. . .
<! sphider_noindex > and <! /sphider_noindex > tags is not indexed, however links in it are followed. 4.5 Ignoring parts of a page by <div id='abc'> Ignoring parts of a page by the <! sphider_noindex > tags requires direct access to the page, because the tags need to be added (edited) to the page. A more flexible method, . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ menu[0-5] / 4.6 Indexing only parts of a page by <div id='abc'> If enabled in Admin settings, the values as defined in the list-file /include/common/divs_use.txt will be used to index only the content between <div . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ table[0-5] / 4.7 Ignore HTML elements defined by <tagname> . . </tagname> This option is foreseen to cooperate with the new HTML5 elements like section, nav, aside, hgroup, article, header, footer, etc. HTML elements . . .
. . .
contain a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */nav[0-5]/ Please keep in mind that element names placed in /include/common/elements_not.txt will be processed case-sensitive. 4.8 Index only HTML elements defined by <tagname> . . </tagname> This is the vice versa . . .
. . .
of all the duplicate content URLs: http://www.example.com/product.php?item=swedish-fish&category=gummy-candy http://www.example.com/product.php?item=swedish-fish&trackingid=1234&sessionid=5678 and Sphider-plus will understand that the duplicates all refer to the canonical URL: http://www.example.com/product.php?item=swedish-fish. The . . .
. . .
charset are supported by the ConvertCharset function and will be used to convert text into UTF-8 Unicode: WINDOWS windows-1250 - Central Europe windows-1251 - Cyrillic windows-1252 - Latin I windows-1253 - Greek windows-1254 - Turkish windows-1255 - Hebrew windows-1256 - Arabic windows-1257 - Baltic windows-1258 - Viet Nam cp874 - Thai - this . . .
. . .
- Baltic windows-1258 - Viet Nam cp874 - Thai - this file is also for DOS DOS cp437 - Latin US cp737 - Greek cp775 - BaltRim cp850 - Latin1 cp852 - Latin2 cp855 - Cyrylic cp857 - Turkish cp860 - Portuguese cp861 - Iceland cp862 - Hebrew cp863 - Canada cp864 - Arabic cp865 - Nordic cp866 - Cyrylic Russian (this is the one, used in IE . . .
. . .
IE Cyrillic (DOS) ) cp869 - Greek2 MAC (Apple) x-mac-cyrillic x-mac-greek x-mac-icelandic x-mac-ce x-mac-roman ISO iso-8859-1 iso-8859-2 iso-8859-3 iso-8859-4 iso-8859-5 iso-8859-6 iso-8859-7 iso-8859-8 iso-8859-9 iso-8859-10 iso-8859-11 iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 . . .
. . .
iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 cp500 cp856 cp875 cp1006 cp1026 koi8-r (Cyrillic) koi8-u (Cyrillic Ukrainian) nextstep us-ascii us-ascii-quotes DSP implementation for NeXT stdenc symbol zdingbat And specially for old Polish programs: mazovia This list is to be read . . .
. . .
Access denied; you need the RELOAD privilege. . . Top 10. Enable real-time output of logging data Up to version 1.5 of Sphider-plus, during index / re-index there was no printout available because: - Several servers, especially on Win32, buffer the output from the script until it terminates before transmitting the results to the browser. - . . .
. . .
is seen. - Some versions of Microsoft Internet Explorer only start to display the page after they have received 256 bytes of output. As progress was not presented during index / re- index procedure, waiting for results became a pain in the neck. Selectable in Admin setting together with the update interval (1 - 10 seconds), AJAX technology was . . .
. . .
characters in front of words Warning: This option should be used with special care and not be activated for non ISO-8859 charsets. Some special characters as part of the word ending might be erased by accidental. Top 13. Media search for images, audio streams and videos 13.1 Media indexing Index of media files is enabled by separated Admin . . .
. . .
and 'Found at' - Total clicks - Last clicked - Query input 'Indexed Image Thumbnails' presenting: - Thumbnail 150 x 100 pixel - Image details like title, filename size of original image, link- and thumb-id - Option to delete the thumbnail In order to open the media files all tables contain active links. Media results are also stored in . . .
. . .
Settings' menu of the admin backend: First option: For results of XML product feeds, present the text extract of 350 characters as part of product attributes. If this option is not activated, all products of the XML product feed will be presented in search results, and the hits of the query string will be highlighted in all involved products. . . .
. . .
content of the database tables (those with the same table prefix) will be destroyed by the restore procedure. 16.5 Copy & Move This section of the Database Management will present only those databases, which are correctly configured, assigned and do have a set of installed tables as described in chapter Definition and configuration. This . . .
. . .
results. Never the less to find these x results, the complete database had to been browsed. Starting with version 2.5, an additional clean option is offered as part of the Admin backend. Main advantage of this option is a significant reduction of the search time for any query, because the content of the db could be limited to offer only x . . .
. . .
<link>http://www.abc.de/images/warp.gif</link> <title>warp.gif</title> <x_size>635</x_size> <y_size>98</y_size> </media_result> <media_result> <num>2</num> <type>audio</type> <url>http://www.abc.de/index.php</url> . . .
. . .
the following content: {"query":"colour","ip":"::1","host_name":"guard_007-hoster","query_time":"2016-03-18 10:56:23 AM","consumed":0.016,"total_results":2,"num_of_results":2,"from":1,"to":2,"text_results":[{"num":1,"weight":"100.0","url":"http:\/\/www.english.le-piaggie.info\/Info_eng.pdf","title":" Info_eng","fulltxt":" . . . . . .
. . .
placed on the ground floor, today's big kitchen with the heating fireplace for open and close mode of . . . ","page_size":"5,661.6kb","domain_name":"www.english.le-piaggie.info"},{"num":2,"weight":"50.0","url":"http:\/\/www.english.le-piaggie.info\/html\/description.html","title":" Description","fulltxt":" . . . cable. Additional active . . .
. . .
D.O.C.G. as far as to the Pratomagno mountains 160 olive-trees Detached house, about 800 m to reach the . . . ","page_size":"26.4 kb","domain_name":"www.english.le-piaggie.info"}]} Top . . .


5a2 All required information. Introduction Release and Legal Info Installation Documentation Change Log [ Documentation Summary ] Preamble: The info presented here is valid only for the latest release of Sphider-plus. At present version 4.2024a published February 12, 2024 is the actual release. 1. Settings, customizing and statistics 2. Indexing . . .
. . .
2. Indexing 2.1 Various options 2.2 Allow other hosts in same domain 2.3 Word stemming 2.4 Periodical Re-indexing 2.5 Preferred indexing 2.6 Multithreaded indexing 2.7 Create thumbnails during index procedure 2.8 Prevent indexing of known malware and pishing pages 2.9 Follow and create sitemap files 2.10 Use private sitemap instead of global . . .
. . .
Sitemap file 3. Using the indexer from command line 3.1 All options 3.2 Multithreaded indexing 4. Keeping pages, words and files from being indexed 4.1 robots.txt 4.2 Must include / must not include string list 4.3 Ignoring links 4.4 Ignoring parts of a page by <! sphider_noindex > 4.5 Ignoring parts of a page by <div id='abc'> . . .
. . .
charset' 6. Search modes 6.1 Search with wildcards * 6.2 Strict search ! 6.3 Tolerant search 6.4 Link search 6.5 Media search 6.6 Search only in one domain 6.7 Search in categories 6.8 Greek language support 6.9 Block queries 7. Chronological order for result listing 7.1 Text result listing 7.2 Media result listing 8. PDF converter 9. Clean . . .
. . .
of logging data 11. Error messages and Debug mode 12. Delete secondary characters 13. Media search for images, audio streams and videos 13.1 Media indexing 13.2 Not supported media content 13.3 Search for media content 13.4 Statistics for media content 14. Feed support 14.1 XML product feeds 14.2 RDF, RSD, RSS and Atom feeds 15. Result . . .
. . .
Overview 16.2 Definition and configuration 16.3 Activate / disable database 16.4 Backup & Restore of databases 16.5 Copy & and Move 16.6 Enhancing functionality of multiple database support 17. Search in categories 17.1 Hierachical structure 17.2 Parallel structure 18. User suggested sites 19. Vulnerability protection 19.1 Prevent queries . . .
. . .
templates 22.2 Embed the search engine into existing HTML code 22.3 The different style sheet files 23. JSON, XML and RSS result output [ Documentation ] 1. Settings, customizing and statistics If you want to change settings, behavior and design of Sphider-plus, you can do so by means of the Admin interface. There is a wide range of . . .
. . .
interface. There is a wide range of settings foreseen for Sphider-plus. Separated into different submenus like: Sites: - Add Site - Index only the new - Re-index all - Re-index only preferred URLs - Erase Re-index (available also for individual URLs) - Import/export URL list - Approve sites - Banned domains Categories: - Add, edit, delete - . . .
. . .
URL list - Approve sites - Banned domains Categories: - Add, edit, delete - Create new subcategory under Index: - Basic indexing options - Advanced options Clean: - Clean keywords not associated with any link - Clean links not associated with any site - Clean Category table not associated with any site - Clean Media links - Clear Temp table . . .
. . .
- Clear Search log - Clear 'Most Popular Page Links' log - Clear 'Most Popular Media Links' log - Clear Spider log, separate and bulk delete - Clear Thumbnail images, separate and bulk delete - Clear Text cache - Clear Media cache - Clear IDS log file - Clear flood attempts log file - Clear all entries in addurl or banned table - Truncate all . . .
. . .
flood attempts log file - Clear all entries in addurl or banned table - Truncate all tables in database Settings: - General Settings - Index Log Settings - Spider Settings - Search Settings - Order of Result listing - Suggest Options - Page Indexing Weights Database: - Configure up to 5 databases with unlimited number of table sets - Activate . . .
. . .
Database: - Configure up to 5 databases with unlimited number of table sets - Activate separately for 'Admin', 'Search' user and 'Suggest URL user' - Backup / Restore - Copy / Move - Optimize Templates: In order to enable customer's integration of Sphider-plus into existing sites, HTML templates are prepared for Search form Text result . . .
. . .
Search form Text result listing Media result listing Most popular queries etc. Three different designs are offered, which may be selected in submenu 'Settings'. If the layout does not fit the design of your site (which is normal), you may create new designs and modify the appropriate file /templates/My_template/adminstyle.css . . .
. . .
the appropriate file /templates/My_template/adminstyle.css /templates/My_template/userstyle.css Statistics output: - Top keywords (Top 50 with hit counter). - All indexed thumbnails w 3e80 ith ID3 and EXIF info. - Larges pages offering link URL and file size. - Most Popular Searches for text links offering: Link addr., total clicks, last . . .
. . .
Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Searches for media links offering: Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Links (click counter). - Search log offering: Query, Results, Queried at, Time taken, User IP, Country, Host name (Latest100) - Index log offering: . . .
. . .
Country, Host name (Latest100) - Index log offering: File-name, index date and delete option - sitemap log offering: sitemap.xml output sitemap list offering file/page suffixes - IDS log offering: IP, host, query, impact, involved tags, date and time of intrusion. - Flood attempts log offering: IP, query, date and time of flood attempt. - Auto . . .
. . .
attempts log offering: IP, query, date and time of flood attempt. - Auto Re-index log file - Server info offering: Server software, environment, MySQL, PDF-converter, image functions, php.ini file PHP integration, PHP security info. Each item holding lists of details. All text links, media links and thumbnails are active linked. As stated in . . .
. . .
individual Re-indexing, the periodical Re-indexer could be started and aborted in the "Options" menu of each site. 2.5 Preferred Re-indexing Each new URL added to the Admin backend, could be supplied with a priority level. This level will be used by the option 'Re-index only preferred sites'. Level 1 will be interpreted as most important, while . . .
. . .
less all index results will be stored in log files in sub folder /admin/log/ The names of the log files look like: db2_100524-21.47.56_1.html (log file of first thread) db2_100524-21.48.12_2.html (log file of second thread) and is build by the following items: db2 - Number of database. 100524 - Date (May 24, 2010) 21.47.56 - Time when this thread . . .
. . .
spider.php -new1 php spider.php -new2 etc. The IDs will be added to the name of the corresponding log files like: db2_100524-21.47.56_ID1.html (log file of first thread) db2_100524-21.48.12_ID2.html (log file of second thread) IDs could be defined by personal requirements, but the limitations for file names with respect to the OS should be taken . . .
. . .
but will not erase the content of all the other tables. So the check whether the content of a page has changed (MD5sum) is still available for a fast re-index procedure. Once prepared, multithreaded re-indexing could be invoked by starting several threads and adding individual IDs to the option parameter like: php spider.php -erased1 php . . .
. . .
4.4 Ignoring parts of a page Sphider-plus includes an option to exclude parts of pages from being indexed. Thi 775e s can for example be used to prevent search result flooding when certain keywords appear on certain part in most pages (like a header, footer or a menu). Any part of a page between <! sphider_noindex > and <! . . .
. . .
<! sphider_noindex > and <! /sphider_noindex > tags is not indexed, however links in it are followed. 4.5 Ignoring parts of a page by <div id='abc'> Ignoring parts of a page by the <! sphider_noindex > tags requires direct access to the page, because the tags need to be added (edited) to the page. A more flexible method, . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ menu[0-5] / 4.6 Indexing only parts of a page by <div id='abc'> If enabled in Admin settings, the values as defined in the list-file /include/common/divs_use.txt will be used to index only the content between <div . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ table[0-5] / 4.7 Ignore HTML elements defined by <tagname> . . </tagname> This option is foreseen to cooperate with the new HTML5 elements like section, nav, aside, hgroup, article, header, footer, etc. HTML elements . . .
. . .
contain a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */nav[0-5]/ Please keep in mind that element names placed in /include/common/elements_not.txt will be processed case-sensitive. 4.8 Index only HTML elements defined by <tagname> . . </tagname> This is the vice versa . . .
. . .
of all the duplicate content URLs: http://www.example.com/product.php?item=swedish-fish&category=gummy-candy http://www.example.com/product.php?item=swedish-fish&trackingid=1234&sessionid=5678 and Sphider-plus will understand that the duplicates all refer to the canonical URL: http://www.example.com/product.php?item=swedish-fish. The . . .
. . .
charset are supported by the ConvertCharset function and will be used to convert text into UTF-8 Unicode: WINDOWS windows-1250 - Central Europe windows-1251 - Cyrillic windows-1252 - Latin I windows-1253 - Greek windows-1254 - Turkish windows-1255 - Hebrew windows-1256 - Arabic windows-1257 - Baltic windows-1258 - Viet Nam cp874 - Thai - this . . .
. . .
- Baltic windows-1258 - Viet Nam cp874 - Thai - this file is also for DOS DOS cp437 - Latin US cp737 - Greek cp775 - BaltRim cp850 - Latin1 cp852 - Latin2 cp855 - Cyrylic cp857 - Turkish cp860 - Portuguese cp861 - Iceland cp862 - Hebrew cp863 - Canada cp864 - Arabic cp865 - Nordic cp866 - Cyrylic Russian (this is the one, used in IE . . .
. . .
IE Cyrillic (DOS) ) cp869 - Greek2 MAC (Apple) x-mac-cyrillic x-mac-greek x-mac-icelandic x-mac-ce x-mac-roman ISO iso-8859-1 iso-8859-2 iso-8859-3 iso-8859-4 iso-8859-5 iso-8859-6 iso-8859-7 iso-8859-8 iso-8859-9 iso-8859-10 iso-8859-11 iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 . . .
. . .
iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 cp500 cp856 cp875 cp1006 cp1026 koi8-r (Cyrillic) koi8-u (Cyrillic Ukrainian) nextstep us-ascii us-ascii-quotes DSP implementation for NeXT stdenc symbol zdingbat And specially for old Polish programs: mazovia This list is to be read . . .
. . .
Access denied; you need the RELOAD privilege. . . Top 10. Enable real-time output of logging data Up to version 1.5 of Sphider-plus, during index / re-index there was no printout available because: - Several servers, especially on Win32, buffer the output from the script until it terminates before transmitting the results to the browser. - . . .
. . .
is seen. - Some versions of Microsoft Internet Explorer only start to display the page after they have received 256 bytes of output. As progress was not presented during index / re- index procedure, waiting for results became a pain in the neck. Selectable in Admin setting together with the update interval (1 - 10 seconds), AJAX technology was . . .
. . .
characters in front of words Warning: This option should be used with special care and not be activated for non ISO-8859 charsets. Some special characters as part of the word ending might be erased by accidental. Top 13. Media search for images, audio streams and videos 13.1 Media indexing Index of media files is enabled by separated Admin . . .
. . .
and 'Found at' - Total clicks - Last clicked - Query input 'Indexed Image Thumbnails' presenting: - Thumbnail 150 x 100 pixel - Image details like title, filename size of original image, link- and thumb-id - Option to delete the thumbnail In order to open the media files all tables contain active links. Media results are also stored in . . .
. . .
Settings' menu of the admin backend: First option: For results of XML product feeds, present the text extract of 350 characters as part of product attributes. If this option is not activated, all products of the XML product feed will be presented in search results, and the hits of the query string will be highlighted in all involved products. . . .
. . .
content of the database tables (those with the same table prefix) will be destroyed by the restore procedure. 16.5 Copy & Move This section of the Database Management will present only those databases, which are correctly configured, assigned and do have a set of installed tables as described in chapter Definition and configuration. This . . .
. . .
results. Never the less to find these x results, the complete database had to been browsed. Starting with version 2.5, an additional clean option is offered as part of the Admin backend. Main advantage of this option is a significant reduction of the search time for any query, because the content of the db could be limited to offer only x . . .
. . .
<link>http://www.abc.de/images/warp.gif</link> <title>warp.gif</title> <x_size>635</x_size> <y_size>98</y_size> </media_result> <media_result> <num>2</num> <type>audio</type> <url>http://www.abc.de/index.php</url> . . .
. . .
the following content: {"query":"colour","ip":"::1","host_name":"guard_007-hoster","query_time":"2016-03-18 10:56:23 AM","consumed":0.016,"total_results":2,"num_of_results":2,"from":1,"to":2,"text_results":[{"num":1,"weight":"100.0","url":"http:\/\/www.english.le-piaggie.info\/Info_eng.pdf","title":" Info_eng","fulltxt":" . . . . . .
. . .
placed on the ground floor, today's big kitchen with the heating fireplace for open and close mode of . . . ","page_size":"5,661.6kb","domain_name":"www.english.le-piaggie.info"},{"num":2,"weight":"50.0","url":"http:\/\/www.english.le-piaggie.info\/html\/description.html","title":" Description","fulltxt":" . . . cable. Additional active . . .
. . .
D.O.C.G. as far as to the Pratomagno mountains 160 olive-trees Detached house, about 800 m to reach the . . . ","page_size":"26.4 kb","domain_name":"www.english.le-piaggie.info"}]} Top . . .


5a2 All required information. Introduction Release and Legal Info Installation Documentation Change Log [ Documentation Summary ] Preamble: The info presented here is valid only for the latest release of Sphider-plus. At present version 4.2024a published February 12, 2024 is the actual release. 1. Settings, customizing and statistics 2. Indexing . . .
. . .
2. Indexing 2.1 Various options 2.2 Allow other hosts in same domain 2.3 Word stemming 2.4 Periodical Re-indexing 2.5 Preferred indexing 2.6 Multithreaded indexing 2.7 Create thumbnails during index procedure 2.8 Prevent indexing of known malware and pishing pages 2.9 Follow and create sitemap files 2.10 Use private sitemap instead of global . . .
. . .
Sitemap file 3. Using the indexer from command line 3.1 All options 3.2 Multithreaded indexing 4. Keeping pages, words and files from being indexed 4.1 robots.txt 4.2 Must include / must not include string list 4.3 Ignoring links 4.4 Ignoring parts of a page by <! sphider_noindex > 4.5 Ignoring parts of a page by <div id='abc'> . . .
. . .
charset' 6. Search modes 6.1 Search with wildcards * 6.2 Strict search ! 6.3 Tolerant search 6.4 Link search 6.5 Media search 6.6 Search only in one domain 6.7 Search in categories 6.8 Greek language support 6.9 Block queries 7. Chronological order for result listing 7.1 Text result listing 7.2 Media result listing 8. PDF converter 9. Clean . . .
. . .
of logging data 11. Error messages and Debug mode 12. Delete secondary characters 13. Media search for images, audio streams and videos 13.1 Media indexing 13.2 Not supported media content 13.3 Search for media content 13.4 Statistics for media content 14. Feed support 14.1 XML product feeds 14.2 RDF, RSD, RSS and Atom feeds 15. Result . . .
. . .
Overview 16.2 Definition and configuration 16.3 Activate / disable database 16.4 Backup & Restore of databases 16.5 Copy & and Move 16.6 Enhancing functionality of multiple database support 17. Search in categories 17.1 Hierachical structure 17.2 Parallel structure 18. User suggested sites 19. Vulnerability protection 19.1 Prevent queries . . .
. . .
templates 22.2 Embed the search engine into existing HTML code 22.3 The different style sheet files 23. JSON, XML and RSS result output [ Documentation ] 1. Settings, customizing and statistics If you want to change settings, behavior and design of Sphider-plus, you can do so by means of the Admin interface. There is a wide range of . . .
. . .
interface. There is a wide range of settings foreseen for Sphider-plus. Separated into different submenus like: Sites: - Add Site - Index only the new - Re-index all - Re-index only preferred URLs - Erase Re-index (available also for individual URLs) - Import/export URL list - Approve sites - Banned domains Categories: - Add, edit, delete - . . .
. . .
URL list - Approve sites - Banned domains Categories: - Add, edit, delete - Create new subcategory under Index: - Basic indexing options - Advanced options Clean: - Clean keywords not associated with any link - Clean links not associated with any site - Clean Category table not associated with any site - Clean Media links - Clear Temp table . . .
. . .
- Clear Search log - Clear 'Most Popular Page Links' log - Clear 'Most Popular Media Links' log - Clear Spider log, separate and bulk delete - Clear Thumbnail images, separate and bulk delete - Clear Text cache - Clear Media cache - Clear IDS log file - Clear flood attempts log file - Clear all entries in addurl or banned table - Truncate all . . .
. . .
flood attempts log file - Clear all entries in addurl or banned table - Truncate all tables in database Settings: - General Settings - Index Log Settings - Spider Settings - Search Settings - Order of Result listing - Suggest Options - Page Indexing Weights Database: - Configure up to 5 databases with unlimited number of table sets - Activate . . .
. . .
Database: - Configure up to 5 databases with unlimited number of table sets - Activate separately for 'Admin', 'Search' user and 'Suggest URL user' - Backup / Restore - Copy / Move - Optimize Templates: In order to enable customer's integration of Sphider-plus into existing sites, HTML templates are prepared for Search form Text result . . .
. . .
Search form Text result listing Media result listing Most popular queries etc. Three different designs are offered, which may be selected in submenu 'Settings'. If the layout does not fit the design of your site (which is normal), you may create new designs and modify the appropriate file /templates/My_template/adminstyle.css . . .
. . .
the appropriate file /templates/My_template/adminstyle.css /templates/My_template/userstyle.css Statistics output: - Top keywords (Top 50 with hit counter). - All indexed thumbnails w 3e80 ith ID3 and EXIF info. - Larges pages offering link URL and file size. - Most Popular Searches for text links offering: Link addr., total clicks, last . . .
. . .
Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Searches for media links offering: Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Links (click counter). - Search log offering: Query, Results, Queried at, Time taken, User IP, Country, Host name (Latest100) - Index log offering: . . .
. . .
Country, Host name (Latest100) - Index log offering: File-name, index date and delete option - sitemap log offering: sitemap.xml output sitemap list offering file/page suffixes - IDS log offering: IP, host, query, impact, involved tags, date and time of intrusion. - Flood attempts log offering: IP, query, date and time of flood attempt. - Auto . . .
. . .
attempts log offering: IP, query, date and time of flood attempt. - Auto Re-index log file - Server info offering: Server software, environment, MySQL, PDF-converter, image functions, php.ini file PHP integration, PHP security info. Each item holding lists of details. All text links, media links and thumbnails are active linked. As stated in . . .
. . .
individual Re-indexing, the periodical Re-indexer could be started and aborted in the "Options" menu of each site. 2.5 Preferred Re-indexing Each new URL added to the Admin backend, could be supplied with a priority level. This level will be used by the option 'Re-index only preferred sites'. Level 1 will be interpreted as most important, while . . .
. . .
less all index results will be stored in log files in sub folder /admin/log/ The names of the log files look like: db2_100524-21.47.56_1.html (log file of first thread) db2_100524-21.48.12_2.html (log file of second thread) and is build by the following items: db2 - Number of database. 100524 - Date (May 24, 2010) 21.47.56 - Time when this thread . . .
. . .
spider.php -new1 php spider.php -new2 etc. The IDs will be added to the name of the corresponding log files like: db2_100524-21.47.56_ID1.html (log file of first thread) db2_100524-21.48.12_ID2.html (log file of second thread) IDs could be defined by personal requirements, but the limitations for file names with respect to the OS should be taken . . .
. . .
but will not erase the content of all the other tables. So the check whether the content of a page has changed (MD5sum) is still available for a fast re-index procedure. Once prepared, multithreaded re-indexing could be invoked by starting several threads and adding individual IDs to the option parameter like: php spider.php -erased1 php . . .
. . .
4.4 Ignoring parts of a page Sphider-plus includes an option to exclude parts of pages from being indexed. Thi 775e s can for example be used to prevent search result flooding when certain keywords appear on certain part in most pages (like a header, footer or a menu). Any part of a page between <! sphider_noindex > and <! . . .
. . .
<! sphider_noindex > and <! /sphider_noindex > tags is not indexed, however links in it are followed. 4.5 Ignoring parts of a page by <div id='abc'> Ignoring parts of a page by the <! sphider_noindex > tags requires direct access to the page, because the tags need to be added (edited) to the page. A more flexible method, . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ menu[0-5] / 4.6 Indexing only parts of a page by <div id='abc'> If enabled in Admin settings, the values as defined in the list-file /include/common/divs_use.txt will be used to index only the content between <div . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ table[0-5] / 4.7 Ignore HTML elements defined by <tagname> . . </tagname> This option is foreseen to cooperate with the new HTML5 elements like section, nav, aside, hgroup, article, header, footer, etc. HTML elements . . .
. . .
contain a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */nav[0-5]/ Please keep in mind that element names placed in /include/common/elements_not.txt will be processed case-sensitive. 4.8 Index only HTML elements defined by <tagname> . . </tagname> This is the vice versa . . .
. . .
of all the duplicate content URLs: http://www.example.com/product.php?item=swedish-fish&category=gummy-candy http://www.example.com/product.php?item=swedish-fish&trackingid=1234&sessionid=5678 and Sphider-plus will understand that the duplicates all refer to the canonical URL: http://www.example.com/product.php?item=swedish-fish. The . . .
. . .
charset are supported by the ConvertCharset function and will be used to convert text into UTF-8 Unicode: WINDOWS windows-1250 - Central Europe windows-1251 - Cyrillic windows-1252 - Latin I windows-1253 - Greek windows-1254 - Turkish windows-1255 - Hebrew windows-1256 - Arabic windows-1257 - Baltic windows-1258 - Viet Nam cp874 - Thai - this . . .
. . .
- Baltic windows-1258 - Viet Nam cp874 - Thai - this file is also for DOS DOS cp437 - Latin US cp737 - Greek cp775 - BaltRim cp850 - Latin1 cp852 - Latin2 cp855 - Cyrylic cp857 - Turkish cp860 - Portuguese cp861 - Iceland cp862 - Hebrew cp863 - Canada cp864 - Arabic cp865 - Nordic cp866 - Cyrylic Russian (this is the one, used in IE . . .
. . .
IE Cyrillic (DOS) ) cp869 - Greek2 MAC (Apple) x-mac-cyrillic x-mac-greek x-mac-icelandic x-mac-ce x-mac-roman ISO iso-8859-1 iso-8859-2 iso-8859-3 iso-8859-4 iso-8859-5 iso-8859-6 iso-8859-7 iso-8859-8 iso-8859-9 iso-8859-10 iso-8859-11 iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 . . .
. . .
iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 cp500 cp856 cp875 cp1006 cp1026 koi8-r (Cyrillic) koi8-u (Cyrillic Ukrainian) nextstep us-ascii us-ascii-quotes DSP implementation for NeXT stdenc symbol zdingbat And specially for old Polish programs: mazovia This list is to be read . . .
. . .
Access denied; you need the RELOAD privilege. . . Top 10. Enable real-time output of logging data Up to version 1.5 of Sphider-plus, during index / re-index there was no printout available because: - Several servers, especially on Win32, buffer the output from the script until it terminates before transmitting the results to the browser. - . . .
. . .
is seen. - Some versions of Microsoft Internet Explorer only start to display the page after they have received 256 bytes of output. As progress was not presented during index / re- index procedure, waiting for results became a pain in the neck. Selectable in Admin setting together with the update interval (1 - 10 seconds), AJAX technology was . . .
. . .
characters in front of words Warning: This option should be used with special care and not be activated for non ISO-8859 charsets. Some special characters as part of the word ending might be erased by accidental. Top 13. Media search for images, audio streams and videos 13.1 Media indexing Index of media files is enabled by separated Admin . . .
. . .
and 'Found at' - Total clicks - Last clicked - Query input 'Indexed Image Thumbnails' presenting: - Thumbnail 150 x 100 pixel - Image details like title, filename size of original image, link- and thumb-id - Option to delete the thumbnail In order to open the media files all tables contain active links. Media results are also stored in . . .
. . .
Settings' menu of the admin backend: First option: For results of XML product feeds, present the text extract of 350 characters as part of product attributes. If this option is not activated, all products of the XML product feed will be presented in search results, and the hits of the query string will be highlighted in all involved products. . . .
. . .
content of the database tables (those with the same table prefix) will be destroyed by the restore procedure. 16.5 Copy & Move This section of the Database Management will present only those databases, which are correctly configured, assigned and do have a set of installed tables as described in chapter Definition and configuration. This . . .
. . .
results. Never the less to find these x results, the complete database had to been browsed. Starting with version 2.5, an additional clean option is offered as part of the Admin backend. Main advantage of this option is a significant reduction of the search time for any query, because the content of the db could be limited to offer only x . . .
. . .
<link>http://www.abc.de/images/warp.gif</link> <title>warp.gif</title> <x_size>635</x_size> <y_size>98</y_size> </media_result> <media_result> <num>2</num> <type>audio</type> <url>http://www.abc.de/index.php</url> . . .
. . .
the following content: {"query":"colour","ip":"::1","host_name":"guard_007-hoster","query_time":"2016-03-18 10:56:23 AM","consumed":0.016,"total_results":2,"num_of_results":2,"from":1,"to":2,"text_results":[{"num":1,"weight":"100.0","url":"http:\/\/www.english.le-piaggie.info\/Info_eng.pdf","title":" Info_eng","fulltxt":" . . . . . .
. . .
placed on the ground floor, today's big kitchen with the heating fireplace for open and close mode of . . . ","page_size":"5,661.6kb","domain_name":"www.english.le-piaggie.info"},{"num":2,"weight":"50.0","url":"http:\/\/www.english.le-piaggie.info\/html\/description.html","title":" Description","fulltxt":" . . . cable. Additional active . . .
. . .
D.O.C.G. as far as to the Pratomagno mountains 160 olive-trees Detached house, about 800 m to reach the . . . ","page_size":"26.4 kb","domain_name":"www.english.le-piaggie.info"}]} Top . . .


5a2 All required information. Introduction Release and Legal Info Installation Documentation Change Log [ Documentation Summary ] Preamble: The info presented here is valid only for the latest release of Sphider-plus. At present version 4.2024a published February 12, 2024 is the actual release. 1. Settings, customizing and statistics 2. Indexing . . .
. . .
2. Indexing 2.1 Various options 2.2 Allow other hosts in same domain 2.3 Word stemming 2.4 Periodical Re-indexing 2.5 Preferred indexing 2.6 Multithreaded indexing 2.7 Create thumbnails during index procedure 2.8 Prevent indexing of known malware and pishing pages 2.9 Follow and create sitemap files 2.10 Use private sitemap instead of global . . .
. . .
Sitemap file 3. Using the indexer from command line 3.1 All options 3.2 Multithreaded indexing 4. Keeping pages, words and files from being indexed 4.1 robots.txt 4.2 Must include / must not include string list 4.3 Ignoring links 4.4 Ignoring parts of a page by <! sphider_noindex > 4.5 Ignoring parts of a page by <div id='abc'> . . .
. . .
charset' 6. Search modes 6.1 Search with wildcards * 6.2 Strict search ! 6.3 Tolerant search 6.4 Link search 6.5 Media search 6.6 Search only in one domain 6.7 Search in categories 6.8 Greek language support 6.9 Block queries 7. Chronological order for result listing 7.1 Text result listing 7.2 Media result listing 8. PDF converter 9. Clean . . .
. . .
of logging data 11. Error messages and Debug mode 12. Delete secondary characters 13. Media search for images, audio streams and videos 13.1 Media indexing 13.2 Not supported media content 13.3 Search for media content 13.4 Statistics for media content 14. Feed support 14.1 XML product feeds 14.2 RDF, RSD, RSS and Atom feeds 15. Result . . .
. . .
Overview 16.2 Definition and configuration 16.3 Activate / disable database 16.4 Backup & Restore of databases 16.5 Copy & and Move 16.6 Enhancing functionality of multiple database support 17. Search in categories 17.1 Hierachical structure 17.2 Parallel structure 18. User suggested sites 19. Vulnerability protection 19.1 Prevent queries . . .
. . .
templates 22.2 Embed the search engine into existing HTML code 22.3 The different style sheet files 23. JSON, XML and RSS result output [ Documentation ] 1. Settings, customizing and statistics If you want to change settings, behavior and design of Sphider-plus, you can do so by means of the Admin interface. There is a wide range of . . .
. . .
interface. There is a wide range of settings foreseen for Sphider-plus. Separated into different submenus like: Sites: - Add Site - Index only the new - Re-index all - Re-index only preferred URLs - Erase Re-index (available also for individual URLs) - Import/export URL list - Approve sites - Banned domains Categories: - Add, edit, delete - . . .
. . .
URL list - Approve sites - Banned domains Categories: - Add, edit, delete - Create new subcategory under Index: - Basic indexing options - Advanced options Clean: - Clean keywords not associated with any link - Clean links not associated with any site - Clean Category table not associated with any site - Clean Media links - Clear Temp table . . .
. . .
- Clear Search log - Clear 'Most Popular Page Links' log - Clear 'Most Popular Media Links' log - Clear Spider log, separate and bulk delete - Clear Thumbnail images, separate and bulk delete - Clear Text cache - Clear Media cache - Clear IDS log file - Clear flood attempts log file - Clear all entries in addurl or banned table - Truncate all . . .
. . .
flood attempts log file - Clear all entries in addurl or banned table - Truncate all tables in database Settings: - General Settings - Index Log Settings - Spider Settings - Search Settings - Order of Result listing - Suggest Options - Page Indexing Weights Database: - Configure up to 5 databases with unlimited number of table sets - Activate . . .
. . .
Database: - Configure up to 5 databases with unlimited number of table sets - Activate separately for 'Admin', 'Search' user and 'Suggest URL user' - Backup / Restore - Copy / Move - Optimize Templates: In order to enable customer's integration of Sphider-plus into existing sites, HTML templates are prepared for Search form Text result . . .
. . .
Search form Text result listing Media result listing Most popular queries etc. Three different designs are offered, which may be selected in submenu 'Settings'. If the layout does not fit the design of your site (which is normal), you may create new designs and modify the appropriate file /templates/My_template/adminstyle.css . . .
. . .
the appropriate file /templates/My_template/adminstyle.css /templates/My_template/userstyle.css Statistics output: - Top keywords (Top 50 with hit counter). - All indexed thumbnails w 3e80 ith ID3 and EXIF info. - Larges pages offering link URL and file size. - Most Popular Searches for text links offering: Link addr., total clicks, last . . .
. . .
Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Searches for media links offering: Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Links (click counter). - Search log offering: Query, Results, Queried at, Time taken, User IP, Country, Host name (Latest100) - Index log offering: . . .
. . .
Country, Host name (Latest100) - Index log offering: File-name, index date and delete option - sitemap log offering: sitemap.xml output sitemap list offering file/page suffixes - IDS log offering: IP, host, query, impact, involved tags, date and time of intrusion. - Flood attempts log offering: IP, query, date and time of flood attempt. - Auto . . .
. . .
attempts log offering: IP, query, date and time of flood attempt. - Auto Re-index log file - Server info offering: Server software, environment, MySQL, PDF-converter, image functions, php.ini file PHP integration, PHP security info. Each item holding lists of details. All text links, media links and thumbnails are active linked. As stated in . . .
. . .
individual Re-indexing, the periodical Re-indexer could be started and aborted in the "Options" menu of each site. 2.5 Preferred Re-indexing Each new URL added to the Admin backend, could be supplied with a priority level. This level will be used by the option 'Re-index only preferred sites'. Level 1 will be interpreted as most important, while . . .
. . .
less all index results will be stored in log files in sub folder /admin/log/ The names of the log files look like: db2_100524-21.47.56_1.html (log file of first thread) db2_100524-21.48.12_2.html (log file of second thread) and is build by the following items: db2 - Number of database. 100524 - Date (May 24, 2010) 21.47.56 - Time when this thread . . .
. . .
spider.php -new1 php spider.php -new2 etc. The IDs will be added to the name of the corresponding log files like: db2_100524-21.47.56_ID1.html (log file of first thread) db2_100524-21.48.12_ID2.html (log file of second thread) IDs could be defined by personal requirements, but the limitations for file names with respect to the OS should be taken . . .
. . .
but will not erase the content of all the other tables. So the check whether the content of a page has changed (MD5sum) is still available for a fast re-index procedure. Once prepared, multithreaded re-indexing could be invoked by starting several threads and adding individual IDs to the option parameter like: php spider.php -erased1 php . . .
. . .
4.4 Ignoring parts of a page Sphider-plus includes an option to exclude parts of pages from being indexed. Thi 775e s can for example be used to prevent search result flooding when certain keywords appear on certain part in most pages (like a header, footer or a menu). Any part of a page between <! sphider_noindex > and <! . . .
. . .
<! sphider_noindex > and <! /sphider_noindex > tags is not indexed, however links in it are followed. 4.5 Ignoring parts of a page by <div id='abc'> Ignoring parts of a page by the <! sphider_noindex > tags requires direct access to the page, because the tags need to be added (edited) to the page. A more flexible method, . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ menu[0-5] / 4.6 Indexing only parts of a page by <div id='abc'> If enabled in Admin settings, the values as defined in the list-file /include/common/divs_use.txt will be used to index only the content between <div . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ table[0-5] / 4.7 Ignore HTML elements defined by <tagname> . . </tagname> This option is foreseen to cooperate with the new HTML5 elements like section, nav, aside, hgroup, article, header, footer, etc. HTML elements . . .
. . .
contain a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */nav[0-5]/ Please keep in mind that element names placed in /include/common/elements_not.txt will be processed case-sensitive. 4.8 Index only HTML elements defined by <tagname> . . </tagname> This is the vice versa . . .
. . .
of all the duplicate content URLs: http://www.example.com/product.php?item=swedish-fish&category=gummy-candy http://www.example.com/product.php?item=swedish-fish&trackingid=1234&sessionid=5678 and Sphider-plus will understand that the duplicates all refer to the canonical URL: http://www.example.com/product.php?item=swedish-fish. The . . .
. . .
charset are supported by the ConvertCharset function and will be used to convert text into UTF-8 Unicode: WINDOWS windows-1250 - Central Europe windows-1251 - Cyrillic windows-1252 - Latin I windows-1253 - Greek windows-1254 - Turkish windows-1255 - Hebrew windows-1256 - Arabic windows-1257 - Baltic windows-1258 - Viet Nam cp874 - Thai - this . . .
. . .
- Baltic windows-1258 - Viet Nam cp874 - Thai - this file is also for DOS DOS cp437 - Latin US cp737 - Greek cp775 - BaltRim cp850 - Latin1 cp852 - Latin2 cp855 - Cyrylic cp857 - Turkish cp860 - Portuguese cp861 - Iceland cp862 - Hebrew cp863 - Canada cp864 - Arabic cp865 - Nordic cp866 - Cyrylic Russian (this is the one, used in IE . . .
. . .
IE Cyrillic (DOS) ) cp869 - Greek2 MAC (Apple) x-mac-cyrillic x-mac-greek x-mac-icelandic x-mac-ce x-mac-roman ISO iso-8859-1 iso-8859-2 iso-8859-3 iso-8859-4 iso-8859-5 iso-8859-6 iso-8859-7 iso-8859-8 iso-8859-9 iso-8859-10 iso-8859-11 iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 . . .
. . .
iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 cp500 cp856 cp875 cp1006 cp1026 koi8-r (Cyrillic) koi8-u (Cyrillic Ukrainian) nextstep us-ascii us-ascii-quotes DSP implementation for NeXT stdenc symbol zdingbat And specially for old Polish programs: mazovia This list is to be read . . .
. . .
Access denied; you need the RELOAD privilege. . . Top 10. Enable real-time output of logging data Up to version 1.5 of Sphider-plus, during index / re-index there was no printout available because: - Several servers, especially on Win32, buffer the output from the script until it terminates before transmitting the results to the browser. - . . .
. . .
is seen. - Some versions of Microsoft Internet Explorer only start to display the page after they have received 256 bytes of output. As progress was not presented during index / re- index procedure, waiting for results became a pain in the neck. Selectable in Admin setting together with the update interval (1 - 10 seconds), AJAX technology was . . .
. . .
characters in front of words Warning: This option should be used with special care and not be activated for non ISO-8859 charsets. Some special characters as part of the word ending might be erased by accidental. Top 13. Media search for images, audio streams and videos 13.1 Media indexing Index of media files is enabled by separated Admin . . .
. . .
and 'Found at' - Total clicks - Last clicked - Query input 'Indexed Image Thumbnails' presenting: - Thumbnail 150 x 100 pixel - Image details like title, filename size of original image, link- and thumb-id - Option to delete the thumbnail In order to open the media files all tables contain active links. Media results are also stored in . . .
. . .
Settings' menu of the admin backend: First option: For results of XML product feeds, present the text extract of 350 characters as part of product attributes. If this option is not activated, all products of the XML product feed will be presented in search results, and the hits of the query string will be highlighted in all involved products. . . .
. . .
content of the database tables (those with the same table prefix) will be destroyed by the restore procedure. 16.5 Copy & Move This section of the Database Management will present only those databases, which are correctly configured, assigned and do have a set of installed tables as described in chapter Definition and configuration. This . . .
. . .
results. Never the less to find these x results, the complete database had to been browsed. Starting with version 2.5, an additional clean option is offered as part of the Admin backend. Main advantage of this option is a significant reduction of the search time for any query, because the content of the db could be limited to offer only x . . .
. . .
<link>http://www.abc.de/images/warp.gif</link> <title>warp.gif</title> <x_size>635</x_size> <y_size>98</y_size> </media_result> <media_result> <num>2</num> <type>audio</type> <url>http://www.abc.de/index.php</url> . . .
. . .
the following content: {"query":"colour","ip":"::1","host_name":"guard_007-hoster","query_time":"2016-03-18 10:56:23 AM","consumed":0.016,"total_results":2,"num_of_results":2,"from":1,"to":2,"text_results":[{"num":1,"weight":"100.0","url":"http:\/\/www.english.le-piaggie.info\/Info_eng.pdf","title":" Info_eng","fulltxt":" . . . . . .
. . .
placed on the ground floor, today's big kitchen with the heating fireplace for open and close mode of . . . ","page_size":"5,661.6kb","domain_name":"www.english.le-piaggie.info"},{"num":2,"weight":"50.0","url":"http:\/\/www.english.le-piaggie.info\/html\/description.html","title":" Description","fulltxt":" . . . cable. Additional active . . .
. . .
D.O.C.G. as far as to the Pratomagno mountains 160 olive-trees Detached house, about 800 m to reach the . . . ","page_size":"26.4 kb","domain_name":"www.english.le-piaggie.info"}]} Top . . .


5a2 All required information. Introduction Release and Legal Info Installation Documentation Change Log [ Documentation Summary ] Preamble: The info presented here is valid only for the latest release of Sphider-plus. At present version 4.2024a published February 12, 2024 is the actual release. 1. Settings, customizing and statistics 2. Indexing . . .
. . .
2. Indexing 2.1 Various options 2.2 Allow other hosts in same domain 2.3 Word stemming 2.4 Periodical Re-indexing 2.5 Preferred indexing 2.6 Multithreaded indexing 2.7 Create thumbnails during index procedure 2.8 Prevent indexing of known malware and pishing pages 2.9 Follow and create sitemap files 2.10 Use private sitemap instead of global . . .
. . .
Sitemap file 3. Using the indexer from command line 3.1 All options 3.2 Multithreaded indexing 4. Keeping pages, words and files from being indexed 4.1 robots.txt 4.2 Must include / must not include string list 4.3 Ignoring links 4.4 Ignoring parts of a page by <! sphider_noindex > 4.5 Ignoring parts of a page by <div id='abc'> . . .
. . .
charset' 6. Search modes 6.1 Search with wildcards * 6.2 Strict search ! 6.3 Tolerant search 6.4 Link search 6.5 Media search 6.6 Search only in one domain 6.7 Search in categories 6.8 Greek language support 6.9 Block queries 7. Chronological order for result listing 7.1 Text result listing 7.2 Media result listing 8. PDF converter 9. Clean . . .
. . .
of logging data 11. Error messages and Debug mode 12. Delete secondary characters 13. Media search for images, audio streams and videos 13.1 Media indexing 13.2 Not supported media content 13.3 Search for media content 13.4 Statistics for media content 14. Feed support 14.1 XML product feeds 14.2 RDF, RSD, RSS and Atom feeds 15. Result . . .
. . .
Overview 16.2 Definition and configuration 16.3 Activate / disable database 16.4 Backup & Restore of databases 16.5 Copy & and Move 16.6 Enhancing functionality of multiple database support 17. Search in categories 17.1 Hierachical structure 17.2 Parallel structure 18. User suggested sites 19. Vulnerability protection 19.1 Prevent queries . . .
. . .
templates 22.2 Embed the search engine into existing HTML code 22.3 The different style sheet files 23. JSON, XML and RSS result output [ Documentation ] 1. Settings, customizing and statistics If you want to change settings, behavior and design of Sphider-plus, you can do so by means of the Admin interface. There is a wide range of . . .
. . .
interface. There is a wide range of settings foreseen for Sphider-plus. Separated into different submenus like: Sites: - Add Site - Index only the new - Re-index all - Re-index only preferred URLs - Erase Re-index (available also for individual URLs) - Import/export URL list - Approve sites - Banned domains Categories: - Add, edit, delete - . . .
. . .
URL list - Approve sites - Banned domains Categories: - Add, edit, delete - Create new subcategory under Index: - Basic indexing options - Advanced options Clean: - Clean keywords not associated with any link - Clean links not associated with any site - Clean Category table not associated with any site - Clean Media links - Clear Temp table . . .
. . .
- Clear Search log - Clear 'Most Popular Page Links' log - Clear 'Most Popular Media Links' log - Clear Spider log, separate and bulk delete - Clear Thumbnail images, separate and bulk delete - Clear Text cache - Clear Media cache - Clear IDS log file - Clear flood attempts log file - Clear all entries in addurl or banned table - Truncate all . . .
. . .
flood attempts log file - Clear all entries in addurl or banned table - Truncate all tables in database Settings: - General Settings - Index Log Settings - Spider Settings - Search Settings - Order of Result listing - Suggest Options - Page Indexing Weights Database: - Configure up to 5 databases with unlimited number of table sets - Activate . . .
. . .
Database: - Configure up to 5 databases with unlimited number of table sets - Activate separately for 'Admin', 'Search' user and 'Suggest URL user' - Backup / Restore - Copy / Move - Optimize Templates: In order to enable customer's integration of Sphider-plus into existing sites, HTML templates are prepared for Search form Text result . . .
. . .
Search form Text result listing Media result listing Most popular queries etc. Three different designs are offered, which may be selected in submenu 'Settings'. If the layout does not fit the design of your site (which is normal), you may create new designs and modify the appropriate file /templates/My_template/adminstyle.css . . .
. . .
the appropriate file /templates/My_template/adminstyle.css /templates/My_template/userstyle.css Statistics output: - Top keywords (Top 50 with hit counter). - All indexed thumbnails w 3e80 ith ID3 and EXIF info. - Larges pages offering link URL and file size. - Most Popular Searches for text links offering: Link addr., total clicks, last . . .
. . .
Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Searches for media links offering: Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Links (click counter). - Search log offering: Query, Results, Queried at, Time taken, User IP, Country, Host name (Latest100) - Index log offering: . . .
. . .
Country, Host name (Latest100) - Index log offering: File-name, index date and delete option - sitemap log offering: sitemap.xml output sitemap list offering file/page suffixes - IDS log offering: IP, host, query, impact, involved tags, date and time of intrusion. - Flood attempts log offering: IP, query, date and time of flood attempt. - Auto . . .
. . .
attempts log offering: IP, query, date and time of flood attempt. - Auto Re-index log file - Server info offering: Server software, environment, MySQL, PDF-converter, image functions, php.ini file PHP integration, PHP security info. Each item holding lists of details. All text links, media links and thumbnails are active linked. As stated in . . .
. . .
individual Re-indexing, the periodical Re-indexer could be started and aborted in the "Options" menu of each site. 2.5 Preferred Re-indexing Each new URL added to the Admin backend, could be supplied with a priority level. This level will be used by the option 'Re-index only preferred sites'. Level 1 will be interpreted as most important, while . . .
. . .
less all index results will be stored in log files in sub folder /admin/log/ The names of the log files look like: db2_100524-21.47.56_1.html (log file of first thread) db2_100524-21.48.12_2.html (log file of second thread) and is build by the following items: db2 - Number of database. 100524 - Date (May 24, 2010) 21.47.56 - Time when this thread . . .
. . .
spider.php -new1 php spider.php -new2 etc. The IDs will be added to the name of the corresponding log files like: db2_100524-21.47.56_ID1.html (log file of first thread) db2_100524-21.48.12_ID2.html (log file of second thread) IDs could be defined by personal requirements, but the limitations for file names with respect to the OS should be taken . . .
. . .
but will not erase the content of all the other tables. So the check whether the content of a page has changed (MD5sum) is still available for a fast re-index procedure. Once prepared, multithreaded re-indexing could be invoked by starting several threads and adding individual IDs to the option parameter like: php spider.php -erased1 php . . .
. . .
4.4 Ignoring parts of a page Sphider-plus includes an option to exclude parts of pages from being indexed. Thi 775e s can for example be used to prevent search result flooding when certain keywords appear on certain part in most pages (like a header, footer or a menu). Any part of a page between <! sphider_noindex > and <! . . .
. . .
<! sphider_noindex > and <! /sphider_noindex > tags is not indexed, however links in it are followed. 4.5 Ignoring parts of a page by <div id='abc'> Ignoring parts of a page by the <! sphider_noindex > tags requires direct access to the page, because the tags need to be added (edited) to the page. A more flexible method, . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ menu[0-5] / 4.6 Indexing only parts of a page by <div id='abc'> If enabled in Admin settings, the values as defined in the list-file /include/common/divs_use.txt will be used to index only the content between <div . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ table[0-5] / 4.7 Ignore HTML elements defined by <tagname> . . </tagname> This option is foreseen to cooperate with the new HTML5 elements like section, nav, aside, hgroup, article, header, footer, etc. HTML elements . . .
. . .
contain a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */nav[0-5]/ Please keep in mind that element names placed in /include/common/elements_not.txt will be processed case-sensitive. 4.8 Index only HTML elements defined by <tagname> . . </tagname> This is the vice versa . . .
. . .
of all the duplicate content URLs: http://www.example.com/product.php?item=swedish-fish&category=gummy-candy http://www.example.com/product.php?item=swedish-fish&trackingid=1234&sessionid=5678 and Sphider-plus will understand that the duplicates all refer to the canonical URL: http://www.example.com/product.php?item=swedish-fish. The . . .
. . .
charset are supported by the ConvertCharset function and will be used to convert text into UTF-8 Unicode: WINDOWS windows-1250 - Central Europe windows-1251 - Cyrillic windows-1252 - Latin I windows-1253 - Greek windows-1254 - Turkish windows-1255 - Hebrew windows-1256 - Arabic windows-1257 - Baltic windows-1258 - Viet Nam cp874 - Thai - this . . .
. . .
- Baltic windows-1258 - Viet Nam cp874 - Thai - this file is also for DOS DOS cp437 - Latin US cp737 - Greek cp775 - BaltRim cp850 - Latin1 cp852 - Latin2 cp855 - Cyrylic cp857 - Turkish cp860 - Portuguese cp861 - Iceland cp862 - Hebrew cp863 - Canada cp864 - Arabic cp865 - Nordic cp866 - Cyrylic Russian (this is the one, used in IE . . .
. . .
IE Cyrillic (DOS) ) cp869 - Greek2 MAC (Apple) x-mac-cyrillic x-mac-greek x-mac-icelandic x-mac-ce x-mac-roman ISO iso-8859-1 iso-8859-2 iso-8859-3 iso-8859-4 iso-8859-5 iso-8859-6 iso-8859-7 iso-8859-8 iso-8859-9 iso-8859-10 iso-8859-11 iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 . . .
. . .
iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 cp500 cp856 cp875 cp1006 cp1026 koi8-r (Cyrillic) koi8-u (Cyrillic Ukrainian) nextstep us-ascii us-ascii-quotes DSP implementation for NeXT stdenc symbol zdingbat And specially for old Polish programs: mazovia This list is to be read . . .
. . .
Access denied; you need the RELOAD privilege. . . Top 10. Enable real-time output of logging data Up to version 1.5 of Sphider-plus, during index / re-index there was no printout available because: - Several servers, especially on Win32, buffer the output from the script until it terminates before transmitting the results to the browser. - . . .
. . .
is seen. - Some versions of Microsoft Internet Explorer only start to display the page after they have received 256 bytes of output. As progress was not presented during index / re- index procedure, waiting for results became a pain in the neck. Selectable in Admin setting together with the update interval (1 - 10 seconds), AJAX technology was . . .
. . .
characters in front of words Warning: This option should be used with special care and not be activated for non ISO-8859 charsets. Some special characters as part of the word ending might be erased by accidental. Top 13. Media search for images, audio streams and videos 13.1 Media indexing Index of media files is enabled by separated Admin . . .
. . .
and 'Found at' - Total clicks - Last clicked - Query input 'Indexed Image Thumbnails' presenting: - Thumbnail 150 x 100 pixel - Image details like title, filename size of original image, link- and thumb-id - Option to delete the thumbnail In order to open the media files all tables contain active links. Media results are also stored in . . .
. . .
Settings' menu of the admin backend: First option: For results of XML product feeds, present the text extract of 350 characters as part of product attributes. If this option is not activated, all products of the XML product feed will be presented in search results, and the hits of the query string will be highlighted in all involved products. . . .
. . .
content of the database tables (those with the same table prefix) will be destroyed by the restore procedure. 16.5 Copy & Move This section of the Database Management will present only those databases, which are correctly configured, assigned and do have a set of installed tables as described in chapter Definition and configuration. This . . .
. . .
results. Never the less to find these x results, the complete database had to been browsed. Starting with version 2.5, an additional clean option is offered as part of the Admin backend. Main advantage of this option is a significant reduction of the search time for any query, because the content of the db could be limited to offer only x . . .
. . .
<link>http://www.abc.de/images/warp.gif</link> <title>warp.gif</title> <x_size>635</x_size> <y_size>98</y_size> </media_result> <media_result> <num>2</num> <type>audio</type> <url>http://www.abc.de/index.php</url> . . .
. . .
the following content: {"query":"colour","ip":"::1","host_name":"guard_007-hoster","query_time":"2016-03-18 10:56:23 AM","consumed":0.016,"total_results":2,"num_of_results":2,"from":1,"to":2,"text_results":[{"num":1,"weight":"100.0","url":"http:\/\/www.english.le-piaggie.info\/Info_eng.pdf","title":" Info_eng","fulltxt":" . . . . . .
. . .
placed on the ground floor, today's big kitchen with the heating fireplace for open and close mode of . . . ","page_size":"5,661.6kb","domain_name":"www.english.le-piaggie.info"},{"num":2,"weight":"50.0","url":"http:\/\/www.english.le-piaggie.info\/html\/description.html","title":" Description","fulltxt":" . . . cable. Additional active . . .
. . .
D.O.C.G. as far as to the Pratomagno mountains 160 olive-trees Detached house, about 800 m to reach the . . . ","page_size":"26.4 kb","domain_name":"www.english.le-piaggie.info"}]} Top . . .


5a2 All required information. Introduction Release and Legal Info Installation Documentation Change Log [ Documentation Summary ] Preamble: The info presented here is valid only for the latest release of Sphider-plus. At present version 4.2024a published February 12, 2024 is the actual release. 1. Settings, customizing and statistics 2. Indexing . . .
. . .
2. Indexing 2.1 Various options 2.2 Allow other hosts in same domain 2.3 Word stemming 2.4 Periodical Re-indexing 2.5 Preferred indexing 2.6 Multithreaded indexing 2.7 Create thumbnails during index procedure 2.8 Prevent indexing of known malware and pishing pages 2.9 Follow and create sitemap files 2.10 Use private sitemap instead of global . . .
. . .
Sitemap file 3. Using the indexer from command line 3.1 All options 3.2 Multithreaded indexing 4. Keeping pages, words and files from being indexed 4.1 robots.txt 4.2 Must include / must not include string list 4.3 Ignoring links 4.4 Ignoring parts of a page by <! sphider_noindex > 4.5 Ignoring parts of a page by <div id='abc'> . . .
. . .
charset' 6. Search modes 6.1 Search with wildcards * 6.2 Strict search ! 6.3 Tolerant search 6.4 Link search 6.5 Media search 6.6 Search only in one domain 6.7 Search in categories 6.8 Greek language support 6.9 Block queries 7. Chronological order for result listing 7.1 Text result listing 7.2 Media result listing 8. PDF converter 9. Clean . . .
. . .
of logging data 11. Error messages and Debug mode 12. Delete secondary characters 13. Media search for images, audio streams and videos 13.1 Media indexing 13.2 Not supported media content 13.3 Search for media content 13.4 Statistics for media content 14. Feed support 14.1 XML product feeds 14.2 RDF, RSD, RSS and Atom feeds 15. Result . . .
. . .
Overview 16.2 Definition and configuration 16.3 Activate / disable database 16.4 Backup & Restore of databases 16.5 Copy & and Move 16.6 Enhancing functionality of multiple database support 17. Search in categories 17.1 Hierachical structure 17.2 Parallel structure 18. User suggested sites 19. Vulnerability protection 19.1 Prevent queries . . .
. . .
templates 22.2 Embed the search engine into existing HTML code 22.3 The different style sheet files 23. JSON, XML and RSS result output [ Documentation ] 1. Settings, customizing and statistics If you want to change settings, behavior and design of Sphider-plus, you can do so by means of the Admin interface. There is a wide range of . . .
. . .
interface. There is a wide range of settings foreseen for Sphider-plus. Separated into different submenus like: Sites: - Add Site - Index only the new - Re-index all - Re-index only preferred URLs - Erase Re-index (available also for individual URLs) - Import/export URL list - Approve sites - Banned domains Categories: - Add, edit, delete - . . .
. . .
URL list - Approve sites - Banned domains Categories: - Add, edit, delete - Create new subcategory under Index: - Basic indexing options - Advanced options Clean: - Clean keywords not associated with any link - Clean links not associated with any site - Clean Category table not associated with any site - Clean Media links - Clear Temp table . . .
. . .
- Clear Search log - Clear 'Most Popular Page Links' log - Clear 'Most Popular Media Links' log - Clear Spider log, separate and bulk delete - Clear Thumbnail images, separate and bulk delete - Clear Text cache - Clear Media cache - Clear IDS log file - Clear flood attempts log file - Clear all entries in addurl or banned table - Truncate all . . .
. . .
flood attempts log file - Clear all entries in addurl or banned table - Truncate all tables in database Settings: - General Settings - Index Log Settings - Spider Settings - Search Settings - Order of Result listing - Suggest Options - Page Indexing Weights Database: - Configure up to 5 databases with unlimited number of table sets - Activate . . .
. . .
Database: - Configure up to 5 databases with unlimited number of table sets - Activate separately for 'Admin', 'Search' user and 'Suggest URL user' - Backup / Restore - Copy / Move - Optimize Templates: In order to enable customer's integration of Sphider-plus into existing sites, HTML templates are prepared for Search form Text result . . .
. . .
Search form Text result listing Media result listing Most popular queries etc. Three different designs are offered, which may be selected in submenu 'Settings'. If the layout does not fit the design of your site (which is normal), you may create new designs and modify the appropriate file /templates/My_template/adminstyle.css . . .
. . .
the appropriate file /templates/My_template/adminstyle.css /templates/My_template/userstyle.css Statistics output: - Top keywords (Top 50 with hit counter). - All indexed thumbnails w 3e80 ith ID3 and EXIF info. - Larges pages offering link URL and file size. - Most Popular Searches for text links offering: Link addr., total clicks, last . . .
. . .
Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Searches for media links offering: Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Links (click counter). - Search log offering: Query, Results, Queried at, Time taken, User IP, Country, Host name (Latest100) - Index log offering: . . .
. . .
Country, Host name (Latest100) - Index log offering: File-name, index date and delete option - sitemap log offering: sitemap.xml output sitemap list offering file/page suffixes - IDS log offering: IP, host, query, impact, involved tags, date and time of intrusion. - Flood attempts log offering: IP, query, date and time of flood attempt. - Auto . . .
. . .
attempts log offering: IP, query, date and time of flood attempt. - Auto Re-index log file - Server info offering: Server software, environment, MySQL, PDF-converter, image functions, php.ini file PHP integration, PHP security info. Each item holding lists of details. All text links, media links and thumbnails are active linked. As stated in . . .
. . .
individual Re-indexing, the periodical Re-indexer could be started and aborted in the "Options" menu of each site. 2.5 Preferred Re-indexing Each new URL added to the Admin backend, could be supplied with a priority level. This level will be used by the option 'Re-index only preferred sites'. Level 1 will be interpreted as most important, while . . .
. . .
less all index results will be stored in log files in sub folder /admin/log/ The names of the log files look like: db2_100524-21.47.56_1.html (log file of first thread) db2_100524-21.48.12_2.html (log file of second thread) and is build by the following items: db2 - Number of database. 100524 - Date (May 24, 2010) 21.47.56 - Time when this thread . . .
. . .
spider.php -new1 php spider.php -new2 etc. The IDs will be added to the name of the corresponding log files like: db2_100524-21.47.56_ID1.html (log file of first thread) db2_100524-21.48.12_ID2.html (log file of second thread) IDs could be defined by personal requirements, but the limitations for file names with respect to the OS should be taken . . .
. . .
but will not erase the content of all the other tables. So the check whether the content of a page has changed (MD5sum) is still available for a fast re-index procedure. Once prepared, multithreaded re-indexing could be invoked by starting several threads and adding individual IDs to the option parameter like: php spider.php -erased1 php . . .
. . .
4.4 Ignoring parts of a page Sphider-plus includes an option to exclude parts of pages from being indexed. Thi 775e s can for example be used to prevent search result flooding when certain keywords appear on certain part in most pages (like a header, footer or a menu). Any part of a page between <! sphider_noindex > and <! . . .
. . .
<! sphider_noindex > and <! /sphider_noindex > tags is not indexed, however links in it are followed. 4.5 Ignoring parts of a page by <div id='abc'> Ignoring parts of a page by the <! sphider_noindex > tags requires direct access to the page, because the tags need to be added (edited) to the page. A more flexible method, . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ menu[0-5] / 4.6 Indexing only parts of a page by <div id='abc'> If enabled in Admin settings, the values as defined in the list-file /include/common/divs_use.txt will be used to index only the content between <div . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ table[0-5] / 4.7 Ignore HTML elements defined by <tagname> . . </tagname> This option is foreseen to cooperate with the new HTML5 elements like section, nav, aside, hgroup, article, header, footer, etc. HTML elements . . .
. . .
contain a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */nav[0-5]/ Please keep in mind that element names placed in /include/common/elements_not.txt will be processed case-sensitive. 4.8 Index only HTML elements defined by <tagname> . . </tagname> This is the vice versa . . .
. . .
of all the duplicate content URLs: http://www.example.com/product.php?item=swedish-fish&category=gummy-candy http://www.example.com/product.php?item=swedish-fish&trackingid=1234&sessionid=5678 and Sphider-plus will understand that the duplicates all refer to the canonical URL: http://www.example.com/product.php?item=swedish-fish. The . . .
. . .
charset are supported by the ConvertCharset function and will be used to convert text into UTF-8 Unicode: WINDOWS windows-1250 - Central Europe windows-1251 - Cyrillic windows-1252 - Latin I windows-1253 - Greek windows-1254 - Turkish windows-1255 - Hebrew windows-1256 - Arabic windows-1257 - Baltic windows-1258 - Viet Nam cp874 - Thai - this . . .
. . .
- Baltic windows-1258 - Viet Nam cp874 - Thai - this file is also for DOS DOS cp437 - Latin US cp737 - Greek cp775 - BaltRim cp850 - Latin1 cp852 - Latin2 cp855 - Cyrylic cp857 - Turkish cp860 - Portuguese cp861 - Iceland cp862 - Hebrew cp863 - Canada cp864 - Arabic cp865 - Nordic cp866 - Cyrylic Russian (this is the one, used in IE . . .
. . .
IE Cyrillic (DOS) ) cp869 - Greek2 MAC (Apple) x-mac-cyrillic x-mac-greek x-mac-icelandic x-mac-ce x-mac-roman ISO iso-8859-1 iso-8859-2 iso-8859-3 iso-8859-4 iso-8859-5 iso-8859-6 iso-8859-7 iso-8859-8 iso-8859-9 iso-8859-10 iso-8859-11 iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 . . .
. . .
iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 cp500 cp856 cp875 cp1006 cp1026 koi8-r (Cyrillic) koi8-u (Cyrillic Ukrainian) nextstep us-ascii us-ascii-quotes DSP implementation for NeXT stdenc symbol zdingbat And specially for old Polish programs: mazovia This list is to be read . . .
. . .
Access denied; you need the RELOAD privilege. . . Top 10. Enable real-time output of logging data Up to version 1.5 of Sphider-plus, during index / re-index there was no printout available because: - Several servers, especially on Win32, buffer the output from the script until it terminates before transmitting the results to the browser. - . . .
. . .
is seen. - Some versions of Microsoft Internet Explorer only start to display the page after they have received 256 bytes of output. As progress was not presented during index / re- index procedure, waiting for results became a pain in the neck. Selectable in Admin setting together with the update interval (1 - 10 seconds), AJAX technology was . . .
. . .
characters in front of words Warning: This option should be used with special care and not be activated for non ISO-8859 charsets. Some special characters as part of the word ending might be erased by accidental. Top 13. Media search for images, audio streams and videos 13.1 Media indexing Index of media files is enabled by separated Admin . . .
. . .
and 'Found at' - Total clicks - Last clicked - Query input 'Indexed Image Thumbnails' presenting: - Thumbnail 150 x 100 pixel - Image details like title, filename size of original image, link- and thumb-id - Option to delete the thumbnail In order to open the media files all tables contain active links. Media results are also stored in . . .
. . .
Settings' menu of the admin backend: First option: For results of XML product feeds, present the text extract of 350 characters as part of product attributes. If this option is not activated, all products of the XML product feed will be presented in search results, and the hits of the query string will be highlighted in all involved products. . . .
. . .
content of the database tables (those with the same table prefix) will be destroyed by the restore procedure. 16.5 Copy & Move This section of the Database Management will present only those databases, which are correctly configured, assigned and do have a set of installed tables as described in chapter Definition and configuration. This . . .
. . .
results. Never the less to find these x results, the complete database had to been browsed. Starting with version 2.5, an additional clean option is offered as part of the Admin backend. Main advantage of this option is a significant reduction of the search time for any query, because the content of the db could be limited to offer only x . . .
. . .
<link>http://www.abc.de/images/warp.gif</link> <title>warp.gif</title> <x_size>635</x_size> <y_size>98</y_size> </media_result> <media_result> <num>2</num> <type>audio</type> <url>http://www.abc.de/index.php</url> . . .
. . .
the following content: {"query":"colour","ip":"::1","host_name":"guard_007-hoster","query_time":"2016-03-18 10:56:23 AM","consumed":0.016,"total_results":2,"num_of_results":2,"from":1,"to":2,"text_results":[{"num":1,"weight":"100.0","url":"http:\/\/www.english.le-piaggie.info\/Info_eng.pdf","title":" Info_eng","fulltxt":" . . . . . .
. . .
placed on the ground floor, today's big kitchen with the heating fireplace for open and close mode of . . . ","page_size":"5,661.6kb","domain_name":"www.english.le-piaggie.info"},{"num":2,"weight":"50.0","url":"http:\/\/www.english.le-piaggie.info\/html\/description.html","title":" Description","fulltxt":" . . . cable. Additional active . . .
. . .
D.O.C.G. as far as to the Pratomagno mountains 160 olive-trees Detached house, about 800 m to reach the . . . ","page_size":"26.4 kb","domain_name":"www.english.le-piaggie.info"}]} Top . . .


5a2 All required information. Introduction Release and Legal Info Installation Documentation Change Log [ Documentation Summary ] Preamble: The info presented here is valid only for the latest release of Sphider-plus. At present version 4.2024a published February 12, 2024 is the actual release. 1. Settings, customizing and statistics 2. Indexing . . .
. . .
2. Indexing 2.1 Various options 2.2 Allow other hosts in same domain 2.3 Word stemming 2.4 Periodical Re-indexing 2.5 Preferred indexing 2.6 Multithreaded indexing 2.7 Create thumbnails during index procedure 2.8 Prevent indexing of known malware and pishing pages 2.9 Follow and create sitemap files 2.10 Use private sitemap instead of global . . .
. . .
Sitemap file 3. Using the indexer from command line 3.1 All options 3.2 Multithreaded indexing 4. Keeping pages, words and files from being indexed 4.1 robots.txt 4.2 Must include / must not include string list 4.3 Ignoring links 4.4 Ignoring parts of a page by <! sphider_noindex > 4.5 Ignoring parts of a page by <div id='abc'> . . .
. . .
charset' 6. Search modes 6.1 Search with wildcards * 6.2 Strict search ! 6.3 Tolerant search 6.4 Link search 6.5 Media search 6.6 Search only in one domain 6.7 Search in categories 6.8 Greek language support 6.9 Block queries 7. Chronological order for result listing 7.1 Text result listing 7.2 Media result listing 8. PDF converter 9. Clean . . .
. . .
of logging data 11. Error messages and Debug mode 12. Delete secondary characters 13. Media search for images, audio streams and videos 13.1 Media indexing 13.2 Not supported media content 13.3 Search for media content 13.4 Statistics for media content 14. Feed support 14.1 XML product feeds 14.2 RDF, RSD, RSS and Atom feeds 15. Result . . .
. . .
Overview 16.2 Definition and configuration 16.3 Activate / disable database 16.4 Backup & Restore of databases 16.5 Copy & and Move 16.6 Enhancing functionality of multiple database support 17. Search in categories 17.1 Hierachical structure 17.2 Parallel structure 18. User suggested sites 19. Vulnerability protection 19.1 Prevent queries . . .
. . .
templates 22.2 Embed the search engine into existing HTML code 22.3 The different style sheet files 23. JSON, XML and RSS result output [ Documentation ] 1. Settings, customizing and statistics If you want to change settings, behavior and design of Sphider-plus, you can do so by means of the Admin interface. There is a wide range of . . .
. . .
interface. There is a wide range of settings foreseen for Sphider-plus. Separated into different submenus like: Sites: - Add Site - Index only the new - Re-index all - Re-index only preferred URLs - Erase Re-index (available also for individual URLs) - Import/export URL list - Approve sites - Banned domains Categories: - Add, edit, delete - . . .
. . .
URL list - Approve sites - Banned domains Categories: - Add, edit, delete - Create new subcategory under Index: - Basic indexing options - Advanced options Clean: - Clean keywords not associated with any link - Clean links not associated with any site - Clean Category table not associated with any site - Clean Media links - Clear Temp table . . .
. . .
- Clear Search log - Clear 'Most Popular Page Links' log - Clear 'Most Popular Media Links' log - Clear Spider log, separate and bulk delete - Clear Thumbnail images, separate and bulk delete - Clear Text cache - Clear Media cache - Clear IDS log file - Clear flood attempts log file - Clear all entries in addurl or banned table - Truncate all . . .
. . .
flood attempts log file - Clear all entries in addurl or banned table - Truncate all tables in database Settings: - General Settings - Index Log Settings - Spider Settings - Search Settings - Order of Result listing - Suggest Options - Page Indexing Weights Database: - Configure up to 5 databases with unlimited number of table sets - Activate . . .
. . .
Database: - Configure up to 5 databases with unlimited number of table sets - Activate separately for 'Admin', 'Search' user and 'Suggest URL user' - Backup / Restore - Copy / Move - Optimize Templates: In order to enable customer's integration of Sphider-plus into existing sites, HTML templates are prepared for Search form Text result . . .
. . .
Search form Text result listing Media result listing Most popular queries etc. Three different designs are offered, which may be selected in submenu 'Settings'. If the layout does not fit the design of your site (which is normal), you may create new designs and modify the appropriate file /templates/My_template/adminstyle.css . . .
. . .
the appropriate file /templates/My_template/adminstyle.css /templates/My_template/userstyle.css Statistics output: - Top keywords (Top 50 with hit counter). - All indexed thumbnails w 3e80 ith ID3 and EXIF info. - Larges pages offering link URL and file size. - Most Popular Searches for text links offering: Link addr., total clicks, last . . .
. . .
Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Searches for media links offering: Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Links (click counter). - Search log offering: Query, Results, Queried at, Time taken, User IP, Country, Host name (Latest100) - Index log offering: . . .
. . .
Country, Host name (Latest100) - Index log offering: File-name, index date and delete option - sitemap log offering: sitemap.xml output sitemap list offering file/page suffixes - IDS log offering: IP, host, query, impact, involved tags, date and time of intrusion. - Flood attempts log offering: IP, query, date and time of flood attempt. - Auto . . .
. . .
attempts log offering: IP, query, date and time of flood attempt. - Auto Re-index log file - Server info offering: Server software, environment, MySQL, PDF-converter, image functions, php.ini file PHP integration, PHP security info. Each item holding lists of details. All text links, media links and thumbnails are active linked. As stated in . . .
. . .
individual Re-indexing, the periodical Re-indexer could be started and aborted in the "Options" menu of each site. 2.5 Preferred Re-indexing Each new URL added to the Admin backend, could be supplied with a priority level. This level will be used by the option 'Re-index only preferred sites'. Level 1 will be interpreted as most important, while . . .
. . .
less all index results will be stored in log files in sub folder /admin/log/ The names of the log files look like: db2_100524-21.47.56_1.html (log file of first thread) db2_100524-21.48.12_2.html (log file of second thread) and is build by the following items: db2 - Number of database. 100524 - Date (May 24, 2010) 21.47.56 - Time when this thread . . .
. . .
spider.php -new1 php spider.php -new2 etc. The IDs will be added to the name of the corresponding log files like: db2_100524-21.47.56_ID1.html (log file of first thread) db2_100524-21.48.12_ID2.html (log file of second thread) IDs could be defined by personal requirements, but the limitations for file names with respect to the OS should be taken . . .
. . .
but will not erase the content of all the other tables. So the check whether the content of a page has changed (MD5sum) is still available for a fast re-index procedure. Once prepared, multithreaded re-indexing could be invoked by starting several threads and adding individual IDs to the option parameter like: php spider.php -erased1 php . . .
. . .
4.4 Ignoring parts of a page Sphider-plus includes an option to exclude parts of pages from being indexed. Thi 775e s can for example be used to prevent search result flooding when certain keywords appear on certain part in most pages (like a header, footer or a menu). Any part of a page between <! sphider_noindex > and <! . . .
. . .
<! sphider_noindex > and <! /sphider_noindex > tags is not indexed, however links in it are followed. 4.5 Ignoring parts of a page by <div id='abc'> Ignoring parts of a page by the <! sphider_noindex > tags requires direct access to the page, because the tags need to be added (edited) to the page. A more flexible method, . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ menu[0-5] / 4.6 Indexing only parts of a page by <div id='abc'> If enabled in Admin settings, the values as defined in the list-file /include/common/divs_use.txt will be used to index only the content between <div . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ table[0-5] / 4.7 Ignore HTML elements defined by <tagname> . . </tagname> This option is foreseen to cooperate with the new HTML5 elements like section, nav, aside, hgroup, article, header, footer, etc. HTML elements . . .
. . .
contain a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */nav[0-5]/ Please keep in mind that element names placed in /include/common/elements_not.txt will be processed case-sensitive. 4.8 Index only HTML elements defined by <tagname> . . </tagname> This is the vice versa . . .
. . .
of all the duplicate content URLs: http://www.example.com/product.php?item=swedish-fish&category=gummy-candy http://www.example.com/product.php?item=swedish-fish&trackingid=1234&sessionid=5678 and Sphider-plus will understand that the duplicates all refer to the canonical URL: http://www.example.com/product.php?item=swedish-fish. The . . .
. . .
charset are supported by the ConvertCharset function and will be used to convert text into UTF-8 Unicode: WINDOWS windows-1250 - Central Europe windows-1251 - Cyrillic windows-1252 - Latin I windows-1253 - Greek windows-1254 - Turkish windows-1255 - Hebrew windows-1256 - Arabic windows-1257 - Baltic windows-1258 - Viet Nam cp874 - Thai - this . . .
. . .
- Baltic windows-1258 - Viet Nam cp874 - Thai - this file is also for DOS DOS cp437 - Latin US cp737 - Greek cp775 - BaltRim cp850 - Latin1 cp852 - Latin2 cp855 - Cyrylic cp857 - Turkish cp860 - Portuguese cp861 - Iceland cp862 - Hebrew cp863 - Canada cp864 - Arabic cp865 - Nordic cp866 - Cyrylic Russian (this is the one, used in IE . . .
. . .
IE Cyrillic (DOS) ) cp869 - Greek2 MAC (Apple) x-mac-cyrillic x-mac-greek x-mac-icelandic x-mac-ce x-mac-roman ISO iso-8859-1 iso-8859-2 iso-8859-3 iso-8859-4 iso-8859-5 iso-8859-6 iso-8859-7 iso-8859-8 iso-8859-9 iso-8859-10 iso-8859-11 iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 . . .
. . .
iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 cp500 cp856 cp875 cp1006 cp1026 koi8-r (Cyrillic) koi8-u (Cyrillic Ukrainian) nextstep us-ascii us-ascii-quotes DSP implementation for NeXT stdenc symbol zdingbat And specially for old Polish programs: mazovia This list is to be read . . .
. . .
Access denied; you need the RELOAD privilege. . . Top 10. Enable real-time output of logging data Up to version 1.5 of Sphider-plus, during index / re-index there was no printout available because: - Several servers, especially on Win32, buffer the output from the script until it terminates before transmitting the results to the browser. - . . .
. . .
is seen. - Some versions of Microsoft Internet Explorer only start to display the page after they have received 256 bytes of output. As progress was not presented during index / re- index procedure, waiting for results became a pain in the neck. Selectable in Admin setting together with the update interval (1 - 10 seconds), AJAX technology was . . .
. . .
characters in front of words Warning: This option should be used with special care and not be activated for non ISO-8859 charsets. Some special characters as part of the word ending might be erased by accidental. Top 13. Media search for images, audio streams and videos 13.1 Media indexing Index of media files is enabled by separated Admin . . .
. . .
and 'Found at' - Total clicks - Last clicked - Query input 'Indexed Image Thumbnails' presenting: - Thumbnail 150 x 100 pixel - Image details like title, filename size of original image, link- and thumb-id - Option to delete the thumbnail In order to open the media files all tables contain active links. Media results are also stored in . . .
. . .
Settings' menu of the admin backend: First option: For results of XML product feeds, present the text extract of 350 characters as part of product attributes. If this option is not activated, all products of the XML product feed will be presented in search results, and the hits of the query string will be highlighted in all involved products. . . .
. . .
content of the database tables (those with the same table prefix) will be destroyed by the restore procedure. 16.5 Copy & Move This section of the Database Management will present only those databases, which are correctly configured, assigned and do have a set of installed tables as described in chapter Definition and configuration. This . . .
. . .
results. Never the less to find these x results, the complete database had to been browsed. Starting with version 2.5, an additional clean option is offered as part of the Admin backend. Main advantage of this option is a significant reduction of the search time for any query, because the content of the db could be limited to offer only x . . .
. . .
<link>http://www.abc.de/images/warp.gif</link> <title>warp.gif</title> <x_size>635</x_size> <y_size>98</y_size> </media_result> <media_result> <num>2</num> <type>audio</type> <url>http://www.abc.de/index.php</url> . . .
. . .
the following content: {"query":"colour","ip":"::1","host_name":"guard_007-hoster","query_time":"2016-03-18 10:56:23 AM","consumed":0.016,"total_results":2,"num_of_results":2,"from":1,"to":2,"text_results":[{"num":1,"weight":"100.0","url":"http:\/\/wwwenglishle-piaggieinfo\/Infoengpdftitle-piaggie.info\/Info_eng.pdf","title":" Info_eng","fulltxt":" . . . . . .
. . .
placed on the ground floor, today's big kitchen with the heating fireplace for open and close mode of . . . ","page_size":"5,661.6kb","domain_name":"wwwenglishle-piaggieinfo\/Infoengpdftitle-piaggie.info"},{"num":2,"weight":"50.0","url":"http:\/\/wwwenglishle-piaggieinfo\/Infoengpdftitle-piaggie.info\/html\/description.html","title":" Description","fulltxt":" . . . cable. Additional active . . .
. . .
D.O.C.G. as far as to the Pratomagno mountains 160 olive-trees Detached house, about 800 m to reach the . . . ","page_size":"26.4 kb","domain_name":"wwwenglishle-piaggieinfo\/Infoengpdftitle-piaggie.info"}]} Top . . .


5a2 All required information. Introduction Release and Legal Info Installation Documentation Change Log [ Documentation Summary ] Preamble: The info presented here is valid only for the latest release of Sphider-plus. At present version 4.2024a published February 12, 2024 is the actual release. 1. Settings, customizing and statistics 2. Indexing . . .
. . .
2. Indexing 2.1 Various options 2.2 Allow other hosts in same domain 2.3 Word stemming 2.4 Periodical Re-indexing 2.5 Preferred indexing 2.6 Multithreaded indexing 2.7 Create thumbnails during index procedure 2.8 Prevent indexing of known malware and pishing pages 2.9 Follow and create sitemap files 2.10 Use private sitemap instead of global . . .
. . .
Sitemap file 3. Using the indexer from command line 3.1 All options 3.2 Multithreaded indexing 4. Keeping pages, words and files from being indexed 4.1 robots.txt 4.2 Must include / must not include string list 4.3 Ignoring links 4.4 Ignoring parts of a page by <! sphider_noindex > 4.5 Ignoring parts of a page by <div id='abc'> . . .
. . .
charset' 6. Search modes 6.1 Search with wildcards * 6.2 Strict search ! 6.3 Tolerant search 6.4 Link search 6.5 Media search 6.6 Search only in one domain 6.7 Search in categories 6.8 Greek language support 6.9 Block queries 7. Chronological order for result listing 7.1 Text result listing 7.2 Media result listing 8. PDF converter 9. Clean . . .
. . .
of logging data 11. Error messages and Debug mode 12. Delete secondary characters 13. Media search for images, audio streams and videos 13.1 Media indexing 13.2 Not supported media content 13.3 Search for media content 13.4 Statistics for media content 14. Feed support 14.1 XML product feeds 14.2 RDF, RSD, RSS and Atom feeds 15. Result . . .
. . .
Overview 16.2 Definition and configuration 16.3 Activate / disable database 16.4 Backup & Restore of databases 16.5 Copy & and Move 16.6 Enhancing functionality of multiple database support 17. Search in categories 17.1 Hierachical structure 17.2 Parallel structure 18. User suggested sites 19. Vulnerability protection 19.1 Prevent queries . . .
. . .
templates 22.2 Embed the search engine into existing HTML code 22.3 The different style sheet files 23. JSON, XML and RSS result output [ Documentation ] 1. Settings, customizing and statistics If you want to change settings, behavior and design of Sphider-plus, you can do so by means of the Admin interface. There is a wide range of . . .
. . .
interface. There is a wide range of settings foreseen for Sphider-plus. Separated into different submenus like: Sites: - Add Site - Index only the new - Re-index all - Re-index only preferred URLs - Erase Re-index (available also for individual URLs) - Import/export URL list - Approve sites - Banned domains Categories: - Add, edit, delete - . . .
. . .
URL list - Approve sites - Banned domains Categories: - Add, edit, delete - Create new subcategory under Index: - Basic indexing options - Advanced options Clean: - Clean keywords not associated with any link - Clean links not associated with any site - Clean Category table not associated with any site - Clean Media links - Clear Temp table . . .
. . .
- Clear Search log - Clear 'Most Popular Page Links' log - Clear 'Most Popular Media Links' log - Clear Spider log, separate and bulk delete - Clear Thumbnail images, separate and bulk delete - Clear Text cache - Clear Media cache - Clear IDS log file - Clear flood attempts log file - Clear all entries in addurl or banned table - Truncate all . . .
. . .
flood attempts log file - Clear all entries in addurl or banned table - Truncate all tables in database Settings: - General Settings - Index Log Settings - Spider Settings - Search Settings - Order of Result listing - Suggest Options - Page Indexing Weights Database: - Configure up to 5 databases with unlimited number of table sets - Activate . . .
. . .
Database: - Configure up to 5 databases with unlimited number of table sets - Activate separately for 'Admin', 'Search' user and 'Suggest URL user' - Backup / Restore - Copy / Move - Optimize Templates: In order to enable customer's integration of Sphider-plus into existing sites, HTML templates are prepared for Search form Text result . . .
. . .
Search form Text result listing Media result listing Most popular queries etc. Three different designs are offered, which may be selected in submenu 'Settings'. If the layout does not fit the design of your site (which is normal), you may create new designs and modify the appropriate file /templates/My_template/adminstyle.css . . .
. . .
the appropriate file /templates/My_template/adminstyle.css /templates/My_template/userstyle.css Statistics output: - Top keywords (Top 50 with hit counter). - All indexed thumbnails w 3e80 ith ID3 and EXIF info. - Larges pages offering link URL and file size. - Most Popular Searches for text links offering: Link addr., total clicks, last . . .
. . .
Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Searches for media links offering: Link addr., total clicks, last clicked, last query (Top 50) - Most Popular Links (click counter). - Search log offering: Query, Results, Queried at, Time taken, User IP, Country, Host name (Latest100) - Index log offering: . . .
. . .
Country, Host name (Latest100) - Index log offering: File-name, index date and delete option - sitemap log offering: sitemap.xml output sitemap list offering file/page suffixes - IDS log offering: IP, host, query, impact, involved tags, date and time of intrusion. - Flood attempts log offering: IP, query, date and time of flood attempt. - Auto . . .
. . .
attempts log offering: IP, query, date and time of flood attempt. - Auto Re-index log file - Server info offering: Server software, environment, MySQL, PDF-converter, image functions, php.ini file PHP integration, PHP security info. Each item holding lists of details. All text links, media links and thumbnails are active linked. As stated in . . .
. . .
individual Re-indexing, the periodical Re-indexer could be started and aborted in the "Options" menu of each site. 2.5 Preferred Re-indexing Each new URL added to the Admin backend, could be supplied with a priority level. This level will be used by the option 'Re-index only preferred sites'. Level 1 will be interpreted as most important, while . . .
. . .
less all index results will be stored in log files in sub folder /admin/log/ The names of the log files look like: db2_100524-21.47.56_1.html (log file of first thread) db2_100524-21.48.12_2.html (log file of second thread) and is build by the following items: db2 - Number of database. 100524 - Date (May 24, 2010) 21.47.56 - Time when this thread . . .
. . .
spider.php -new1 php spider.php -new2 etc. The IDs will be added to the name of the corresponding log files like: db2_100524-21.47.56_ID1.html (log file of first thread) db2_100524-21.48.12_ID2.html (log file of second thread) IDs could be defined by personal requirements, but the limitations for file names with respect to the OS should be taken . . .
. . .
but will not erase the content of all the other tables. So the check whether the content of a page has changed (MD5sum) is still available for a fast re-index procedure. Once prepared, multithreaded re-indexing could be invoked by starting several threads and adding individual IDs to the option parameter like: php spider.php -erased1 php . . .
. . .
4.4 Ignoring parts of a page Sphider-plus includes an option to exclude parts of pages from being indexed. Thi 775e s can for example be used to prevent search result flooding when certain keywords appear on certain part in most pages (like a header, footer or a menu). Any part of a page between <! sphider_noindex > and <! . . .
. . .
<! sphider_noindex > and <! /sphider_noindex > tags is not indexed, however links in it are followed. 4.5 Ignoring parts of a page by <div id='abc'> Ignoring parts of a page by the <! sphider_noindex > tags requires direct access to the page, because the tags need to be added (edited) to the page. A more flexible method, . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ menu[0-5] / 4.6 Indexing only parts of a page by <div id='abc'> If enabled in Admin settings, the values as defined in the list-file /include/common/divs_use.txt will be used to index only the content between <div . . .
. . .
a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */ table[0-5] / 4.7 Ignore HTML elements defined by <tagname> . . </tagname> This option is foreseen to cooperate with the new HTML5 elements like section, nav, aside, hgroup, article, header, footer, etc. HTML elements . . .
. . .
contain a regexp pattern. The regexp needs to be introduced by */ and must be ended with another slash. Example: */nav[0-5]/ Please keep in mind that element names placed in /include/common/elements_not.txt will be processed case-sensitive. 4.8 Index only HTML elements defined by <tagname> . . </tagname> This is the vice versa . . .
. . .
of all the duplicate content URLs: http://www.example.com/product.php?item=swedish-fish&category=gummy-candy http://www.example.com/product.php?item=swedish-fish&trackingid=1234&sessionid=5678 and Sphider-plus will understand that the duplicates all refer to the canonical URL: http://www.example.com/product.php?item=swedish-fish. The . . .
. . .
charset are supported by the ConvertCharset function and will be used to convert text into UTF-8 Unicode: WINDOWS windows-1250 - Central Europe windows-1251 - Cyrillic windows-1252 - Latin I windows-1253 - Greek windows-1254 - Turkish windows-1255 - Hebrew windows-1256 - Arabic windows-1257 - Baltic windows-1258 - Viet Nam cp874 - Thai - this . . .
. . .
- Baltic windows-1258 - Viet Nam cp874 - Thai - this file is also for DOS DOS cp437 - Latin US cp737 - Greek cp775 - BaltRim cp850 - Latin1 cp852 - Latin2 cp855 - Cyrylic cp857 - Turkish cp860 - Portuguese cp861 - Iceland cp862 - Hebrew cp863 - Canada cp864 - Arabic cp865 - Nordic cp866 - Cyrylic Russian (this is the one, used in IE . . .
. . .
IE Cyrillic (DOS) ) cp869 - Greek2 MAC (Apple) x-mac-cyrillic x-mac-greek x-mac-icelandic x-mac-ce x-mac-roman ISO iso-8859-1 iso-8859-2 iso-8859-3 iso-8859-4 iso-8859-5 iso-8859-6 iso-8859-7 iso-8859-8 iso-8859-9 iso-8859-10 iso-8859-11 iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 . . .
. . .
iso-8859-12 iso-8859-13 iso-8859-14 iso-8859-15 iso-8859-16 MISCELLANEOUS gsm0338 (ETSI GSM 03.38) cp037 cp424 cp500 cp856 cp875 cp1006 cp1026 koi8-r (Cyrillic) koi8-u (Cyrillic Ukrainian) nextstep us-ascii us-ascii-quotes DSP implementation for NeXT stdenc symbol zdingbat And specially for old Polish programs: mazovia This list is to be read . . .
. . .
Access denied; you need the RELOAD privilege. . . Top 10. Enable real-time output of logging data Up to version 1.5 of Sphider-plus, during index / re-index there was no printout available because: - Several servers, especially on Win32, buffer the output from the script until it terminates before transmitting the results to the browser. - . . .
. . .
is seen. - Some versions of Microsoft Internet Explorer only start to display the page after they have received 256 bytes of output. As progress was not presented during index / re- index procedure, waiting for results became a pain in the neck. Selectable in Admin setting together with the update interval (1 - 10 seconds), AJAX technology was . . .
. . .
characters in front of words Warning: This option should be used with special care and not be activated for non ISO-8859 charsets. Some special characters as part of the word ending might be erased by accidental. Top 13. Media search for images, audio streams and videos 13.1 Media indexing Index of media files is enabled by separated Admin . . .
. . .
and 'Found at' - Total clicks - Last clicked - Query input 'Indexed Image Thumbnails' presenting: - Thumbnail 150 x 100 pixel - Image details like title, filename size of original image, link- and thumb-id - Option to delete the thumbnail In order to open the media files all tables contain active links. Media results are also stored in . . .
. . .
Settings' menu of the admin backend: First option: For results of XML product feeds, present the text extract of 350 characters as part of product attributes. If this option is not activated, all products of the XML product feed will be presented in search results, and the hits of the query string will be highlighted in all involved products. . . .
. . .
content of the database tables (those with the same table prefix) will be destroyed by the restore procedure. 16.5 Copy & Move This section of the Database Management will present only those databases, which are correctly configured, assigned and do have a set of installed tables as described in chapter Definition and configuration. This . . .
. . .
results. Never the less to find these x results, the complete database had to been browsed. Starting with version 2.5, an additional clean option is offered as part of the Admin backend. Main advantage of this option is a significant reduction of the search time for any query, because the content of the db could be limited to offer only x . . .
. . .
<link>http://www.abc.de/images/warp.gif</link> <title>warp.gif</title> <x_size>635</x_size> <y_size>98</y_size> </media_result> <media_result> <num>2</num> <type>audio</type> <url>http://www.abc.de/index.php</url> . . .
. . .
the following content: {"query":"colour","ip":"::1","host_name":"guard_007-hoster","query_time":"2016-03-18 10:56:23 AM","consumed":0.016,"total_results":2,"num_of_results":2,"from":1,"to":2,"text_results":[{"num":1,"weight":"100.0","url":"http:\/\/www.english.le-piaggie.info\/Info_eng.pdf","title":" Info_eng","fulltxt":" . . . . . .
. . .
placed on the ground floor, today's big kitchen with the heating fireplace for open and close mode of . . . ","page_size":"5,661.6kb","domain_name":"www.english.le-piaggie.info"}{num2weight500urlhttp\/\/wwwenglishle-piaggieinfo\/html\/descriptionhtmltitle"num":2,"weight":"50.0","url":"http:\/\/www.english.le-piaggie.info\/html\/description.html","title":" Description","fulltxt":" . . . cable. Additional active . . .
. . .
D.O.C.G. as far as to the Pratomagno mountains 160 olive-trees Detached house, about 800 m to reach the . . . ","page_size":"26.4 kb","domain_name":"www.english.le-piaggie.info"}]} Top . . .