Sphider-plus

statistics 2. Indexing 2.1 Various options 2.2 Allow other hosts in same domain 2.3 Word stemming 2.4 Periodical Re-indexinging 2.5 Preferred indexing 2.6 Multithreaded indexing 2.7 Create thumbnails during index procedure 2.8 Prevent indexing of known malware and pishing pages 2.9 Follow and create sitemap files 2.10 Use private sitemap instead . . .
. . .
result listing 7.1 Text result listing 7.2 Media result listing 8. PDF converter 9. Clean resources during index / Re-indexing. 10. Enable real-time output of logging data 11. Error messages and Debug mode 12. Delete secondary characters 13. Media search for images, audio streams and videos 13.1 Media indexing 13.2 Not supported media content 13.3 . . .
. . .
foreseen for Sphider-plus. Separated into different submenus like: Sites: - Add Site - Index only the new - Re-indexing all - Re-indexing only preferred URLs - Erase Re-indexing (available also for individual URLs) - Import/export URL list - Approve sites - Banned domains Categories: - Add, edit, delete - Create new subcategory under Index: - Basic . . .
. . .
tags, date and time of intrusion. - Flood attempts log offering: IP, query, date and time of flood attempt. - Auto Re-indexing log file - Server info offering: Server software, environment, MySQL, PDF-converter, image functions, php.ini file PHP integration, PHP security info. Each item holding lists of details. All text links, media links and . . .
. . .
that only the starting page is indexed, depth 1 indexes the starting page and all the pages linked from it etc. Re-indexing: By selecting this mod, indexing is forced even if the page already has been indexed. Re-indexing only detects changes of the pages to be Re-indexinged. Modifications in Admin settings are not recognized. Erase Re-indexing: By . . .
. . .
even if the page already has been indexed. Additionally this mod will Clear Sphider-plus database before the Re-indexing process. It will leave the following untouched: - Categories - Query log - Sites and all options: spider-depth, last indexed, can leave domain, title, description, - URL Must include, URL must Not include. If settings have . . .
. . .
whether the query 'walk', 'walks', walked' or 'walking' is entered (for English stemming). 2.4 Periodical Re-indexinging This mode offers automatically Re-indexinging of all sites, or site specific, started periodically at a defined time interval. In Admin backend the time interval is selectable to 3 hours, 12 hours, 1 day, 1 week or 1 month. . . .
. . .
interval is selectable to 3 hours, 12 hours, 1 day, 1 week or 1 month. Also the count of periodically performed Re-indexinging procedures could be defined in Admin backend. Once started, the Re-indexing procedures will silently work in the background without creating monitor output, but like in all other indexing modes, writing the index results . . .
. . .
a log file showing the status of the periodical indexer is created, presenting all dates and times when the Re-indexing procedures were started, as well as the count. This additional log file is available for the Admin in 'Statistics' menu called 'Auto Re-indexing log file'. The periodical Re-indexinger could be started and aborted in Admin . . .

added / modified: New item in Admin settings: Perform a segmentation of Chinese and Korean text during index / re-index procedure. Will divide phrases like 帽子和服装 into the base words 帽子 and 和 and 服装 , so that all will become searchable. Valid for Chinese sites with charset: GB2312, GBK and GB18030 Valid for Korean sites with charset: EUC-KR . . .
. . .
reloaded frames (e.g. by JavaScript). New item in Admin setting: Enable to decode BBCode during index / re-index into standard HTML If selected, code like [url=http://abc.de/][b]abc.de[/b][/url] will be converted to <;a href="http://abc.de">;<;strong abc.de<;/strong>;<;/a>; New item in Admin settings: Enable to . . .
. . .
will be converted to Čapek and Döhl New options in Admin settings: - Use whitelist in order to enable index / re-index only those pages that include any the words in whitelist - Use whitelist in order to enable index / re-index only those pages that include all the words in whitelist Improved 'Follow sitemap.xml' procedure: If . . .
. . .
Sitemap files are available, Sphider-plus will process the secondary Sitemaps and extract all links for index / re-index. Also gzip-compressed files (Index Sitemap files as well as the Sitemap files) will be processed. Improved index / re-index procedure: If charset of a site to be indexed is undetectable, because it is not HTML standard . . .
. . .
interrupted. Preferred charset as defined in Admin settings will be used for the involved link. Improved index / re-index procedure: If Sphider-plus is relocated by http 301 or 302, links found at the relocated site will also be followed. For new sites, as per default the spider-depth is now set to 'full'. Improved UTF-8 support: Conversion . . .
. . .
is now set to 'full'. Improved UTF-8 support: Conversion into UTF-8 charset now is obligatory. Improved index and re-index procedures for Cyrillic and Greek languages to support upper and lower case characters. Bug fixed that prevented to continue suspended index procedures. 'Continue suspended index procedure' enabled now also for 're-index' . . .
. . .
suspended index procedures. 'Continue suspended index procedure' enabled now also for 're-index' and 'Erase & re-index'. Improved search functions for search with wildcards and for strict search. Improved category search: - Selected category name is highlighted in headline of result listing. - If activated in Admin setting, categories which . . .

PHP functions are: fopen(); file_get_contents(); md5_file(); 3 new features for command line operation: - Erase & Re-index all sites ( -eall ) - Index all new URLs in database which had not jet been indexed ( -new ) - Re-index all meanwhile erased sites ( -erased ) New feature: In order to index XLS files, a converter for Exel files was . . .
. . .
Erase/clean site-specific data from MySQL database and thumbnails folder for a selected site. New Admin setting: Re-index all meanwhile erased sites. New Admin setting: Show complete list during import and export of URLs, or hide output. 24 language specific common files holding a list of words to be ignored during index (stop words). Added or . . .

November 2008 New item in Admin / Settings / General Settings: - Enable Debug mode. If selected, during index / re-index procedure the following information will be presented individual for each page: - New links found here - New keywords found here 1f41 For more details, please notice chapter Error messages and Debug mode New item in Admin / . . .
. . .
different languages, are to be indexed, this feature improves overview. Common words to be ignored during index / re-index procedure can be placed in individual files. The common word files should not be used, if 'phrase search' is the standard type of search. Sphider-plus will become problems to find complete phrases. Therefore, in Admin . . .
. . .
If the content of a page to be indexed / re- indexed contains one word of the blacklist, it will not be indexed / re-indexed. To be activated / deactivated in Admin settings For more details, please notice chapter Use of Blacklist New feature: Use a whitelist. The content of a page to be indexed / re- indexed must contain at minimum one word of . . .
. . .
content of a page to be indexed / re- indexed must contain at minimum one word of the whitelist to be indexed / re-indexed. To be activated / deactivated in Admin settings For more details, please notice chapter Use of Whitelist New feature: If available, show multiple hits of search result (per page) in result listing. To be defined (1 - 9) . . .
. . .
sitemap files. ../ is the root folder of the Sphider-plus installation. If the charset of a page to be indexed / re-indexed is not detectable, the home charset as defined in Admin settings is used. Improved search function for non-Latin symbols. Search function enabled for queries containing an apostrophe. Included query input protection . . .
. . .
queries containing an apostrophe. Included query input protection against Directory Traversals. Bug fixed in index/re-index procedure that prevented indexing of last word in full text that should be stored as new keyword. Improved storage of keywords in index/re-index procedure. Updated Romanian language file, thanks to CyBerNet. Some file types . . .
. . .
Romanian language file, thanks to CyBerNet. Some file types added to exclusion list in order not to be indexed / re-indexed. Thanks to clubmaster3. Improved Admin Log-in for Microsoft IIS. Thanks to bobyn. Involved files that have been modified / added for this release: /addurl.php /search.php /admin/admin.php /admin/admin_header.php . . .

links of a site. If <sitemapindex . . . > is detected, also multiple sitemap files are processed. Periodical Re-index Re-indexing could be performed automatically, repeated every selected time interval. Admin selectable intervals for 3 hours, 12 hours, 1 day, 1 week, or 1 month. Multithreaded indexing In order to reduce the time for . . .
. . .
to reduce the time for indexing, 1-10 parallel running threads might be activated in Admin settings. Preferred Re-index While invoking this option, the admin may select a suitable level for the next index procedure. Thus, only those URLs, containing the according level, will be Re-indexed.. Erase Re-index and Continue suspended index . . .
. . .
Hebrew. Links found in the converted files will be followed. Debug mode Offering detailed information during index/Re-index: New links, keywords, frames and media found per link. To be activated separately for Admin backend and User interface. Automatic detection of users preferred dialog language. Admin selectable for self-detection of the . . .

of duplicate media content. Improved image indexing by reducing the required download time. Improved index / re-index procedure to avoid 'MySQL server has gone away' messages. prototype.js framework adapted to cooperate with XHTML valid parameter handling. XHTML1.0 output for - Admin interface - Search form and Result listing - Suggest URL . . .

of each site will be stored in backup file and re-imported. New Admin setting: Clean query log during index / re-index and also for all erase functions. Will reset all 'Search' statistics in Admin backend, as well as the 'Most popular search' table at the bottom of result listing New feature in Admin backend: When opening the Admin . . .

with Sphider: v.1.3.5 In front of version 2.6 the following modifications have been added: New indexing feature: Re-indexinging could be performed periodically. Once started, this mode will automatically Re-indexing all sites periodically. The time interval is Admin selectable for 3 hours, 12 hours, 1 day, 1 week or 1 month. Also the count of . . .
. . .
New indexing feature: Re-indexinging could be performed periodically. Once started, this mode will automatically Re-indexing all sites periodically. The time interval is Admin selectable for 3 hours, 12 hours, 1 day, 1 week or 1 month. Also the count of periodically performed Re-indexinging procedures is Admin selectable. For details see chapter: . . .
. . .
count of periodically performed Re-indexinging procedures is Admin selectable. For details see chapter: Periodical Re-indexinging New feature for media search: Find media results not only by media 'tile', but also by EXIF and ID3 info To be activated in Admin backend. New option in Admin settings called: "Use string list of 'URL Must Not include' . . .
. . .
function: Searching for 'Sites', the result listing now will present also the 'Options' button to select Edit, Re-indexing, Erase & Re-indexing, Erase, Delete, Pages, Browse and Statistics Improved index procedure for media indexing: No longer accepting dead links. In order to become indexed, the media file must be present. Improved index procedure . . .
. . .
and 'FLUSH TABLE' will only be used, if the following Admin setting is activated: 'Clean resources during index / Re-indexing and also for search function' Improved 'Settings' interface in Admin backend. After pressing 'Save', now additionally presenting the eventually existing dependencies and the necessarily modified settings. Improved 'Approve . . .

the converter needs no adoption to the Operating System. New feature: Index only prioritized sites. Level depended re-index of only those URLs, containing the according level. For details please notice the chapter: Prioritized indexing New option: Admin's 'Sites' table sorted by index priority. New feature: Create a thumbnail of all Internet URLs . . .

caches for text and media results, configurable in Admin settings. - Automatic cleaning of caches during 'Erase & Re-inde' procedure. - If debug mode is enabled, activity/status of cache is presented in Result listing. For more details, please notice chapter Result cache for text and media queries Enlarged Admin statistics. In table 'Search . . .
. . .
New item in Admin Settings (Section: Index Log Settings): Suppress browser output of logging data during index / Re-inde. This item will speed up index / Re-inde procedure and prevent browser overflow on huge amount of sites to be indexed. If activated, this setting also disables the real-time output of logging data. New feature: Use the . . .
. . .
November 2008 New item in Admin / Settings / General Settings: - Enable Debug mode. If selected, during index / Re-inde procedure the following information will be presented individual for each page: - New links found here - New keywords found here For more details, please notice chapter Error messages and Debug mode New item in Admin / . . .
. . .
different languages, are to be indexed, this feature improves overview. Common words to be ignored during index / Re-inde procedure can be placed in individual files. The common word files should not be used, if 'phrase search' is the standard type of search. Sphider-plus will become problems to find complete phrases. Therefore, in Admin . . .
. . .
If the content of a page to be indexed / re- indexed contains one word of the blacklist, it will not be indexed / Re-indeed. To be activated / deactivated in Admin settings For more details, please notice chapter Use of Blacklist New feature: Use a whitelist. The content of a page to be indexed / re- indexed must contain at minimum one word of . . .
. . .
content of a page to be indexed / re- indexed must contain at minimum one word of the whitelist to be indexed / Re-indeed. To be activated / deactivated in Admin settings For more details, please notice chapter Use of Whitelist New feature: If available, show multiple hits of search result (per page) in result listing. To be defined (1 - 9) . . .
. . .
sitemap files. ../ is the root folder of the Sphider-plus installation. If the charset of a page to be indexed / Re-indeed is not detectable, the home charset as defined in Admin settings is used. Improved search function for non-Latin symbols. Search function enabled for queries containing an apostrophe. Included query input protection . . .
. . .
queries containing an apostrophe. Included query input protection against Directory Traversals. Bug fixed in index/Re-inde procedure that prevented indexing of last word in full text that should be stored as new keyword. Improved storage of keywords in index/Re-inde procedure. Updated Romanian language file, thanks to CyBerNet. Some file types . . .
. . .
Romanian language file, thanks to CyBerNet. Some file types added to exclusion list in order not to be indexed / Re-indeed. Thanks to clubmaster3. Improved Admin Log-in for Microsoft IIS. Thanks to bobyn. Involved files that have been modified / added for this release: /addurl.php /search.php /admin/admin.php /admin/admin_header.php . . .