Ex-crawler is divided in three separate programs, the main one is of course the Ex-Crawler server. The other ones are the graphical Distributed crawling client and the (Web-) search engine using the crawled informations.
- Multiplatform, flexible and scalable (Web-) crawler written in Java.
- Supported protocols: http, https, ftp - Supported file formats: html, pdf, doc etc.
- Comes with it's own socket server, where distributed computing clients connect and where you can maintain the server (via telnet or client)
- As for instance adding new URLs, editing configurations etc. including User Access Levels and User Accounts - perfectly integrated with the client and the search engine frontend
- For a full list of the server commands and features take a look here
- The informations are stored in an MySQL or PostgreSQL or MSSQL Database, even SQLLITE is supported
- Plugin support - The server comes with a very flexible and easy to use plugin System, with it you can integrate new features in minutes, without changing the server core
- We tried to make plugin development so easy, that you can learn developing a plugin in less then five minutes. You just copy your finished plugin to the matching folder!
- Plugins can be run on server start, every crawled website or image, on time scheduler (every minute, hourly, daily, even weekly)
- Statistics, Scheduler, Stats for users, and much more
Distributed crawling client (not released yet, use SVN)
- Attractive, modern, multiplatform graphical distributed crawling client written in Java (Swing)
- Uses the idle time of the clients computer, when the user works no bandwith / cpu resources are used
- Multiple server support, easy server switching, adding of new servers, proxy support
- Autoupdate, Personal settings - easy adaption on own projects
- Management of the server, user login and much more
- User stats and ranks
- and much more