Aperture - Crawler

Aperture is a open source java framework  for extracting and querying full-text content and metadata from various information systems (file systems, web sites, mail boxes, ...). It works good. It provides incremental crawling too. Its a awesome product i ever seen. Just take a look at this guys...:)

You can find more information from the below link.

http://aperture.sourceforge.net/