ApacheCon EU 2014

Apache Nutch is an extensible and scalable web crawler based on Hadoop. This talk gives an overview of the crawler flow of work, its main components, job execution, the underlying data structures, and how it integrates with other Apache projects (Hadoop, Gora, Solr, Tika and HBase). The extensible plugin architecture is demonstrated by giving examples, how plugins help to adapt the crawler to specific use cases.
History, recent, and future developments of the Apache Nutch project are outlined, as well as the two branches under active development: the stable 1.x branch and the 2.x which is based on Apache Gora to abstract from storage back-ends.