nlpcrawl is a "suite of tools [that works] asynchronously in order to crawl the web for relevant content". nlpcrawl is developed at the University of Latvia in order to efficiently crawl web-pages and perform natural language operations on the downloaded content. The crawler software is licensed in full under the permissible BSD 3-part license. For more information on the underlying language research, please contact Dr Guntis Bārzdiņš, guntis at latnet dot lv, at the University of Latvia's Institute of Mathematics and Computer Science.

The intent of nlpcrawl is to provide a portable, robust, and simple set of tools to acquire mark-up data (i.e., not images, binary documents, or other "rich media") and keep it fresh. The system is heavily tuned for per-language crawling, or in other words, crawling pages in a particular language.

The nlpcrawl suite of tools are being developed by Kristaps Džonsons, kristaps dot dzonsons at latnet dot lv, at the University of Latvia's Institute of Mathematics and Computer Science.

Some features of the system follow:

• politeness policy: highly tunable politeness with sane defaults (10 seconds, 2 pages per burst)
• path ascendancy: tunable link path-ascendancy (Cothey, 2004)
• pipelining: if the underlying libcurl is up-to-date, the system will pipeline bursts of requests (6 pages) (Fielding, et al, 1999)
• language selection: language selection is enforced at several stages, from character sets to dictionary matching, with an emphasis on early enforcement
• standards compliance: the system makes constant use of a variety of standards, from URI construction to HTTP headers (Berners-Lee, et al, 2005; Fielding, et al, 1999; Berners-Lee, et all, 1996)
• multi-threaded operation in speed-critical areas (page fetching)

This project has been used to crawl fairly small volumes (several hundred thousand pages, ten to twenty gigabytes). The politeness policy is fairly new and will need testing before becoming stable.

Some outstanding features follow:

• spider-trap detection
• conformance to Robots Exclusion Standard (Koster, 1994)
• simple one-command controlling daemon
• in-line archiving of downloaded corpus

25-05-2007: fetcher is now multi-threaded for significantly greater performance, politeness policy fixed (was not polite), "parent" field culled from database, charset and langlen reduced to 32 bytes (from 64 bytes)

21-05-2007: optional path-ascendancy, politeness policy implemented, signal handling routines modified to be simpler (no execution from signal stack)

03-05-2007: bug-fixes, stability, and documentation updates. Small optimisations to the database. Statistic collection is active but a coherent tool for extraction is not yet available. The system now regularly collects over ten gigabytes of data during each [test] run.

28-04-2007: all basic elements in place and thoroughly tested. Stable release candidate tagged. Focus will now change to testing, then writing a controlling utility and a scan-window auto-optimisation tool.

23-04-2007: significant code has been re-written in order to scale upward properly. The nlpc-fetch(1), nlpc-scan(1), and nlpc-vrfy(1) are routinely run in parallel to collect gigabyte-range data. Development focus is on database stability (i.e., deadlocking). All utilities but these and nlpc-list(1) are on hold until the first three components stabilise.

29-03-2007: nlpc-scan(1) now fully scans databases and extracts addresses. This utility is heavily standards-compliant (RFC 2396, see manual). One may freely use a combination of nlpc-scan(1) and nlpc-fetch(1).

24-03-2007: nlpc-fetch(1) updated to recognise expiration; a new utility, nlpc-list(1), that lists database entities; and the nlpc-up(1) utility is at its first version

22-03-2007: nlpc-fetch(1) should work and nlpc-scan(1) may be used to seed a database

13-03-2007: pre-release peek at design documentation

Pre-release manuals have been released for comment (these manuals are installed in Unix manual format along with system executables):

• nlpcrawl(1): canonical documentation for database structure and environment.
• nlpc-fetch(1): fetch pages from accessible media.
• nlpc-up(1): prune, kill, or revive stale pages.
• nlpc-scan(1): extract address from fresh pages.
• nlpc-list(1): list entities in an nlpc database.
• nlpc-vrfy(1): examine pages for validity.

The Unix manuals are, and will continue to be, the canonical source for system documentation. In order to compile, you'll need Berkeley DB, libcurl, and libxml. Edit the Makefile to system-dependent values, then execute make and make install. nlpcrawl should compile and run in any Unix environment. Development platform is Debian GNU/Linux "etch" (amd64).

A general view of the state transitions and their correspondence to utility follows:

Figure: inter-utility diagram	Figure: inter-state diagram

A common execution strategy is to simply run all daemons and collect pages. These may be further processed with other tools. In the following example, daemons are started in order to collect language-specific pages. The database rests in /tmp/nlpcrawl/db; the database files, in /tmp/nlpcrawl/data; and the file cache in /tmp/nlpcrawl/cache. These are the defaults. In this example, the file /usr/share/dict/lang contains a utf-8 dictionary of words. The system restricts itself to http and https schemes; and utf-8, windows-1257, iso-8859-4, and iso-8859-13 content-type character sets. This combination of parameters is entirely context-dependent.

$ nlpc-up &
$ nlpc-vrfy --filter-charset "utf-8 windows-1257 iso-8859-4 iso-8859-13" 
  --filter-dict /usr/share/dict/lang &
$ nlpc-fetch --filter-lang lv --head-lang lv --filter-scheme "http https" 
  --filter-charset "utf-8 windows-1257 iso-8859-4 iso-8859-13"
  --head-charset "utf-8 windows-1257 iso-8859-4 iso-8859-13" &
$ nlpc-scan --filter-scheme "http https" http://the-seed-site &
					

• Cothey, V., "Web-crawling reliability". Journal of the American Society for Information Science and Technology, 55 (14), 2004.
• Berners-Lee, T., Fielding, R., and Frystyk, H., Hypertext Transfer Protocol: HTTP/1.0, RFC 1945, May 1996.
• Berners-Lee, T., Fielding, R., and Masinter, L., Uniform Resource Identifier (URI): Generic Syntax, STD 66, RFC 3986, January 2005
• Fielding, R., Gettys, J., Mogul, J., Frystyk, H., Masinter, L., Leach, P., and Berners-Lee, T., Hypertext Transfer Protocol -- HTTP/1.1, RFC 2616, June 1999.
• Koster, M., "A Standard for Robot Exclusion", June 1994.
• Pemberton, S., et al, "XHTML 1.0: The Extensible HyperText Markup Language (Second Edition)", August 2002.
• Raggett, D., "HTML 3.2 Reference Specification", January 1997.
• Raggett, D., Le Hors, A., and Jacobs, I., "HTML 4.0 Specification", April 1998 (revision).
• Raggett, D., Le Hors, A., and Jacobs, I., "HTML 4.01 Specification", December 1999.