prolog-mpi

Abstract

prolog-mpi is an application suite for allowing Prolog (specifically SWI-Prolog) applications to take advantage of MPI environments, like grids, clusters, or multiprocessors. It must be used as a standalone application: instead of launching the Prolog executable, one launches pl-mpi. The executing context is then free to use the exported predicates for distributing data.

The system is currently under development at the University of Latvia's IMCS by Kristaps Dzonsons, kristaps dot dzonsons at latnet dot lv.

In brief, pl-mpi(1) operates by starting a number of MPI processes linked with the SWI-Prolog library. The primary process handles user input (as if one were to start an interactive Prolog session). The Prolog system on this process has extensions for handling the custom predicates. Requests to these predicates are generally fielded by another MPI process (to allow asynchronous operation, i.e., so that control returns quickly to the Prolog interactive session). This second process arbitrates jobs between the Prolog session and slave processes. Slave processes actually execute the predicates.

While the system is good for simple jobs, complex preëxisting Prolog scripts may require re-writing to operate correctly. Distributed predicate inputs must have no dependencies between calls and may not be complex types. At this time, only float, integer and string inputs (and outputs) are supported. pl-mpi(1) has interfaces for both synchronous and asynchronous requests.

The prolog-mpi suite is licensed in full under the BSD license. The LICENSE file contains details.

News

24-01-2007: version 0.2.1 released: automatic testing of the asynchronous facilities as a text fixture. System is also ported to GNU/Linux (AMD64). See the Documentation for supported systems. Various other small fixes throughout. Testing system significantly cleaned up. Note that the Benchmarks are significantly more reasonable under GNU/Linux and OpenMPI (ranging under 0,1 s.) (old plots still shown).

Goals

The primary goal of prolog-mpi is to create a quality, stable system for allowing Prolog scripts to take advantage of multicomputer environments via MPI. The following features must be available:

usability: the system must be usable and functional, that is, have features allowing it to scale upward and downward with a minimum of required intervention and a maximum of available configuration
documentation: the system must be fully documented in a variety of available formats
reliability: the system must have a test harness routinely testing the system's various predicates (with all inputs/outputs, etc.) for correctness
benchmarking: data should be made available as to the maximum and minimum performance of the distributed predicates

Later versions of prolog-mpi will probably also support the following:

after asynchronous distribution, waiting on lists of pending jobs (instead of just one)
state-machine-based collector to account for efficient handling of dense and heavily-bursty return patterns (consider the -p fast option to pl-mpi-test(1) with a large input size).

Status

The system suite works as documented. At this time, the interface is considered complete enough for general use, and new features will be slow in coming.

Full documentation is available. The system has a rigorous testing mechanism (pl-mpi-test(1)). Several benchmark utilities are available. An asynchronous scheduling systems is in place and the appropriate predicates exported and fully documented. The asynchronous component is also tested in pl-mpi-test(1).

A system for revoking active jobs is also underway. In other words, jobs active on a slave node should have a way of being cancelled during operation. This will likely be accomplished by each slave node forking before initialisation. The parent would be responsible for MPI communication and the child for prolog. If a child is busy and need be "cancelled", the parent can simply kill and respawn. Communication would likely occur through a series of pipes.

Old News

16-01-2007: version 0.2.0 released: initial export of asynchronous predicates. These allow for multiple concurrent jobs. Although this functionality has been internally available, this is the first version that exports the appropriate Prolog predicates to the user interface. The user interface front-end is still experimental, though the standard synchronous predicates have not changed. Also, invalid input types (with mpidist/5 and such) will no longer cause an abort trap; rather, a notice is posted to standard error and the relevant predicate returns false immediately.

12-01-2007: version 0.1.6 released: profiling utilities fixed (incorrect micro-millisecond conversion). Benchmarks updated. Tracing also added (un-documented) but lacks processing utilities (for now). Asynchronous primitives re-added but un-documented: need testing. This is primarily a bug-fix for the millisecond conversions.

09-01-2007: version 0.1.5 released: significant updates to functionality, mostly related to profiling. Several new utilities added to profile run-time systems. Benchmarks added to site.

22-12-2006: version 0.1.4 released: documentation and manual page update.

21-12-2006: version 0.1.3 documentation has been updated. Memory leak issues fixed. Predicate-calling subroutines now install a foreign frame in Prolog, so the data is discarded faster.

15-12-2006: version 0.1.3 is tagged in CVS. The documentation now reflects this version. Note well: after a significant number of test runs (over 500), the system eventually crashes due to lack of memory. Reason currently unknown.

Download

We encourage using only the most current version (documentation links reflect current version).