This chapter is an adaptation for ACL of material which I contributed a few years back to the Common Lisp Cookbook:

That piece drew its examples from the LispWorks MP implementation of the time (version 4.2) and LW users might prefer to refer to that as well. (But note that it doesn’t mention some important features such as concurrent MP which we’ll come to later on.)

The first question to consider is: why bother with MP? Sometimes there’s no need to bother: your application is so straightforward that you need not concern yourself with processes at all. But in many other cases it’s difficult to imagine how a sophisticated application can be written without MP. How will you get it to respond to events in real time? For example:

It’s fairly likely when you’re working with processes that—every now and again—one of them will run away from you and either fall motionless at your feet or make a CPU-intensive break for freedom. If you’ve managed to get your Lisp tangled you may be short on options for halting a runaway process, short of terminating the Lisp session altogether. So before you go any further, you should learn how to kill processes in your Lisp development environment. This is easiest if you’re working in an IDE.

(mp:process-run-function "foo" (lambda () (loop)))

If you aren’t in an IDE, let’s hope you can find (or fire up) a listener which isn’t totally hosed and into which you can type your implementation’s equivalent of a process-kill command (see the section called “Basics” below).

Typically, you don’t have to do anything to obtain, load or start up your Lisp’s MP system: if you’re running the Lisp then you’re running MP with it. This isn’t always the case (e.g. LW on Unix workstations). If you’re at all uncertain, read the manual for your implementation and use whatever facility has been provided for starting MP in your image.

ACL exports its MP utilities from the MULTIPROCESSING package, nicknamed MP. We’ll assume here that you’re cl:useing this:

(use-package :mp)

The general idea is simple enough: pass a function to process-run-function and it’ll run in its own process. (This may or may not be represented by a real thread at the OS level.)

In the following example, you create a process called "Incrementing *foo*". The function foo is invoked (with no arguments) in that process.

(defvar *foo* 0)
(defun foo () (incf *foo*))

(process-run-function "Incrementing *foo*" 'foo)
 =>
#<PROCESS Incrementing *foo*(3) @ #x40689562>

*foo*
 =>
1

;; By the time you've typed this, the new process has almost certainly
;; terminated. Note that its printed representation has (subtly) changed.

**
 =>
#<PROCESS Incrementing *foo* @ #x40689562>

To kill a process programmatically (typically: from another process), call process-kill with that process as its sole argument. ACL causes the process to unwind (by cl:throwing out of its execution stack) and then removes it from *all-processes*.

CL-USER(1): (process-kill
             (process-run-function "Don't forget to say goodbye"
                                   (lambda ()
                                     (unwind-protect
                                         (loop)
                                       (print "Goodbye"
                                              #.*standard-output*)))))
NIL
CL-USER(2):
"Goodbye"

If you didn’t already have a handle on the process, use process-name-to-process. This function has signature

(process-name-to-process name &key abbrev (error t))

If abbrev is set, then the string name only has to match the beginning of a process name for that process to be returned. If error is true then failure to match the name to any process will signal an error.

Note the difference between:

(let ((table (make-hash-table)))
  (loop for i from 1 to 20 do
        (process-run-function "One closure"
                              (lambda () (setf (gethash i table) t))))
  table)

and:

(let ((table (make-hash-table)))
  (loop for i from 1 to 20 do
        (process-run-function "Twenty different bindings"
                              (lambda (j) (setf (gethash j table) t))
                              i))
  table)

In the first case, all twenty processes share the same closure variable i. The processes execute asynchronously: there’s no way to tell exactly when or in what order they’ll execute, or how they’ll interleave with the process which created them. In this case (try it!) you will observe that all 20 processes are typically initialized before any of them have a chance to start running. By far the easiest way of dealing with this is to ensure that variables which need to be private to each process are bound on a per-process basis.

Every process in your Lisp system has its own private state. The following aspects of state will vary between different processes:

On the other hand, the following are globally set rather than bound in a Lisp system and so will not vary between processes:

To see how per-process variable bindings can lead you astray, consider the following examples:

CL-USER(1): (defvar *foo* nil)

*FOO*
CL-USER(2): (process-run-function "Global set *foo*"
                                  (lambda () (setf *foo* 1)))

#<PROCESS Global set *foo*(7) @ #x20eb96d2>
CL-USER(3): *foo*
1
CL-USER(4): (process-run-function
             "Bind in new process, invisible elsewhere."
             (lambda ()
               (let ((*foo* 2))
                 (sleep 5)              ; affects just this process
                 (setf *foo* 3)
                 (print *foo* #.*standard-output*))))
#<PROCESS Bind in new process, invisible elsewhere.(7) @ #x20ec651a>
CL-USER(5): *foo*                       ; during the 5 second sleep
1
CL-USER(6):
3

CL-USER(7): *foo*
1
CL-USER(8): (let ((*foo* 4))
              (process-run-function
               "Bind in old process, can't see elsewhere."
               (lambda () (print *foo* #.*standard-output*))))
#<PROCESS Bind in old process, can't see elsewhere.(7) @ #x20ed1802>
CL-USER(9):
1

In both lines 1 and 2, *foo* is set globally. This means that every process shares the symbol’s value. In line 4, *foo* is bound only within the new process; the original process (i.e. the listener) does not see either this binding or the result of a setf within the binding. Similarly, in line 8 the binding is present only in the original process.

To create a binding which will be present at the beginning of a new process, specify an :initial-bindings value:

CL-USER(10): (process-run-function '(:name "Bind around new process."

                                     ;; Note the "dotted list" format - an
                                     ;; unpleasant trap for the unwary.

                                     :initial-bindings ((*foo* . 5))
                                     )
                                   (lambda ()
                                     (print *foo* #.*standard-output*)))
#<PROCESS Bind around new process.(7) @ #x20fc5e52>
CL-USER(11):
5

CL-USER(12):

An obvious way to test whether processes are behaving as you imagine they ought is to get them to print messages to the listener. For example, you might feel justified in trying something like:

CL-USER(12): (process-run-function "Where's my output?"
                                   (lambda () (print 99)))
#<PROCESS Where's my output?(7) @ #x20e7027a>
CL-USER(13):

Where indeed is your output? The answer is that your new process has a different *standard-output* to the listener, and that’s where your output has gone to. Here is how you might find out where precisely that is:

CL-USER(13): (process-run-function "Where indeed?"
                                   (lambda ()
                                     (print *standard-output*
                                            #.*standard-output*)))
#<PROCESS Where indeed?(7) @ #x20e820ea>
CL-USER(14):
#<TERMINAL-SIMPLE-STREAM [initial terminal io] fd 0/1 @ #x201184aa>

(process-run-function "Why not here?"
                      'print
                      *standard-output*
                      #.*standard-output*)

An entirely more annoying situation occurs when two or more processes share the same *standard-input*. This is can happen if you’re running Lisp from a Unix command line. Consider this interaction (ACL on FreeBSD):

CL-USER(1): *current-process*
#<PROCESS Initial Lisp Listener(2) @ #x4052e332>
CL-USER(2): (process-run-function "foo" 'error "bar")
#<PROCESS foo(3) @ #x405f757a>
CL-USER(3):
Error: bar

Restart actions (select using :continue):
 0: Abort entirely from this (lisp) process.
[Current process: foo]
[1] CL-USER(1): *current-process*
#<PROCESS Initial Lisp Listener(2) @ #x4052e332>
CL-USER(4): *current-process*
#<PROCESS foo(3) @ #x405f757a>
[Current process: foo]
[1] CL-USER(2):

Two processes (the original Lisp listener and your own "foo") have the same *standard-input*. The Lisp scheduler appears to be switching between them each time it generates a prompt. (Things could be worse. Another Lisp in which I tried that out reliably switched contexts part-way through reading lines of input from the prompt, which made bailing out somewhat hit and miss.)

The best approach to this particular hazard is to avoid it altogether. If you’re compelled to debug a multi-processing application at the command line, consider using SLIME (Chapter 18, SLIME).

In all the above examples, a process is created to run a simple function and then halt. In a typical application at least some of your processes will run an event loop of some sort. An event loop is a function which repeatedly waits for an event external to that process to occur. When an event is noticed, it is dispatched (maybe to another process) for processing and the event loop cycles back to its waiting state. The “other process” here might already exist (perhaps running an event loop of its own) or might be created specifically to perform this task (i.e. handle a single event) and then terminate.

It might be tempting to construct an event loop using cl:sleep. For example:

(defun bogus-event-loop ()
  (loop
   (sleep 1)                            ; THIS IS WRONG
   (when (something-has-happened)
     (act-on-that-thing))))

(process-run-function "Bogus event loop"
                      'bogus-event-loop)

This is a poor choice, because you’re condemned to wait for the sleep to return before you can perform the wake-up test. (Also, it’s conceivable that your implementation cannot sleep one single process without sleeping the whole Lisp executable. In such cases, any processing required before the predicate something-has-happened can return true will never happen.)

Consider instead:

(defun improved-event-loop ()
  (loop
   (process-wait "Waiting for something to happen"
                 'something-has-happened)
   (act-on-that-thing)))

The arguments to process-wait are a string (which you should use for describing what this process is waiting for), a function and optionally arguments to that function. The call to process-wait will not return until this wait-function returns true.

There’s often a risk that the wait-function will never return true, meaning that the waiting process might wait forever. If you don’t want that to happen, consider instead using process-wait-with-timeout. This function has signature

(process-wait-with-timeout reason timeout wait-function &rest arguments)

where timeout can be any non-negative real number. If the wait-function succeeds before that number of seconds has passed, it returns true; otherwise it stops waiting and returns nil. For example:

(defun check-wait-for-file (file timeout)
  (or (probe-file file)
      (process-wait-with-timeout (format nil "Waiting for \"~a\"" file)
                                 timeout
                                 'probe-file file)
      (error "File \"~a\" still not found" file)))

Related to process-wait-with-timeout is the macro with-timeout:

(with-timeout (seconds &rest timeout-forms) &body body)

The body is run as an implicit progn. If it takes longer than seconds to do this, execution throws out and the timeout-forms are run, returning the result of the last form.

An important mode of communication between processes is the interrupt: a message sent by one process to another, to ask that process to perform some task. This might come in handy if:

The safest approach is to use a message queue. One process “owns” the queue and other processes can “post” tasks to it; the owning process deals with them one at a time when it’s good and ready to do so.

ACL supports this paradigm with the class queue (a first in, first out data structure), atomic operators enqueue and dequeue, and the predicate queue-empty-p. The accessor process-property-list is also useful here.

;; Call (event-handler) to set up a handling process which processes events
;; from its queue, always completing one before going on to the next. Use
;; handle-in-process below to add events to the queue.

(defun event-handler ()
  (process-run-function "Event handler"
                        'handle-events))

(defun handle-events ()
  (let ((queue (make-instance 'queue)))

    ;; Put the queue where client processes can find it

    (setf (getf (process-property-list *current-process*) 'queue)
          queue)

    (loop ;; Wait for a request to arrive (when one does the queue
          ;; will stop being empty)

          (process-wait "Waiting for request in queue"
                        (lambda () (not (queue-empty-p queue))))

          ;; Now we know a request has arrived, peel it off the queue
          ;; and run it

          (let ((request (dequeue queue)))
            (funcall request)))))

(defun handle-in-process (process function &rest arguments)
  (let* ((queue (getf (process-property-list process) 'queue))
         (no-result (cons nil nil))
         (result no-result))

    ;; Send function to other process

    (enqueue queue
             (lambda ()
               (setf result (apply function arguments))))

    ;; Wait for the value of RESULT to change, and then return it

    (process-wait (format nil "Waiting for response from ~a"
                          (process-name process))
                  (lambda () (not (eq result no-result))))
    result))

When you’re working on a data structure you sometimes need to be certain that whatever you’re doing to it will be complete before any other process touches that data. Such an operation is referred to as a critical section, or as being atomic.

You cannot assume that any operator is atomic unless your implementation’s documentation has explicitly told you that this is the case. Some implementations support atomic versions of common operators, for example atomic-push; when they do that’s a firm warning that the CL counterparts (in this case cl:push) are not thread-safe.

Traditionally Lisp implements MP by emuluating multi-tasking. When it comes down to it, only one task is being performed at a time. Forthcoming releases of both ACL and LW will support symmetric multiprocessing: concurrent threads running in parallel on multiple cores. This will allow for considerable performance gains but will have implications for the enforcement of atomicity. Both implementations will introduce new operators for controlling concurrency as well as a number of primitives for atomic read-modify-write operations.

Some other implementations already provide symmetric MP, albeit with less sophisticated controls. In all cases, you are strongly advised to read your documentation with regard to concurrency. For ACL users an upgrade note is available here:

Sometimes it’s important to control access to a resource, so that only one process is operating on it at a time. You could do this is by restricting access from your code to that resource and only allowing one specialized process to touch it, using an event handler to enforce this. However there are two potential drawbacks to this approach:

A lock is a simple Lisp object, which can be held by no more than one process at a time. A process attempting to hold a lock which is already in use will hang (i.e. be forced to wait) until the lock is freed. In the following example, the locking mechanism is reduced to its most minimal form.

;; Create a lock and remember it.

(defvar *lock* (make-process-lock))

(defun use-resource-when-free (stream)

  ;; Callers must wait at this point for the lock to be freed,
  ;; before they can proceed with the body of this form.

  (with-process-lock (*lock*)
    (use-resource-anyway stream)

    ;; When we exit this form, the lock is freed automatically
    ;; and other processes will be allowed to claim it.
    ))

(defun use-resource-anyway (stream)
  (let ((name (process-name *current-process*)))
    (format stream "~&Starting ~a." name)
    (sleep 1)
    (format stream "~&Ending ~a." name)))

(defun test (lock-p)
  (let ((run-function (if lock-p
                          'use-resource-when-free
                        'use-resource-anyway)))
    (dotimes (id 3)
      (process-run-function (format nil "process ~a" id)
                            run-function *standard-output*))))

Recall that AllegroCache connections are not thread-safe: you should not allow two processes to “use the same connection” at the same time. We’re now in a position to consider potential solutions to this problem.

Another option is to maintain a pool of several connections from which processes can request a connection and to which they can be guaranteed to return it:

(defun process-run-with-connection (name-or-options function &rest args)
  (process-run-function name-or-options
                        (lambda ()

                          ;; Per-process binding of *allegrocache* keeps
                          ;; the connection private

                          (let* ((*allegrocache* (request-connection)))
                            (unwind-protect
                                (apply function args)
                              (return-connection *allegrocache*))))))

;; Use of a QUEUE ensures atomicity (we don't want two processes picking
;; up the same connection)

(defvar *connections* (make-instance 'queue))

(defun request-connection ()
  (or (dequeue *connections*)
      (open-network-database ...)))

(defun return-connection (connection)
  (enqueue *connections* connection))

We should end this chapter with a word of caution. MP errors can be very hard to debug. Your only symptom may be “unexplainable” data corruption or “illogical” program flow. You may find the problem hard or impossible to reproduce, even though you have incontrovertible proof of it happening once. Any of this should point you to examine your MP usage. Pay attention to anecdotal evidence. I have in mind a race condition whose outcome flipped after one of my users defragmented his hard drive thus speeding up processes which accessed the disk. Ask yourself: “clearly this is impossible, but how could this have happened?”, “can I document the order on which I depend upon things happening?”, “is anything else unexpected happening?”