2018-01-16

The Quicklisp local-projects mechanism

Quicklisp provides a lot of software, but there’s also a simple way to load things that Quicklisp doesn’t provide. That same mechanism can be used to override libraries Quicklisp does provide.

The local projects mechanism sets up a special directory that is automatically scanned for software to load. Here are a few quick examples.

Trying a library not in Quicklisp

First, imagine that you just heard about a great new library and want to try it. However, it’s not available through Quicklisp yet, only through a git repository on https://example.com/fun-project.git. One easy way to try it:
$ cd ~/quicklisp/local-projects
$ git clone https://example.com/fun-project.git
After the git command completes, and there is a fun-project subdirectory with a fun-project/fun-project.asd file present, the system is visible to ASDF and can be loaded either with ql:quickload or asdf:find-system. When loaded through ql:quickload, Quicklisp will automatically fetch and load any prerequisites automatically as well.

Overriding a library in Quicklisp

Second, imagine that you want to hack on a library that Quicklisp already provides. You don’t want to load and hack on the version from Quicklisp - that software is not under version control, and just represents a snapshot of the project at a particular point in time.

Once again, the procedure is to put the software in the ~/quicklisp/local-projects/ directory:
$ cd ~/quicklisp/local-projects/
$ git clone https://github.com/xach/vecto.git
After the git command completes, (ql:quickload "vecto") will load the library from local-projects rather than from the standard Quicklisp release.

How it works

The local-projects mechanism is relatively automatic. Here’s how it works underneath, and how to fix problems that might crop up.

ASDF has an extensible mechansim (the asdf:*system-definition-search-functions* variable) for searching for system files. Quicklisp extends this mechanism with a function that does the following, all in the context of the local-projectsdirectory.
  1. If there is no file named system-index.txt, it is created by scanning the directory tree for system files (matching "*.asd"). Each pathname is added to the file.
  2. If the system-index.txt file exists, but its timestamp is older than its containing directory, the directory is rescanned and the index recreated.
  3. The system-index.txt is searched for any entry with a pathname-name that matches the desired system name. If there’s a match, matching pathname is probed. If it still exists, it is returned. If it has disappeared, the system-index.txt is recreated as in step 1 and the search is retried.
  4. Otherwise the system search is deferred to the remaining ASDF system search functions.
When there are multiple system files with the same name in the directory tree, the one with the shortest full pathname name is returned. In the case of a pathname length tie, the one that is #'string< is returned.

Timestamp problems can sometimes crop up with step 2 above. For example, if you have a directory local-projects/my-project/ and you create local-projects/my-project/supporting-system.asd, the timestamp of local-projects/ is not updated and supporting-system.asd won’t be automatically added to the system index file.

There are a couple ways to force an update of the system index file. Within Lisp, you can use (ql:register-local-projects) to immediately regenerate system-index.txt. Outside of Lisp, you can use the touch command (or an equivalent) to update the timestamp of the local-projects directory, which will trigger a rebuild of the index on the next attempt to find systems..

Because of how the system index file is created (and recreated as needed), Quicklisp must have write access to the local-projects directory to make use of it.

Configuration

The local-projects mechanism is configured through a special variable ql:*local-project-directories*. By default, it includes only the local-projects subdirectory in the Quicklisp install directory, but you can add or remove directories at any time to have more places scanned for systems.
To disable the local-projects mechanism entirely, set ql:*local-project-directories* to NIL.

Build failures with ASDF 3.3.1

SBCL 1.4.3 ships with ASDF 3.3.1, and a number of Quicklisp projects have build problems as a result. Linedit, mgl, micmac, cl-string-match, and others are affected.

Here is a build failure report for yesterday. (You can ignore the gendl failures - it's a special case.) If anyone has ways to fix these projects, please do so as soon as you can - otherwise they will be removed from the January Quicklisp dist update in a few weeks.

2018-01-05

Download stats for December, 2017

Here are the raw Quicklisp download stats for December, 2017:

27247  alexandria
23604  closer-mop
21195  anaphora
20818  cl-ppcre
20690  split-sequence
20360  let-plus
20153  iterate
20032  babel
19888  trivial-features
18779  trivial-gray-streams
18077  bordeaux-threads
17215  cffi
16969  more-conditions
16966  trivial-garbage
16258  puri
16049  flexi-streams
15447  nibbles
14567  utilities.print-items
14366  usocket
13449  esrap
13366  chunga
13149  cl+ssl
12853  cl-base64
12701  chipz
12408  trivial-backtrace
12365  drakma
 9502  cl-fad
 9335  asdf-flv
 9270  cl-yacc
 8593  fiveam
 8050  parse-number
 7899  closure-common
 7893  cxml
 7878  log4cl
 7798  local-time
 7646  ironclad
 7621  architecture.hooks
 7347  named-readtables
 7343  parser.common-rules
 6783  plexippus-xpath
 6767  cl-json
 6708  lift
 6050  optima
 5425  lparallel
 5234  cl-clon
 5107  cxml-stp
 5031  xml.location
 4858  utilities.print-tree
 4855  cl-dot
 4430  cl-store
 4055  fare-quasiquote
 3963  fare-utils
 3816  inferior-shell
 3815  fare-mop
 3707  cl-unicode
 3432  cl-interpol
 3321  slime
 2919  trivial-utf-8
 2848  cl-utilities
 2830  metabang-bind
 2744  quri
 2628  uuid
 2415  trivial-types
 2375  cl-annot
 2372  cl-syntax
 2299  cl-slice
 2255  md5
 2247  trivial-indent
 2234  array-utils
 2229  plump
 2227  documentation-utils
 2226  static-vectors
 2219  gettext
 2107  symbol-munger
 2101  arnesi
 2092  collectors
 2087  access
 2086  fast-io
 2065  djula
 2056  cl-locale
 2051  cl-parser-combinators
 2014  hunchentoot
 1910  simple-date-time
 1844  ieee-floats
 1625  yason
 1352  rfc2388
 1293  monkeylib-binary-data
 1171  osicat
 1163  salza2
 1153  utilities.binary-dump
 1135  postmodern
 1044  trivial-shell
 1015  prove
  980  diff
  949  cl-who
  942  asdf-system-connections
  936  command-line-arguments
  933  cl-containers
  931  cl-custom-hash-table
  925  metatilities-base