2018-01-16

The Quicklisp local-projects mechanism

Quicklisp provides a lot of software, but there’s also a simple way to load things that Quicklisp doesn’t provide. That same mechanism can be used to override libraries Quicklisp does provide.

The local projects mechanism sets up a special directory that is automatically scanned for software to load. Here are a few quick examples.

Trying a library not in Quicklisp

First, imagine that you just heard about a great new library and want to try it. However, it’s not available through Quicklisp yet, only through a git repository on https://example.com/fun-project.git. One easy way to try it:
$ cd ~/quicklisp/local-projects
$ git clone https://example.com/fun-project.git
After the git command completes, and there is a fun-project subdirectory with a fun-project/fun-project.asd file present, the system is visible to ASDF and can be loaded either with ql:quickload or asdf:find-system. When loaded through ql:quickload, Quicklisp will automatically fetch and load any prerequisites automatically as well.

Overriding a library in Quicklisp

Second, imagine that you want to hack on a library that Quicklisp already provides. You don’t want to load and hack on the version from Quicklisp - that software is not under version control, and just represents a snapshot of the project at a particular point in time.

Once again, the procedure is to put the software in the ~/quicklisp/local-projects/ directory:
$ cd ~/quicklisp/local-projects/
$ git clone https://github.com/xach/vecto.git
After the git command completes, (ql:quickload "vecto") will load the library from local-projects rather than from the standard Quicklisp release.

How it works

The local-projects mechanism is relatively automatic. Here’s how it works underneath, and how to fix problems that might crop up.

ASDF has an extensible mechansim (the asdf:*system-definition-search-functions* variable) for searching for system files. Quicklisp extends this mechanism with a function that does the following, all in the context of the local-projectsdirectory.
  1. If there is no file named system-index.txt, it is created by scanning the directory tree for system files (matching "*.asd"). Each pathname is added to the file.
  2. If the system-index.txt file exists, but its timestamp is older than its containing directory, the directory is rescanned and the index recreated.
  3. The system-index.txt is searched for any entry with a pathname-name that matches the desired system name. If there’s a match, matching pathname is probed. If it still exists, it is returned. If it has disappeared, the system-index.txt is recreated as in step 1 and the search is retried.
  4. Otherwise the system search is deferred to the remaining ASDF system search functions.
When there are multiple system files with the same name in the directory tree, the one with the shortest full pathname name is returned. In the case of a pathname length tie, the one that is #'string< is returned.

Timestamp problems can sometimes crop up with step 2 above. For example, if you have a directory local-projects/my-project/ and you create local-projects/my-project/supporting-system.asd, the timestamp of local-projects/ is not updated and supporting-system.asd won’t be automatically added to the system index file.

There are a couple ways to force an update of the system index file. Within Lisp, you can use (ql:register-local-projects) to immediately regenerate system-index.txt. Outside of Lisp, you can use the touch command (or an equivalent) to update the timestamp of the local-projects directory, which will trigger a rebuild of the index on the next attempt to find systems..

Because of how the system index file is created (and recreated as needed), Quicklisp must have write access to the local-projects directory to make use of it.

Configuration

The local-projects mechanism is configured through a special variable ql:*local-project-directories*. By default, it includes only the local-projects subdirectory in the Quicklisp install directory, but you can add or remove directories at any time to have more places scanned for systems.
To disable the local-projects mechanism entirely, set ql:*local-project-directories* to NIL.

Build failures with ASDF 3.3.1

SBCL 1.4.3 ships with ASDF 3.3.1, and a number of Quicklisp projects have build problems as a result. Linedit, mgl, micmac, cl-string-match, and others are affected.

Here is a build failure report for yesterday. (You can ignore the gendl failures - it's a special case.) If anyone has ways to fix these projects, please do so as soon as you can - otherwise they will be removed from the January Quicklisp dist update in a few weeks.

2018-01-05

Download stats for December, 2017

Here are the raw Quicklisp download stats for December, 2017:

27247  alexandria
23604  closer-mop
21195  anaphora
20818  cl-ppcre
20690  split-sequence
20360  let-plus
20153  iterate
20032  babel
19888  trivial-features
18779  trivial-gray-streams
18077  bordeaux-threads
17215  cffi
16969  more-conditions
16966  trivial-garbage
16258  puri
16049  flexi-streams
15447  nibbles
14567  utilities.print-items
14366  usocket
13449  esrap
13366  chunga
13149  cl+ssl
12853  cl-base64
12701  chipz
12408  trivial-backtrace
12365  drakma
 9502  cl-fad
 9335  asdf-flv
 9270  cl-yacc
 8593  fiveam
 8050  parse-number
 7899  closure-common
 7893  cxml
 7878  log4cl
 7798  local-time
 7646  ironclad
 7621  architecture.hooks
 7347  named-readtables
 7343  parser.common-rules
 6783  plexippus-xpath
 6767  cl-json
 6708  lift
 6050  optima
 5425  lparallel
 5234  cl-clon
 5107  cxml-stp
 5031  xml.location
 4858  utilities.print-tree
 4855  cl-dot
 4430  cl-store
 4055  fare-quasiquote
 3963  fare-utils
 3816  inferior-shell
 3815  fare-mop
 3707  cl-unicode
 3432  cl-interpol
 3321  slime
 2919  trivial-utf-8
 2848  cl-utilities
 2830  metabang-bind
 2744  quri
 2628  uuid
 2415  trivial-types
 2375  cl-annot
 2372  cl-syntax
 2299  cl-slice
 2255  md5
 2247  trivial-indent
 2234  array-utils
 2229  plump
 2227  documentation-utils
 2226  static-vectors
 2219  gettext
 2107  symbol-munger
 2101  arnesi
 2092  collectors
 2087  access
 2086  fast-io
 2065  djula
 2056  cl-locale
 2051  cl-parser-combinators
 2014  hunchentoot
 1910  simple-date-time
 1844  ieee-floats
 1625  yason
 1352  rfc2388
 1293  monkeylib-binary-data
 1171  osicat
 1163  salza2
 1153  utilities.binary-dump
 1135  postmodern
 1044  trivial-shell
 1015  prove
  980  diff
  949  cl-who
  942  asdf-system-connections
  936  command-line-arguments
  933  cl-containers
  931  cl-custom-hash-table
  925  metatilities-base

2017-12-29

December 2017 Quicklisp dist update now available

New projects:
  • cl-ascii-table — Common Lisp library to present tabular data in ascii-art table. — MIT
  • cl-editdistance — A Common Lisp implementation of edit distance. — CC-BY-4.0
  • cl-ntriples — CL-NTRIPLES is a simple basic parser for Ntriples data. — BSD
  • cl-proj — CL-PROJ provides Proj.4 library bindings — BSD
  • de.setf.wilbur — a fork of net.sourceforge.wilbur updated for mcl and sbcl — LLGPL
  • vectors — Some utilities for using vectors — LLGPL
Updated projectsa-cl-loggerableahungry-fleecearchitecture.builder-protocolarchitecture.service-provideraws-sign4babelbeastcaclecarrierceplchancerychanlchronicitycl+sslcl-conllucl-cudacl-digraphcl-diskspacecl-disquecl-emojicl-enumerationcl-fadcl-fondcl-gamepadcl-gobject-introspectioncl-gpiocl-graphcl-iconvcl-interpolcl-liballegrocl-messagepack-rpccl-neovimcl-ntp-clientcl-pcgcl-portaudiocl-portmanteaucl-ppcrecl-rabbitcl-railcl-random-forestcl-spidevcl-strcl-string-matchcl-unificationcl-webkitclazycloser-mopclxcommonqtconfiguration.optionscroatoancurry-compose-reader-macrosdecltdelta-debugesrapf2clfare-csvfare-scriptsfemlispfiascoflexi-streamsfxmlgamebox-frame-managergamebox-gridsgamebox-mathglkitgraphhalftoneharmonyhornerhtml-templatehunchentootlegitlichat-protocollichat-serverliblichat-tcp-serverlisp-criticlocal-timemaidenmcclimmore-conditionsnew-opookoscoverlordpapyrusparser.common-rulesparser.inipostmodernprojecturedqbase64qt-libsqtoolsquickprojectsb-cgascalplserapeumsimple-tasksstaplestumpwmtap-unit-testthe-cost-of-nothingtinaatrivial-bit-streamstrivial-package-managerubiquitousuiopurl-rewriteutilities.binary-dumputilities.print-itemsutilities.print-treevectoworkout-timerxhtmlambdazs3.

To get this update, use (ql:update-dist "quicklisp").

This update was created with an older version of SBCL. The latest SBCL includes ASDF 3.3.1, which breaks a handful of projects in ways that have not been resolved yet. If you use the latest SBCL, see the build failure report to get an idea of what might not work properly.

Enjoy!

2017-12-04

November 2017 Quicklisp download stats


Here are the raw download counts for the top 100 projects in Quicklisp for November:

16654  alexandria
14200  closer-mop
13130  split-sequence
12669  cl-ppcre
12667  anaphora
12397  babel
12381  trivial-features
11989  iterate
11818  trivial-gray-streams
11520  bordeaux-threads
10907  let-plus
10821  cffi
10380  trivial-garbage
 9687  flexi-streams
 9603  puri
 9396  nibbles
 8847  usocket
 8712  more-conditions
 8008  cl+ssl
 7807  trivial-backtrace
 7741  cl-base64
 7707  chunga
 7505  utilities.print-items
 7409  chipz
 7269  esrap
 7098  drakma
 6996  cl-fad
 5862  cl-yacc
 5862  ironclad
 5524  named-readtables
 5337  local-time
 5239  parse-number
 4966  cxml
 4966  closure-common
 4878  fiveam
 4712  asdf-flv
 4565  cl-json
 4515  log4cl
 4375  bt-semaphore
 4258  architecture.hooks
 3904  plexippus-xpath
 3886  lparallel
 3745  parser.common-rules
 3576  lift
 3486  optima
 3238  cl-dot
 3166  slime
 3109  cl-unicode
 3101  cl-interpol
 3061  cxml-stp
 3036  cl-store
 3013  cl-clon
 2910  xml.location
 2890  trivial-utf-8
 2730  utilities.print-tree
 2660  uuid
 2606  fare-utils
 2595  md5
 2521  fare-quasiquote
 2489  metabang-bind
 2488  static-vectors
 2401  fare-mop
 2400  cl-utilities
 2398  inferior-shell
 2246  ieee-floats
 2227  quri
 2174  fast-io
 1961  hunchentoot
 1956  trivial-types
 1930  cl-annot
 1921  cl-syntax
 1711  symbol-munger
 1687  trivial-indent
 1679  collectors
 1671  arnesi
 1671  access
 1661  rfc2388
 1642  cl-slice
 1631  documentation-utils
 1626  array-utils
 1623  yason
 1623  plump
 1619  cl-parser-combinators
 1614  gettext
 1609  cl-locale
 1606  djula
 1602  cl-who
 1496  simple-date-time
 1415  osicat
 1366  parenscript
 1358  monkeylib-binary-data
 1305  postmodern
 1245  lisp-unit
 1239  trivial-shell
 1233  command-line-arguments
 1227  asdf-system-connections
 1223  cl-containers
 1221  metatilities-base
 1198  salza2
 1167  parse-float

November 2017 Quicklisp dist update now available

New projects:
  • cacle — Extensible cache services for Common Lisp — MIT
  • ccl-compat — Clozure CL compatibility module — LLGPL
  • ccldoc — create lisp documentation using s-expressions — Apache License 2.0
  • chancery — A library for procedurally generating text, inspired by Tracery. — MIT/X11
  • cl-flac — Bindings to libflac, a simple FLAC decoding library — Artistic
  • cl-portmanteau — cl-portmanteau — OSI approved 3-clause 'New BSD License'
  • clack-pretend — A testing and debugging tool for Clack — Apache License, version 2.0
  • clath — Clath is single sign-on middleware for Clack. It allows basic login with OAuth1.0a, OAuth2 and OpenID Connect. — Apache License, version 2.0
  • lisp-chat — An experimental chat irc-like — MIT
  • mockingbird — A small stubbing and mocking library for Common Lisp — MIT
  • qbase64 — Fast and flexible base64 encoder and decoder — BSD-3-Clause
  • specialization-store — The specialization store system provides a new kind of function, called a store function, whose behavior depends on the types of objects passed to the function. — Simplified BSD License variant
  • template-function — A system for generating functions from a template. — Simplified BSD License variant
  • trivial-package-manager — Functions for installing packages from distro-specific package manager. — LLGPL
Updated projectsahungry-fleecealso-alsaantikarray-operationsasdf-vizaws-sign4cellsceplcepl.sdl2ceramicchirpchungacl-anacl-ansi-termcl-asynccl-bplustreecl-charmscl-conllucl-cudacl-custom-hash-tablecl-data-framecl-fadcl-formscl-gdcl-gdatacl-graphcl-kyoto-cabinetcl-liballegrocl-messagepackcl-messagepack-rpccl-mixedcl-mpg123cl-neovimcl-online-learningcl-ppcrecl-pythoncl-random-forestcl-readlinecl-rulescl-satcl-sat.glucosecl-sat.minisatcl-sdl2cl-slicecl-tesseractcl-tiledcl-unicodecl-virtualboxcl-whoclachecloser-mopclssclxcoleslawconfiguration.optionscroatoandeedsdexadoreazy-projecteventfdfare-quasiquotefemlispflexi-streamsfs-watcherfxmlgamebox-dgengamebox-frame-managergettextglsl-specharmonyhu.dwim.partial-evalhu.dwim.quasi-quoteinquisitorironcladjsonrpcjsownlasslegionlegitlet-pluslichat-protocollichat-serverliblichat-tcp-clientlisp-namespacelocal-timelog4cllquerymaidenmaxpcmcclimmetabang-bindmk-string-metricsmodularizemoiramtifnibblesoclcloookoverlordparser.common-rulesplumppostmodernproveqlotqmyndqt-libsremote-jsrtg-mathscalplsdl2kitserapeumsimple-currencysketchstumpwmterminfotriviatrivial-batterytrivial-benchmarktrivial-clipboardtrivial-updatetrivial-wsunix-optsutilities.print-itemsvarjoverbosezenekindarl.

Removed projects: de.setf.wilbur, odd-streams.

To get this update, use (ql:update-dist "quicklisp"). Enjoy!

2017-11-06

October 2017 download stats

Here are the top 100 projects from Quicklisp for October, by "raw" download count.

16626  alexandria
15213  closer-mop
13436  anaphora
13420  split-sequence
12954  babel
12879  trivial-features
12719  iterate
12464  cl-ppcre
11798  bordeaux-threads
11724  let-plus
11693  trivial-gray-streams
11389  cffi
11281  trivial-garbage
10622  puri
 9884  nibbles
 9732  more-conditions
 9611  flexi-streams
 9026  usocket
 8602  cl+ssl
 8544  utilities.print-items
 8153  cl-base64
 8130  chunga
 7889  chipz
 7791  drakma
 7691  esrap
 7635  trivial-backtrace
 6308  ironclad
 5910  cl-yacc
 5691  cl-fad
 5302  parse-number
 4996  named-readtables
 4984  fiveam
 4959  asdf-flv
 4886  log4cl
 4756  bt-semaphore
 4736  local-time
 4701  lparallel
 4647  closure-common
 4638  cxml
 4594  architecture.hooks
 4552  lift
 3784  plexippus-xpath
 3586  cl-json
 3569  trivial-utf-8
 3322  optima
 3157  parser.common-rules
 3144  cl-clon
 2837  uuid
 2819  cxml-stp
 2705  xml.location
 2700  metabang-bind
 2624  cl-dot
 2475  utilities.print-tree
 2473  slime
 2458  cl-unicode
 2456  cl-interpol
 2273  md5
 2267  cl-store
 2232  fare-utils
 2204  fare-quasiquote
 2108  inferior-shell
 2105  fare-mop
 1769  cl-utilities
 1706  quri
 1671  ieee-floats
 1625  static-vectors
 1605  fast-io
 1547  trivial-types
 1545  cl-annot
 1536  cl-syntax
 1437  utilities.binary-dump
 1431  trivial-indent
 1364  trivial-mimes
 1335  asdf-system-connections
 1334  array-utils
 1329  symbol-munger
 1320  cl-containers
 1318  metatilities-base
 1318  plump
 1302  cl-slice
 1296  hunchentoot
 1280  access
 1267  arnesi
 1266  collectors
 1258  gettext
 1236  djula
 1226  cl-parser-combinators
 1221  cl-locale
 1187  postmodern
 1164  rfc2388
 1159  yason
 1121  simple-date-time
 1050  command-line-arguments
  956  cl-sqlite
  951  cl-log
  947  osicat
  943  salza2
  913  py-configparser
  903  cl-markdown
  903  asdf-finalizers