A small step in the right direction: https for quicklisp.org

I had a joke slide at ELS last week that explained why Quicklisp was so easy to install: just use curl install.quicklisp.net | sudo sh.  (Don't try this.) Although Quicklisp's installation isn't as risky as piping random code into a root shell, it does have its own problems. Several people at the conference asked me when I would add more security features to Quicklisp.

As of this week, www.quicklisp.org is available through an https connection. Any requests that come in over http are redirected to the equivalent https location. That means you can have some confidence that the information there is provided by me, rather than intercepted and replaced by a third party.

The main Quicklisp website is only part of the story. The software to install and use Quicklisp is hosted on another domain, beta.quicklisp.org. That domain now has optional https access, so that any URL may be accessed either through https or http.

That means the bootstrap file quicklisp.lisp is available via https, and so is the PGP key I use to sign client software and dist metadata. (That key is also available via various PGP keyservers.) If you have programs that fetch quicklisp.lisp or software archives directly from beta.quicklisp.org, I encourage you to update them to use https instead of http.

Why doesn't beta.quicklisp.org use https exclusively? Unfortunately, the Quicklisp client code itself does not know how to connect via https, so turning off http access would break Quicklisp completely. It will take more time to update the Quicklisp client code to use https.

Implementing https for quicklisp.org is a small, but important, first step toward making the use of Quicklisp safer. If you have any questions or concerns, please get in touch via zach@quicklisp.org.


Dealing with "redundant" libraries

People sometimes submit libraries to Quicklisp that have significant functional overlap with other libraries. When someone wants to add a new library for functionality that is well-covered by other libraries, I don't always add it right away. There are a few questions I ask first.

First, are you aware that there are many other libraries that fit a similar need? If not, is it possible to use one of those instead? It is possible that an existing library is more complete and mature than a new library. And adding a new library can make it harder for someone else to choose a good option.

If you're aware of other libraries, is it possible that the essential new functionality your library provides can be incorporated into an existing library? 

If you're aware of other libraries, but still feel you need a new one, and it can't be incorporated into an existing library, I'll go ahead and add the library. 

Here are a few recent examples of that process playing out:
I don't reject libraries for lack of novelty. It's always possible that a new library becomes so clearly superior to all other options that it is the go-to library of the future. But it's best when that happens because you're aware of the existing options (and their strengths and limitations) rather than ignorant of them.


New feature: Quicklisp library bundles

Quicklisp library bundles are self-contained sets of systems that are exported from Quicklisp and loadable without involving Quicklisp.

I added the feature to the Quicklisp client on April 18th and announced it at ELS on the 20th. To get it, use (ql:update-client). It will be available when you restart for the next session.

I've wanted to provide this feature since the beginning of Quicklisp. I primarily pictured this as a kind of delivery tool, where your project should have access to a specific set of supporting libraries without the complexity of Quicklisp loaded in as well.

The interface is pretty simple: (ql:bundle-systems '("foo" "bar" "baz") :to "my-bundle/") will produce a file named "my-bundle/bundle.lisp" that, when loaded, will make "foo", "bar", and "baz" accessible via ASDF without involving Quicklisp at all. All the libraries on which "foo", "bar", and "baz" depend are also included in the bundle, recursively.

The bundle can be relocated without any issues. All its pathnames and indexes are relative to its bundle.lisp.

When bundle.lisp is loaded, the systems in the bundle take precedence over all other ASDF systems. Multiple bundles can be loaded at the same time; the most recently loaded one has the highest precedence. And a bundle can be loaded more than once. If loaded again, it will be moved to the top of the precedence list again.

Bundles also have a local-projects directory that adds some of the automagic from the Quicklisp local-projects feature. Bundle local-projects systems take precedence over the bundle's "built-in" systems.

Quicklisp library bundles are also documented! But they are only lightly tested. If you think library bundles might help you solve a problem, please give them a try and let me know if you have any problems or questions.


April 2015 Quicklisp dist update now available

This Quicklisp update is supported by my employer, Clozure Associates. If you need commercial support for Quicklisp, or any other Common Lisp programming needs, it's available via Clozure Associates.

New projects:
  • birch — A simple Common Lisp IRC client library — MIT
  • bytecurry.mocks — Tools to mock functions for unit tests — MIT
  • carrier — An async HTTP client — MIT
  • cl-cookie — HTTP cookie manager — BSD 2-Clause
  • cl-coveralls — Coverage tracker for Coveralls — BSD 2-Clause
  • cl-groupby — groupby: A higher order function named groupby as known from Scala. — MIT
  • cl-openstack-client — OpenStack client libraries — Apache-2.0
  • cl-poker-eval — 7-card hand poker evaluator — BSD
  • cl-shellwords — Common Lisp port of Ruby's shellwords.rb, for escaping and splitting strings to be passed to a shell. — MIT
  • cl-singleton-mixin — provides singleton-mixin class. — MIT
  • dexador — Yet another HTTP client for Common Lisp — MIT
  • duologue — High-level user interaction library for Common Lisp — MIT
  • dyna — Dyna is an AWS DynamoDB ORM for Common Lisp. — MIT
  • find-port — Find open ports programmatically. — MIT
  • fmt — Extensible format-like facility — MIT
  • folio2 — the folio2 functional-idioms system — Lisp Lesser GNU Public License
  • immutable-struct — Library that encourage the use of functional programming + pattern matching — LLGPL
  • jonathan — JSON encoder and decoder. — MIT
  • json-responses — Canned JSON responses for Hunchentoot — MIT
  • kebab — Common Lisp string,symbol,keyword PascalCase <=> camelCase <=> snake_case <=> kebab-case(lisp-case) converter. — LLGPL
  • lack — A minimal Clack — LLGPL
  • metap — Metap provides metaclass propagation along class inheritance structure. — MIT
  • nsort — Natural or Numeric Sort — BSD Simplified
  • plump-bundle — A binary storage format for Plump documents. — Artistic
  • proc-parse — Procedural vector parser — BSD 2-Clause
  • quadtree — Quadtree data structure in Common Lisp — MIT
  • quasiquote-2.0 — Writing macros that write macros. Effortless. — MIT
  • scriba — A markup format similar to Scribe. — MIT
  • should-test — Minimal yet feature-rich Common Lisp test framework. — MIT
  • string-escape — Emacs and Python style string escapes in #"..." form. — GPLv3
  • thorn — A CommonDoc extension for entering special characters. — MIT
  • trivia — NON-optimized pattern matcher compatible with OPTIMA, with extensible optimizer interface and clean codebase — LLGPL
  • trivia.balland2006 — Optimizer for Trivia based on (Balland 2006) — LLGPL
  • type-i — Type Inference Utility on Fundamentally 1-arg Predicates — LLGPL
  • type-r — Collections of accessor functions and patterns to access the elements in compound type specifier, e.g. `dimensions' in `(array element-type dimensions)' — LLGPL
  • unix-opts — minimalistic parser of command line arguments — MIT
Updated projects: 3bmd, access, antik, arrow-macros, buffalo, buildapp, chanl, cl-ana, cl-annot, cl-ansi-term, cl-async, cl-charms, cl-dbi, cl-dot, cl-factoring, cl-gobject-introspection, cl-grace, cl-indeterminism, cl-launch, cl-libyaml, cl-mlep, cl-mtgnet, cl-mw, cl-netstring-plus, cl-openid, cl-ply, cl-python, cl-quickcheck, cl-random, cl-readline, cl-reddit, cl-redis, cl-rlimit, cl-sdl2, cl-slug, cl-svg, cl-syntax, cl-tcod, cl-vectors, cl-voxelize, cl-yaml, clack, classimp, clavier, clhs, clim-widgets, clinch, clipper, clos-fixtures, closer-mop, clsql-helper, clx, codata-recommended-values, colleen, com.informatimago, common-doc, common-doc-plump, common-html, commonqt, croatoan, defclass-std, djula, drakma, eazy-project, esrap-liquid, fare-memoization, fast-http, fast-io, femlisp, gendl, graph, gsll, hdf5-cffi, hl7-client, hl7-parser, http-body, hu.dwim.util, hunchentoot, hyperluminal-mem, integral, interface, introspect-environment, js-parser, jsown, jwacs, lass, let-over-lambda, lfarm, linedit, lisp-interface-library, lisp-invocation, lisp-namespace, local-time, lucerne, magicffi, mcclim, media-types, mgl-pax, mk-string-metrics, nibbles, ningle, nst, plump, protobuf, qtools, quri, quux-time, racer, rutils, scalpl, scriptl, sdl2kit, serapeum, shellpool, simple-rgb, skippy, slime, st-json, staple, stumpwm, symbol-munger, trivial-backtrace, trivial-benchmark, trivial-debug-console, trivial-download, trivial-update, umlisp, verbose, vertex, vgplot, weblocks-stores, weft, workout-timer, xmls, zaws, zcdb, zpb-exif, zpng.

Removed projects: autoproject, brlapi, cambl, cl-couch, cl-ledger, hctsmsl, nekthuth, red-black.

To get these updates, use (ql:update-dist "quicklisp").



March 2015 download stats

Here are the top 100 download for last month:

 5853   alexandria
 4149   cl-ppcre
 3465   closer-mop
 3382   trivial-features
 3296   babel
 3086   named-readtables
 2977   cffi
 2814   flexi-streams
 2795   bordeaux-threads
 2785   trivial-gray-streams
 2784   cl+ssl
 2722   cl-fad
 2641   trivial-garbage
 2315   nibbles
 2313   usocket
 2277   chunga
 2251   anaphora
 2235   cl-base64
 2218   optima
 2142   split-sequence
 1942   ironclad
 1839   puri
 1803   fiveam
 1768   iterate
 1753   drakma
 1729   chipz
 1641   cl-colors
 1629   trivial-backtrace
 1586   local-time
 1559   md5
 1493   slime
 1486   let-plus
 1401   fare-utils
 1392   fare-quasiquote
 1340   cl-ansi-text
 1190   prove
 1180   hunchentoot
 1098   cl-unicode
 1072   trivial-types
 1053   rfc2388
 1021   metabang-bind
 1010   cl-utilities
  986   cl-interpol
  961   cl-syntax
  950   trivial-utf-8
  889   introspect-environment
  846   quicklisp-slime-helper
  827   cl-annot
  802   parse-number
  757   st-json
  716   quri
  710   osicat
  709   postmodern
  686   cl-marshal
  658   trivial-mimes
  654   xsubseq
  651   lparallel
  648   plump
  645   jsown
  641   asdf-system-connections
  614   ieee-floats
  613   trivial-indent
  605   metatilities-base
  604   uuid
  603   array-utils
  601   cl-containers
  596   cl-json
  590   lquery
  584   fast-http
  572   cl-sqlite
  567   salza2
  556   clss
  538   clack
  500   static-vectors
  495   clx
  490   command-line-arguments
  464   cl-markdown
  463   py-configparser
  462   dynamic-classes
  455   circular-streams
  446   asdf-finalizers
  446   zpb-ttf
  446   cl-log
  445   fast-io
  443   cl-abnf
  442   garbage-pools
  440   buildapp
  435   cl-mssql
  425   cl-who
  399   zpng
  397   esrap
  395   http-body
  394   vecto
  391   cl-csv
  389   cl-vectors
  388   iolib
  358   closure-common
  353   lisp-namespace
  351   cl-opengl
  351   cl-dbi


March 2015 Quicklisp dist update now available

New projects:
  • arrow-macros — arrow-macros provides clojure-like arrow macros and diamond wands. — MIT
  • buffalo — A LALR(1) parser generator for Common Lisp — MIT/X11
  • burgled-batteries — Lisp-Python interface — MIT
  • burgled-batteries.syntax — Embedded Python syntax for burgled-batteries — MIT
  • cl-libyaml — A binding to the libyaml library. — MIT
  • cl-mlep — cl-mlep is a Common Lisp Machine Learning library for Educational Purposes. — MIT
  • cl-mtgnet
  • cl-netstring-plus — A simple library for sending and receiving messages with a netstring-like encoding
  • cl-reddit — Reddit client api library — BSD
  • cl-yaml — A YAML parser and emitter. — MIT
  • cl4store — 4store — BSD
  • clavier — Clavier: A Common Lisp validation library — MIT
  • clipper — File attachment library. — MIT
  • clods-export — Common Lisp OpenDocument spreadsheet export library — MIT
  • common-doc-plump — Translate a Plump DOM into a CommonDoc document and back. — MIT
  • descriptions — A domain model meta level description library — MIT
  • eazy-process — Yet Another Portable Library for Process Handling / Subshell Invokation — MIT
  • hl7-client — hl7-client - send HL7-Messages over TCP/IP with MLLP — BSD
  • hl7-parser — Decoder and Encoder for ER7 formatted HL7-Messages — BSD
  • hyperluminal-mem — High-performance serialization library, designed for untrusted data — LLGPL
  • linewise-template — Linewise file/stream processor for code generation etc. — BSD-3-Clause
  • lisp-namespace — Provides LISP-N --- extensible namespaces in Common Lisp. — LLGPL
  • media-types — Query and compare media types. — MIT
  • ryeboy — Riemann client — BSD
  • shellpool — A library for running external programs from Common Lisp — MIT/X11-style
  • transparent-wrap — A signature-preserving wrapper generator for functions and macros. — LLGPL
  • trivial-debug-console — A library for a well-behaved debug console
  • vertex — A markup language with TeX syntax. — MIT
  • weft
  • workout-timer — Workout timer — MIT
Updated projects: architecture.service-provider, asdf-linguist, avatar-api, basic-binary-ipc, binge, blackbird, caveman, chanl, chillax, chrome-native-messaging, cl+ssl, cl-acronyms, cl-ana, cl-ansi-term, cl-async, cl-autowrap, cl-charms, cl-cli, cl-colors, cl-conspack, cl-csv, cl-custom-hash-table, cl-db3, cl-dbi, cl-devil, cl-fsnotify, cl-gobject-introspection, cl-gss, cl-i18n, cl-l10n, cl-libuv, cl-marshal, cl-messagepack, cl-mock, cl-murmurhash, cl-mysql, cl-openal, cl-opengl, cl-prime-maker, cl-python, cl-rabbit, cl-read-macro-tokens, cl-readline, cl-redis, cl-rethinkdb, cl-rlimit, cl-sdl2, cl-slug, cl-speedy-queue, cl-twitter, cl-webkit, cl-xul, cl-yaclyaml, clack, clack-errors, clfswm, clim-widgets, clip, clos-fixtures, closer-mop, clot, clpmr, clsql, clss, clx, cobstor, colleen, com.informatimago, common-doc, common-html, commonqt, corona, crane, defclass-std, defenum, dissect, djula, drakma-async, eazy-gnuplot, eazy-project, eco, esrap, esrap-liquid, external-program, fast-http, form-fiddle, gbbopen, gendl, http-body, hunchensocket, integral, iolib, lisp-executable, lparallel, lquery, lucerne, mgl-pax, micmac, mime4cl, mk-string-metrics, modularize-interfaces, myway, named-readtables, net4cl, nibbles, optima, osicat, packet, pgloader, piping, plump, pooler, qlot, qmynd, qtools, quri, racer, rcl, rock, scalpl, sclf, sdl2kit, serapeum, sheeple, simple-currency, skippy, smtp4cl, south, spinneret, squirl, st-json, staple, stmx, stumpwm, sxql, talcl, trivial-arguments, trivial-benchmark, trivial-download, trivial-extract, trivial-mimes, trivial-update, unix-options, until-it-dies, utils-kt, utm, verbose, weblocks, weblocks-utils, woo, wookie.

Removed projects: 3bil, cl-binaural, cl-openstack, cluck, lisphys, oct.

I removed 3bil because it is effectively unmaintained and apparently unused. cl-openstack is no longer maintained; a replacement library will soon be available. I can't build cl-binaural, cluck, lisphys, or oct on SBCL any more.

Sorry to skip a release in February. A holiday and a number of library problems conspired to hold off the release. As a result there are a larger number of updates this month than usual. If you have any problems, please don't hesitate to discuss them on the mailing list or email me directly.

To get this update, use (ql:update-dist "quicklisp").


February 2015 download stats

Here are the top 100 download for last month:

 4270   alexandria
 2825   trivial-features
 2753   cl-ppcre
 2740   babel
 2464   closer-mop
 2316   cl-fad
 2309   bordeaux-threads
 2211   cffi
 2126   iterate
 2103   trivial-garbage
 2063   slime
 2042   flexi-streams
 1944   trivial-gray-streams
 1867   split-sequence
 1771   anaphora
 1744   chunga
 1651   cl+ssl
 1646   usocket
 1601   local-time
 1558   cl-base64
 1441   metabang-bind
 1327   trivial-backtrace
 1304   named-readtables
 1292   drakma
 1277   nibbles
 1249   puri
 1234   ironclad
 1230   md5
 1206   optima
 1152   chipz
  988   trivial-types
  948   hunchentoot
  893   cl-utilities
  885   cl-unicode
  813   rfc2388
  784   let-plus
  777   quicklisp-slime-helper
  777   stefil
  753   cl-interpol
  745   trivial-utf-8
  729   cl-syntax
  724   cl-annot
  714   quri
  706   postmodern
  658   parse-number
  630   fiveam
  611   cl-colors
  591   salza2
  561   cl-json
  542   log4cl
  514   cl-ansi-text
  511   lparallel
  500   cl-who
  482   prove
  482   ieee-floats
  481   zpb-ttf
  473   zpng
  461   vecto
  453   cl-sqlite
  437   uuid
  433   cl-vectors
  425   cl-yacc
  423   st-json
  417   asdf-system-connections
  403   osicat
  394   parenscript
  385   metatilities-base
  383   cl-containers
  374   clx
  368   buildapp
  367   closure-common
  364   clack
  355   yason
  346   command-line-arguments
  345   fare-utils
  341   iolib
  335   cxml
  335   cl-marshal
  324   static-vectors
  312   fare-quasiquote
  309   xsubseq
  308   dynamic-classes
  304   py-configparser
  301   cl-markdown
  299   fast-http
  297   asdf-finalizers
  293   cl-log
  288   external-program
  286   trivial-indent
  286   cl-abnf
  282   garbage-pools
  279   plump
  279   cl-mssql
  274   contextl
  271   cl-opengl
  268   trivial-mimes
  264   html-template
  262   fast-io
  251   array-utils
  249   zs3