2015-05-28

Looking for more metadata

A few days ago I linked to a report showing a lot of systems that failed to build. They failed because I added an option in the Quicklisp build environment that signals an error if a system lacks the description, author, and license metadata.

This isn't a standard feature of ASDF or Quicklisp. No projects are going to be dropped next month because of it. It's an optional piece of the build system, one that I added so I could see how many systems are missing that useful data and how likely it is that people will care.

I really want to use the :description option and show it as output in the REPL when searching for systems with something like system-apropos. I also want to make it easy to quickly determine the license of a given system, so you can figure out if it's compatible with your project. And having author information readily available will make it easier to contact someone regarding the project.

ASDF system metadata is a good choice for storing this information because it's not Quicklisp-specific. Anyone can gather and use this data if it's present in the systems. I hope that in the future every system in every project will have as much useful and accurate metadata as possible.

So what should you do if you want to help with this goal?

First, if you maintain a system and it's in the report, please update each of its system definitions with :description, :author, and :license information. A good description should be no longer than a tweet,and give an idea of what the system is for. The author information should include a name and email address. The license should be short and refer to a well-known license if possible, or give information about where to read the full license otherwise.

If you're not the maintainer of a system, but you want to file an issue or bug report, consider making a polite request to the author that they update their systems to include the extra info. (If you can, make sure nobody else has submitted the request first.)

2015-05-08

ASDF 3 is coming to Quicklisp soon

In the next week or two, I'm going to update Quicklisp so that if ASDF is missing, or not at least version 3.1, it will fetch and load ASDF 3.1.4. This should be a pretty conservative change. Almost all implementations already include ASDF 3.

If you'd like to try the new Quicklisp client that fetches ASDF 3 if needed, see this post to the Quicklisp mailing list.

2015-05-06

May 2015 Quicklisp dist update now available

This Quicklisp update is supported by my employer, Clozure Associates. If you need commercial support for Quicklisp, or any other Common Lisp programming needs, it's available via Clozure Associates.

New projects:

  • bytecurry.asdf-ext — ASDF extension(s) for generating atdoc documentation. — MIT
  • cl-durian — dynamic html generation from list structures (interpolation friendly) — WTFPL
  • cl-marklogic — Common Lisp library for accessing MarkLogic Server. — LGPL3
  • cl-pslib — A CFFI wrapper for the pslib library, a library for generating PostScript files. — LLGPL
  • cl-pslib-barcode — A barcode generator for the cl-pslib library. — LLGPL
  • cl-simple-concurrent-jobs — A simple API for running concurrent jobs and collecting the results — BSD 2-Clause
  • cl-sophia — High-level API for Sophia key-value storage — WTFPL
  • cl-strftime — Common Lisp compiler for the strftime language. — MIT
  • cl-tga — TGA file loader — MIT
  • erudite — Literate Programming System for Common Lisp — MIT
  • intel-hex — A library to handle Intel HEX format. — MIT
  • snakes — Python style generators for Common Lisp. — Apache 2.0

Updated projects: alexandria, antik, arrow-macros, babel, beirc, birch, blackbird, buffalo, bytecurry.mocks, caveman, chanl, chipz, cl+ssl, cl-ana, cl-async, cl-charms, cl-enumeration, cl-gobject-introspection, cl-grace, cl-graph, cl-i18n, cl-launch, cl-mtgnet, cl-netstring-plus, cl-ply, cl-quickcheck, cl-rabbit, cl-read-macro-tokens, cl-readline, cl-rethinkdb, cl-sdl2, cl-singleton-mixin, cl-slug, cl-voxelize, cl-yaml, clack, clack-errors, clim-widgets, climacs, clinch, clipper, closer-mop, colleen, common-doc, common-html, common-lisp-stat, commonqt, corona, dartsclhashtree, dartsclmessagepack, defclass-std, dyna, eazy-process, exscribe, f2cl, fare-csv, fast-http, function-cache, gbbopen, generic-comparability, gsll, hu.dwim.delico, hu.dwim.stefil, hu.dwim.syntax-sugar, hu.dwim.util, immutable-struct, inferior-shell, jonathan, json-responses, lass, let-over-lambda, lev, lisp-interface-library, lisp-invocation, lisp-matrix, lquery, lucerne, metap, mexpr, mgl-pax, nibbles, ningle, nsort, perlre, plump, proc-parse, qlot, qtools, quadtree, quasiquote-2.0, scalpl, scriba, scribble, serapeum, shellpool, should-test, shuffletron, staple, stmx, stumpwm, thorn, transparent-wrap, trivial-download, usocket, with-c-syntax, wookie.

To get this update, use (ql:update-dist "quicklisp").

2015-05-05

April 2015 download stats

Here are the top 100 downloads for last month:
 4230  alexandria
 3401  trivial-features
 3377  babel
 3366  cl-ppcre
 3102  cffi
 2717  bordeaux-threads
 2620  closer-mop
 2611  trivial-garbage
 2605  cl-fad
 2575  flexi-streams
 2574  trivial-gray-streams
 2471  cl+ssl
 2366  nibbles
 2245  usocket
 2228  cl-base64
 2210  split-sequence
 2191  chunga
 2087  slime
 2047  iterate
 2046  trivial-backtrace
 2042  drakma
 2002  anaphora
 1925  ironclad
 1691  puri
 1682  chipz
 1653  local-time
 1520  named-readtables
 1455  md5
 1399  hunchentoot
 1259  metabang-bind
 1152  cl-colors
 1124  let-plus
 1070  optima
 1051  cl-unicode
 1020  trivial-utf-8
  992  cl-syntax
  950  rfc2388
  946  cl-interpol
  943  trivial-types
  941  cl-annot
  934  cl-ansi-text
  846  postmodern
  839  prove
  803  parse-number
  799  asdf-system-connections
  791  uuid
  780  cl-utilities
  765  cl-containers
  765  metatilities-base
  760  quicklisp-slime-helper
  701  fast-io
  695  jsown
  672  ieee-floats
  671  lparallel
  664  static-vectors
  658  cl-json
  640  plump
  605  fiveam
  601  zpng
  594  lquery
  590  trivial-indent
  587  clss
  578  buildapp
  566  array-utils
  559  xsubseq
  557  cl-sqlite
  547  salza2
  546  quri
  542  command-line-arguments
  540  osicat
  521  garbage-pools
  520  fast-http
  515  cl-mssql
  509  cl-who
  508  cl-vectors
  506  clx
  505  iolib
  503  py-configparser
  499  dynamic-classes
  497  asdf-finalizers
  497  cl-log
  496  cl-marshal
  494  cl-markdown
  484  trivial-mimes
  483  cl-abnf
  477  clack
  465  fare-utils
  450  ningle
  418  zpb-ttf
  408  cl-dbi
  408  st-json
  404  http-body
  403  circular-streams
  398  closure-common
  392  fare-quasiquote
  380  cl-csv
  380  cxml
  377  parenscript
  373  myway
  352  map-set

2015-05-01

A small step in the right direction: https for quicklisp.org

I had a joke slide at ELS last week that explained why Quicklisp was so easy to install: just use curl install.quicklisp.net | sudo sh.  (Don't try this.) Although Quicklisp's installation isn't as risky as piping random code into a root shell, it does have its own problems. Several people at the conference asked me when I would add more security features to Quicklisp.

As of this week, www.quicklisp.org is available through an https connection. Any requests that come in over http are redirected to the equivalent https location. That means you can have some confidence that the information there is provided by me, rather than intercepted and replaced by a third party.

The main Quicklisp website is only part of the story. The software to install and use Quicklisp is hosted on another domain, beta.quicklisp.org. That domain now has optional https access, so that any URL may be accessed either through https or http.

That means the bootstrap file quicklisp.lisp is available via https, and so is the PGP key I use to sign client software and dist metadata. (That key is also available via various PGP keyservers.) If you have programs that fetch quicklisp.lisp or software archives directly from beta.quicklisp.org, I encourage you to update them to use https instead of http.

Why doesn't beta.quicklisp.org use https exclusively? Unfortunately, the Quicklisp client code itself does not know how to connect via https, so turning off http access would break Quicklisp completely. It will take more time to update the Quicklisp client code to use https.

Implementing https for quicklisp.org is a small, but important, first step toward making the use of Quicklisp safer. If you have any questions or concerns, please get in touch via zach@quicklisp.org.

2015-04-30

Dealing with "redundant" libraries

People sometimes submit libraries to Quicklisp that have significant functional overlap with other libraries. When someone wants to add a new library for functionality that is well-covered by other libraries, I don't always add it right away. There are a few questions I ask first.

First, are you aware that there are many other libraries that fit a similar need? If not, is it possible to use one of those instead? It is possible that an existing library is more complete and mature than a new library. And adding a new library can make it harder for someone else to choose a good option.

If you're aware of other libraries, is it possible that the essential new functionality your library provides can be incorporated into an existing library? 

If you're aware of other libraries, but still feel you need a new one, and it can't be incorporated into an existing library, I'll go ahead and add the library. 

Here are a few recent examples of that process playing out:
I don't reject libraries for lack of novelty. It's always possible that a new library becomes so clearly superior to all other options that it is the go-to library of the future. But it's best when that happens because you're aware of the existing options (and their strengths and limitations) rather than ignorant of them.

2015-04-28

New feature: Quicklisp library bundles

Quicklisp library bundles are self-contained sets of systems that are exported from Quicklisp and loadable without involving Quicklisp.

I added the feature to the Quicklisp client on April 18th and announced it at ELS on the 20th. To get it, use (ql:update-client). It will be available when you restart for the next session.

I've wanted to provide this feature since the beginning of Quicklisp. I primarily pictured this as a kind of delivery tool, where your project should have access to a specific set of supporting libraries without the complexity of Quicklisp loaded in as well.

The interface is pretty simple: (ql:bundle-systems '("foo" "bar" "baz") :to "my-bundle/") will produce a file named "my-bundle/bundle.lisp" that, when loaded, will make "foo", "bar", and "baz" accessible via ASDF without involving Quicklisp at all. All the libraries on which "foo", "bar", and "baz" depend are also included in the bundle, recursively.

The bundle can be relocated without any issues. All its pathnames and indexes are relative to its bundle.lisp.

When bundle.lisp is loaded, the systems in the bundle take precedence over all other ASDF systems. Multiple bundles can be loaded at the same time; the most recently loaded one has the highest precedence. And a bundle can be loaded more than once. If loaded again, it will be moved to the top of the precedence list again.

Bundles also have a local-projects directory that adds some of the automagic from the Quicklisp local-projects feature. Bundle local-projects systems take precedence over the bundle's "built-in" systems.

Quicklisp library bundles are also documented! But they are only lightly tested. If you think library bundles might help you solve a problem, please give them a try and let me know if you have any problems or questions.