2012-01-03

Speeding up system info

Quicklisp uses two text files for information about project releases (releases.txt) and systems (systems.txt). Whenever information about a system was needed (for example, where its system file can be found), both files were loaded completely, from scratch.

I made it that way because it was pretty easy. People immediately noticed that it was also pretty slow, especially when using (asdf:load-system "...") instead of (ql:quickload "..."). The penalty for frequent loading and reloading of the metadata got worse as the number of Quicklisp systems grew.

Today I released a client update that can load metadata from a fast on-disk hash table, a CDB file. The big flat files are converted to CDB once, as needed, and thereafter getting metadata is super-speedy. On my laptop, the improvement for lookups is about 100x; your results will depend on the speed of your disk.

To get the update, use (ql:update-client) and restart Lisp.

If this change causes you any trouble, please let me know via the Quicklisp mailing list.

UPDATE: This change causes trouble. I'm going to put out a fix as soon as I can. Please don't update; if you did update, you can revert with something like this:

    cd ~/quicklisp/
    wget http://beta.quicklisp.org/quickstart/quicklisp-2011111500.tgz
    tar xzvf quicklisp-2011111500.tgz
    rm -rf ~/.cache/common-lisp/

UPDATE 2: I can't reproduce the trouble I saw earlier. If you run into trouble, let me know. I really need cases I can reproduce to fix things.

UPDATE 3: Thanks to Mike Clarke, I found a serious problem with the CDB scheme. A fix will be available within a day or two.

3 comments:

  1. Is the CDB conversion / query library integral part of quicklisp? It might be very useful for other projects.

    ReplyDelete
  2. I basically cut & pasted code from my ZCDB library into Quicklisp, so no, it is not tightly integrated.

    ReplyDelete
  3. I did not know about ZCDB, but I liked the format so much that I have adopted it for ECL's help file (the one that stores all the documentation strings from the library) and reimplemented it as a loadable module (require :ecl-cdb) respecting the ZCDB interface.

    ReplyDelete