Wish list: parallel fetching in Quicklisp Controller

Fetching updates for all the projects in Quicklisp takes a long time. For each project it does a "git pull" or "svn update" or equivalent, depending on how the project is managed. It does this one at a time, serially.

There's no reason it couldn't do more than one update at a time. An ideal system would fetch some of the projects in parallel, taking care not to fetch multiple projects from the same site at a time. For example, there are dozens and dozens of github-hosted projects, but the Quicklisp controller should only have one github connection open at a time. Same with common-lisp.net, dwim.hu, bitbucket, and the same for other sites that host multiple projects.

It's possible that lparallel or lfarm could help with this task.

Are you up for adding this feature to quicklisp-controller? Hacks and glory await!

update Orivej Desh has done it! Thanks Orivej!


Quicklisp Controller source now available

Quicklisp Controller, the software I use to create Quicklisp dists, is now on github. It has modest documentation, and isn't really in a state that anyone could casually pick up and use, but it's now available.

I meant to do this last year after my talk at the Boston Lisp Meeting, but procrastinated. No longer. Feel free to check it out.

There's also a smaller, weirder program I use to upload the dist to Amazon: quicklisp-dist.


August 2013 dist update now available

New projects: cl-autowrap, cl-date-time-parser, cl-ledger, cl-paymill, cl-string-match, epigraph, trivial-tco, vgplot.

Updated projects: access, asdf-dependency-grovel, bt-semaphore, cl-6502, cl-async, cl-bencode, cl-cairo2, cl-enchant, cl-fbclient, cl-gendoc, cl-grace, cl-gss, cl-html-parse, cl-opengl, cl-redis, clfswm, closer-mop, clsql-helper, clws, coleslaw, com.informatimago, common-lisp-stat, commonqt, css-selectors, diff, esrap, exscribe, fs-watcher, gbbopen, gendl, hu.dwim.debug, hu.dwim.def, hu.dwim.perec, hu.dwim.quasi-quote, hu.dwim.stefil, hunchentoot, lfarm, lisp-executable, lisp-unit, lquery, macroexpand-dammit, optima, quicksearch, rfc3339-timestamp, shelly, simple-date-time, spinneret, stmx, swank-crew, uiop, uuid, weblocks, weblocks-utils.

Renamed projects: cl-hctsmsl has been renamed hctsmsl.

I built this dist with ASDF 3. In the near future, Quicklisp will provide ASDF 3 on systems where it is not present. Please let me know if you have any trouble loading anything.

If you'd like to search for useful Quicklisp libraries, I encourage you to give Quickdocs a try!