2013-08-31

Wish list: parallel fetching in Quicklisp Controller

Fetching updates for all the projects in Quicklisp takes a long time. For each project it does a "git pull" or "svn update" or equivalent, depending on how the project is managed. It does this one at a time, serially.

There's no reason it couldn't do more than one update at a time. An ideal system would fetch some of the projects in parallel, taking care not to fetch multiple projects from the same site at a time. For example, there are dozens and dozens of github-hosted projects, but the Quicklisp controller should only have one github connection open at a time. Same with common-lisp.net, dwim.hu, bitbucket, and the same for other sites that host multiple projects.

It's possible that lparallel or lfarm could help with this task.

Are you up for adding this feature to quicklisp-controller? Hacks and glory await!

update Orivej Desh has done it! Thanks Orivej!

1 comment:

  1. What about concurrent compilation? Given a system, launch a subprocess, compile the system, report back. If you're loading a system with three dependencies, compile them in parallel. Wait for the longest one. Hmm. Must look into it.

    ReplyDelete