text:
This is the root of the common Lisp Directory. The whole tree of tags starts here.
If you are new to the Common Lisp Directory, you should probably first have a look at the "By topic" link below or at the "Tags" and "Index" links above.
The items lists (Libraries, documents, events, etc.) are limited to about 20 items randomly taken from the whole lists. Click on the titles or the "View all (n more)" links to see the full lists of items with their descriptions.
The CLD is written in Common Lisp.
By Topic (681 items) Resources for Common Lisp sorted by topic
Frequently Asked Questions (8 items) The Common Lisp Directory FAQ
Information (189 items) Various sources for information
Software (621 items)
By Support (93 items) Resources sorted by kind of support (books, web-sites, etc...)
People, Groups and Organizations (330 items) Humans related to Common Lisp
Programming Languages (162 items)
Standards and Specifications (57 items)
By Operating System (454 items) Resourses sorted by Operating System
By License (588 items) Resources sorted by their license
Human Languages (27 items)
By location (172 items)
Other resources (3 items) Resources that are not already in the other tags
Lisp Jobs (2 items) Lisp Jobs opportunities
cl-buchberger | spatial-trees | Wu-Decimal | DEFPATT | Glouton | nuclblog: a blog engine for use with Hunchentoot | Alpaca | GNU Scientific Library for Lisp | FTD - The Flexi-Trivial Dired | CL-BibTeX | McCLIM | TwinLisp | S-Dot: A Common Lisp Interface to Graphviz Dot | cl-semantic | Simple Grid Protocol | Acclaim | csv-parser | Maxima | ch-asdf | sequence-search/replace | cl-win32ole | YstokHTML - collection of HTML generator, parser a... | Float numbers input-output | ZLIB | clg | ASDlite | Genhash | Wilbur Semantic Web Toolkit for CLOS | fare-csv | Tabcode | Maintaining dynamic state | ch-asdf-markup | Ironclad | CL-ENUMERATIONS | Core Services
View all (634 more)...
CDR Blog | BioLisp.org | Literate Programming in Lisp | CDR 1: The CLOS Metaobject Protocol | The Idiot's Guide to Special Variables and Lexical... | Paradigms of Artificial Intelligence Programming | CLIM Screencast - Installation and Hello World | Some Useful Lisp Algorithms: Part 1 | MOP Retrospective - Gregor Kiczales speaks to lis... | LISP Deserves a Fresh Look | Common Lisp Interface Manager - CLIM II Specificat... | Common Lisp - Myths and Legends | CDR 2: A generic hash table interface specificatio... | Common Lisp FAQ | Practical Common Lisp (video) | CAPI-OVERVIEW | Dynamic Learning Center | Functional Jobs | "A Survey of Current CLOS MOP Implementations" | Daniel Bobrow CLOS video from 87
View all (167 more)...
Nicolas Lamirault | Nikodemus Siivola | Sean D. Gillespie | Luigi Panzeri | Magnus Henoch | Stig Erik Sandø | Dave Watson | Alain Picard | Scott Williams | Emre Sevinc | Niklas Kambeitz | Robert Strandh | Marko Kocic | Daniel Dickison | Stewart M. Clamen | Chris Riesbeck | Daniel Lowe | Christian Queinnec | Jan Idzikowski | Erik Huelsmann | Thomas Atkins | Jeremy Smith | Björn Lindberg | Timothy Moore | Kevin Rosenberg | Juho Snellman | Sean E. O'Connor | Stelian Ionescu | Tim Bradshaw | Nathan Froyd
View all (281 more)...
International Lisp Conference 2009 | European Common Lisp Meeting 2008 | International Lisp Conference 2007 | 3rd European Lisp Workshop | LispNYC May 2006 Meeting - Peter Seibel | European Common Lisp Meeting 2006 | LispNYC November 2005 meeting | 2nd European LISP and Scheme Workshop | International Lisp Conference 2005 | European Common Lisp Meeting 2005 | 1st European Lisp and Scheme Workshop | International Lisp Conference 2003 | Japan Lisp User Group Meeting 2000
Open Genera | Scieneer Common Lisp | MCL | Ufasoft Common Lisp | CLForJava | Sacla | CMUCL | Steel Bank Common Lisp | LispWorks | Steel Bank Studio | Emacs Common Lisp (NaCL) | OpenMCL | CLISP | MusCL | Allegro Common Lisp | SBCL | Parrot Common Lisp | Golden Common Lisp | Liquid Common Lisp | Armed Bear Common Lisp
View all (8 more)...
The Common Lisp Gardeners | Lisp-it | Department of Computer Science and Engineering, FE... | Common-Lisp.biz | Debian | SISCOG - Sistemas Cognitivos Lta | CRISPYLOGICS | Ystok-Systema | Albuquerque Lisp and Scheme Users | Core Group | Consequor Consulting AG | HPC Platform | Cadence Design Systems GmbH | bese | Clozure Associates | RavenPack | HyperMeta Inc. | Ravenbrook | The Tech Co-op | Gothenburg Lisp User Group
View all (9 more)...
YstokCard - smart card API (PC/SC)(Modified)| YstokGrid(Modified)| CL-Typesetting(Modified)| CL-PDF(Modified)| Jean-Philippe Paradis(Added)| Armed Bear Common Lisp(Annotated)| Allegro Common Lisp(Modified)| CLFSWM - A(nother) Common Lisp FullScreen Window M...(Modified)| Ystok-URI - library for representing, parsing, and...(Added)| Clon, the Command-Line Options Nuker(Modified)| Common Lisp Reasoner(Modified)| ASDlite(Modified)| Ystok-Library(Modified)| Ystok-Local-Time(Modified)| YHTML-Template(Modified)| cl-gdata(Added)| Michael Filonenko(Added)| cl-portaudio(Added)| YstokHTML - collection of HTML generator, parser a...(Added)| Common Lisp User Space File System(Added)| cl-btree(Added)
View all last modified items...
(Notes are similar to comments but for adding precisions or facts)
You must be logged to add a note
You must be logged to add a comment
Comments and Notes Welcome
You can comment and/or add notes to all the tags and items of the Common Lisp Directory.
Submitted by admin 3390 days 18 hours 54 min ago
Comments added to RSS feeds
Now the commented items are added to the RSS feeds so that it's possible to see what have been commented recently.
Submitted by marc.battyani 3330 days 22 hours 44 min ago
unicode possibilities
It would be nice if cl-user would display unicode input for CJK text. With PCL and On Lisp getting new printings in Japan, it might be beneficial to attract more brain-share from that side of the world. Would this be considered out of cl-user.net's scope?
Submitted by charliemac 2426 days 16 hours 29 min ago
Perhaps it would be interesting to open up a "universities" category (or subcategory of Organization) to list universities that teach lisp?
Submitted by informatimago 2419 days 7 hours 46 min ago
re: universities
This is a good idea. Done.
Submitted by marc.battyani 2416 days 11 hours 59 min ago
re: unicode possibilities
I have to think about it.
The use of English as the standard language is prefered so that the information is useable by most lispers.
I don't mind some other language content though.
Submitted by marc.battyani 2416 days 11 hours 31 min ago
Change descriptions may accept invalid characters
Attempting to view the "Last Modified Items" RSS feed in the Google Chrome browser gives the error: "error on line 10 at column 2048: PCDATA invalid Char value 12". Adding the feed in Mozilla Thunderbird: "<feed URL> is not a valid feed."
Submitted by WalterGR 1543 days 18 hours 30 min ago
Repository access
Many sites have a link to view the repository such as:
Source code repository http://common-lisp.net/cgi-bin/viewcvs.cgi/?cvsroot=cl-semantic
however, I can't find any place that describes how to convert that link into something that can be used to actually check out the source code. I'm very suppressed this hasn't been an issue before.
Submitted by blake1024 1178 days 7 hours 11 min ago


text:
- Using Google Chrome: I would start Google Chrome and communicate with each tab using webSockets exposed by Chrome for remote debugging (debugging over wire). This way I can control each tab, load a new page and once the page is loaded I fetch the DOM of the loaded webpage.
- Using phantomjs: phantomjs uses webkit to load pages and provides a headless browsing option. As explained in the examples of phantomjs, I use page.open to open a new URL and then fetch the dom once the page is loaded by evaluating javascript on the page.
My goal is to crawl pages as fast as I can and if the page does not load in the first 10 seconds, declare it failed and move on. I understand that each page takes a while to load, so to increase the number of pages I load per second, I open many tabs in Chrome or start multiple parallel processes using phantomjs. The following is the performance that I observe:
- If I open more than 20 tabs in Chrome / 20 phantomjs instances, the CPU usage rockets up.
- Due to the high CPU usage, a lot of pages take more than 10seconds to load and hence I have a higher failure rate (~80% of page load requests failing)
- If I intend to keep the fails to less than 5% of the total requests, I cannot load more than 1 URL per second.
After trying out both the webkit based systems, it feels like the performance bottleneck is the webkit rendering engine and hence would like to understand from other users here, the number of URLs per second that I can expect to crawl. My hardware configuration is:
- Processor: Intel® Core™ i7-2635QM (1 processor, 4 cores)
- Graphics card: AMD Radeon HD 6490M (256MB)
- Memory: 4GB
- Network bandwidth is good enough to be able to load pages more than the performance that I am observing
The question I am trying to ask this mailing list is, does any one have experience using webkit for crawling web pages for a random set of URLs (say picking 10k URLs from twitter stream), how many URLs can I reasonably expect to crawl per second?

subset: commoncrawl