Packages, systems, modules, libraries — WHAT?

In my experience, at least once a week in the list of c.l.l or other Lisp-list, "rookies" are confused about what is associated with packages. Talking about the "boot" package "requirement" (requiring) package, surprised that after downloading the system need to use token bags etc. it irritates Me, I also think that this may be one reason why beginners think that using libraries in Lisp more complicated than it really is.

I usually stop trying to write useful explanation, and, naturally, this explanation is very simple. I created this page so next time just post here instead of again and again to explain the same thing.

First of all it is necessary to have a clear head. The term "package" is heavily overloaded. In Linux distributions like Debian or Gentoo is "packages", "packages" in programming languages Java, Perl or Python. It is likely that you came to Lisp with a preconceived opinion as to what "package" or it needs to be.

Packages

Package in Common Lisp is called a full-fledged element of the language, the semantics of which are clearly defined by the standard. Moreover, of all discussed on this page terms, this is the only one that has (in the context of Common Lisp) unambiguous definition. Packages are, strictly speaking, containers for characters. We can say that they are needed to help in the organization of separate name spaces in your programs.

In Common Lisp has functions and macros to create, modify, investigate, and remove packages. A very good introduction to packages (and symbols) can be found in Chapter 21 a wonderful book Practical Common Lisp Peter Seibel. The definition of the term is in Chapter 11 (online version) standard ANSI Common Lisp specification.

In General, about this all. Technically speaking, you are downloading packages. You can download (with the help of LOAD) code, which in turn will create the package, and that's a significant difference.

Also, if your Lisp complains that can not find a package that means the package as a Lisp object is not in the image (i.e. FIND-PACKAGE returns NIL), because it still has not been created. This does not mean that the Lisp machine had a look in the file system and found nothing. (A common reason for such failures is that events happen in the wrong order. More on that below.)

System, unlike packages, are not even mentioned in standard. However, experienced Lisp programmers know the term, because they potrebuetsya know and apply some tool of determination of systems. Most notable today ASDF (used by most Lisp libraries with open source code); another well-known tool of determination of systems much older ASDF — MK:DEFSYSTEM. Some developers also put their tools definition systems together with distributions, see, for example, Common Defsystem for LispWorks.

In this way the system is, strictly speaking, is a set of code plus the instructions for its processing, for example, dependencies on other systems that should be loaded/compiled in the first place, etc. in Other words, the tool of determination of systems its purpose is similar to make or Ant.

In addition, the determination tool systems are typically much more Common Defsystem may, for example, to integrate files of COM type libraries, ASDF is fully extensible and has been used, among other things, compile the files in C. It is also often used to define test kits describe the system.
Although ASDF and very popular, he is not omnipresent. It comes pre-installed with many Lisp systems like SBCL, OpenMCL or AllegroCL, it is likely that it will boot in other Lisp systems, but this fact does not make it a part of Common Lisp. This set of code with no explicit specifications, and with different versions that are incompatible with each other.
Go figure...

Modules

Standard specifies modules only superficially. There are two things you need to know about REQUIRE, PROVIDE and *MODULES* — this functionality is not recommended (deprecated), and depends on the implementation. Do not be concerned about the fact that this functionality is not recommended. All distributions today contain these functions, and the probability that there will be a new ANSI standard, and all implementations will suddenly put them away, of course, small. That's what we need to worry about is that REQUIRE may be convenient, but not a portable method (if you, of course, disturb the mechanisms of tolerance).

For example, in LispWorks you can use

the

(require "foreign-parser")

to download parser capable of reading the definition in C, but it doesn't work on OpenMCL. You can also call

the

(require :asdf)

to download ASDF on OpenMCL, but not in LispWorks.

Some distributions offer hooks to configure REQUIRE, and there are extensions like common-lisp-controller, connecting the REQUIRE with ASDF, but in the General case, the module is such a thing, which depends on the implementation and which should not be confused with the system (ASDF), and, especially with packages.

Libraries

Most likely you will not find a clear definition of what a library is. Most people think of it as a collection of code designed to perform one or more specific tasks and is distributed as a single unit, usually in the form of a compressed archive, which is available somewhere to download. In fact, this vague definition is, I think, the most appropriate when talking about programs written in Lisp. Most Lisp libraries today include the determination of (ASDF) of the system, but it is not necessary. It is possible, depending on the production method, it will be a module in your Lisp system, but it is also not necessary. In addition, the library typically defines one or more packages and might not determine a single one.

And, by agreement, or because of a lack of imagination, may give and often is the situation when the library "Ku" comes with the definition of "Ku", which can be downloaded as a module "Ku". After downloading the code, you will receive a new package, called "Ku". Four different entities with the same name! I admit that it is confusing, but I hope that the previous few paragraphs have helped to clarify the situation slightly.

But I have still nothing works!

Often people complain that they can't compile a file containing code like this:

the

;; this line can also be written (require :cl-ppcre)
(asdf:oos 'asdf:load-op :cl-ppcre)

(defun my-simple-number-scanner (string)
(cl-ppcre:scan "^[0-9]+$" string))

Why? Why can I load this file but cannot compile it? And why I can compile it after downloading? Is not it strange?

No, not strange. The compiler reads the first form (which is an instruction to compile — if necessary — and load-system CL-PPCRE, but not to implement it. In the end, the compiler is only interested in compiling the code. After the first form it goes to the second form, the definition of this function. It is possible to error message because Lisp-scanner trying to read this form, detects the character sequence "cl-ppcre:scan", which should indicate a external symbol from package CL-PPCRE, but the package CL-PPCRE yet. In the process of loading system CL-PPCRE, among other things, create a package CL-PPCRE, but it hasn't happened yet. Read Chapter 3 CLHS.
You can use EVAL-WHEN to tell the compiler to load CL-PPCRE before reading the second form. However, you should find another way of organizing your code. The first form is simply an announcement that your code depends on the system of CL-PPCRE. This must not be in the same file as Lisp code. Write a definition for your program and place the subject there.

Article based on information from habrahabr.ru