Common Lisp/External libraries/ASDF/Budden's infrequently asked questions

Budden's infrequently asked questions about ASDF

This information is non-official, not coming from asdf maintainters, highly opinionated, can duplicate some parts of manual, be incorrect, or be formulated in bad English.

What does "upward" and "downward" mean?

I don't know. According to manual, "operation propagates downward" means that asdf first does operation for components of system and then for system itself. "operation propagates upward" means that operation if first performed on the system and than on its components. But this seem to be not relevant for dependencies of system and this is the problem for me.

How do I make portable lisp program with its own fasl cache in a given directory?

(from asdf-devel mailing list) https://mailman.common-lisp.net/pipermail/asdf-devel/2015-October/004962.html

;; this is a fragment of my sbclrc (user-level lisp initialization script)
;; note we rely on the fact that we have no :uiop loaded
;; if :uiop is present in the image at the beginning, this answer won't help you
(assert (null (find-package :uiop)))
(require :uiop)
(defconstant *clcon-uiop-user-cache-override* #P"c:/clcon/fasl-cache/asdf/")
(setf uiop:*user-cache* *clcon-uiop-user-cache-override*)
(require 'asdf)
(defun check-output-translations-ok ()
  "Call it now and once again at the end of loading to ensure that fasls are placed to a right place"
  (assert (equalp (asdf:apply-output-translations "c:/aaa.bbb")
    (merge-pathnames "c/aaa.bbb" *clcon-uiop-user-cache-override*))))
(check-output-translations-ok)

#-quicklisp
(let ((quicklisp-init "c:/clcon/quicklisp/setup.lisp"
                                       ))
  (when (probe-file quicklisp-init)
    (load quicklisp-init))) 

; swank is handled separatedly (maybe irrelevant to asdf, but relevant
; to task of creating "portable file releases"
(load (merge-pathnames "swank-loader.lisp" (ql:where-is-system :swank)))
(setq swank-loader::*fasl-directory* "c:/clcon/fasl-cache/swank/")
(swank-loader:init)

;;; Here I load all my systems

; after loading all our code check that no one smashed the variable.
; this is only a limited protection, there is no protection against
; binding
(check-output-translations-ok)

Note: this was not recommended by asdf maintainers, but for me it just did a job.

:second system depends on :first system, both are loaded into image. I have changed a file belonging to :first system. Does (asdf:load-system :first) reload :second?

As my experiment showed, no (at asdf 3.1.5). Manual states: "But for code that you are actively developing, debugging, or otherwise modifying, you should use load-system, so ASDF will pick on your modifications and transitively re-build the modified files and everything that depends on them." But this only true for the system which name you pass to load-system.

I want my file to be loaded, not compiled. What to do?

1. Read the statement in the asdf FAQ.

2. If you are still not convinced, try the following:

;; Contents of s1.asd:
(defsystem :s1
  :components 
  ((:static-file "file-to-load.lisp")
   (:file "loader" :depends-on ("file-to-load.lisp"))))
;; EOF s1.asd 
;; ------------------------------
;; Contents of file-to-load.lisp
(eval-when (:compile-toplevel :load-toplevel) (error "I must be loaded as a source"))
(eval-when (:execute) (print "file-to-load.lisp is loading as a source")) 
;; EOF file-to-load.lisp
;; ------------------------------
;; Contents of loader.lisp
(load (merge-pathnames "file-to-load.lisp" #.*compile-file-pathname*))
;; EOF loader.lisp

3. Another approach is my asdf-load-source-cl-file (tested at SBCL and asdf 3.1.5). Here is an example system definition from the sources:

(asdf:defsystem :asdf-load-source-cl-file-example-system
  :defsystem-depends-on (:asdf-load-source-cl-file)
  :serial t
  :components
  ((:load-source-cl-file "file-to-load")
   ))

I want to delete all fasl files. How do I find them?

This might help:

(asdf:apply-output-translations (asdf:system-source-directory (asdf:find-system :s1)))

Which systems are currently loaded in the image?

asdf:already-loaded-systems is a function which returns names of loaded systems. It is almost documented in the manual. Also you can read the value of asdf::*defined-systems* . Note this is unexported variable.

How do I "make clean" on asdf system?

There is no way to do just make clean. Loading system has side effects on lisp image. E.g.

packages can be created
methods on generic functions can be defined
existing functions can be modified
variables can be modified (e.g. *features*)

So, think "uninstall" instead of "must clean". To implement "uninstall", we must have kept track of all changes made during install, but I don't know if someone even tried to implement it.

If you want just to delete all fasls, look at arnesi for the beginning. Code is written in 2005, so it can be outdated. I didn't test it. Quote from asdf.lisp follows

(defclass clean-op (asdf:operation)
  ((for-op :accessor for-op :initarg :for-op :initform 'asdf:compile-op))
  (:documentation "Removes any files generated by an asdf component."))

(defmethod asdf:perform ((op clean-op) (c asdf:component))
  "Delete all the output files generated by the component C."
  (dolist (f (asdf:output-files (make-instance (for-op op)) c))
    (when (probe-file f)
      (delete-file f))))

(defmethod asdf:operation-done-p ((op clean-op) (c asdf:component))
  "Returns T when the output-files of (for-op OP) C don't exist."
  (dolist (f (asdf:output-files (make-instance (for-op op)) c))
    (when (probe-file f) (return-from asdf:operation-done-p nil)))
  t)

What about asdf:run-program + SBCL + Windows ?

I don't know right answer. First of all, running a program did never work well for me in SBCL for Windows. This was especially true when I called SBCL from EMACS. I posted a bug to SBCL launchpad, but it was ignored so far. Maybe I didn't read manuals correctly, but I believe that there are bugs in fact. So I wrote a small Delphi program CallBatFromGuiDetached.exe which have a following features:

it can run another program, console or GUI, and either wait for completion or not
it can run it in a specified directory
it can redirect stout and stderr to files
exit code can be extracted

You can still have problems with character encodings. Feel free to fix em. Code to run program is the following:

(defun call-bat (bat args &key (wait t) redirect-output directory)
  (let ((arglist nil))
    (check-type redirect-output (or null string))
    (when redirect-output
      (setf arglist (append arglist `("-s" ,redirect-output))))
    (setf arglist (append arglist (list bat) args))
    (apply #'sb-ext:run-program
           "c:/clcon/bin/util/CallBatFromGuiDetached.exe"
           arglist
           :wait wait
           (if directory (list :directory directory))
           )))

Source code and binary of CallBatFromGuiDetached.exe are included into clcon distribution. Some messages and comments are in Russian. Examples:

Start notepad.exe and not wait:

(call-bat "notepad.exe" () :wait nil)

Start notepad.exe and wait:

(call-bat "notepad.exe" ())

Start non-existing command and extract error code:

(sb-impl::process-exit-code (call-bat "nonexisting_command" ()))

Start dir in c:\tmp and redirect output to files:

(call-bat "cmd" '("/c" "dir" "/b" "/s") :redirect-output "outputs" :directory "c:/tmp")

stdout and stderr will be saved to c:\tmp\outputs.stdError.txt and c:\tmp\outputs.stdOutput.txt.

Where are good examples of defining custom component classes?

I don't know about good, but asdf manual recommends CFFI-GROVEL first of all. When you load-system where you have :grovel-file component, the component is processed by asdf as follows:

1. Special lisp function, cffi-grovel:process-grovel-file is called on file X which name is given as component's name. Result is written to a lisp file Y.

2. File Y is being compiled and loaded as any other usual lisp file.

cffi-grovel's asdf.lisp demonstrates:

how to create new component class
how to deal with asdf versions
how to register class so that it can be used as an asdf component type
how to define methods

To dig into more detail, let's look at the

(defmethod component-depends-on ((op compile-op) (c process-op-input))
  `((process-op ,c) ,@(call-next-method)))

The code states that to do compile-op (compiling a source), we should do process-op first. input-files method for compile-op is redefined so that produced lisp file is compiled instead of original file:

(defmethod input-files ((op compile-op) (c process-op-input))
  (list (first (output-files 'process-op c))))

Methods for process-op are defined so that it calls groveller.

Manual states that it is enough to define new class of the component and it will be available for using in systems. In what package must the component class name reside?

In grovel's asdf.lisp we see the code:

(in-package :cffi-grovel)
...
;; Allow for naked :grovel-file ... in asdf definitions.
(setf (find-class 'asdf::cffi-grovel-file) (find-class 'grovel-file))

After reading source of UIOP/UTILITY:COERCE-CLASS I concluded that commentary is incorrect and should be:

;; Allow for naked :cffi-grovel-file ... in asdf definitions.

So, we can specify component as

(:cffi-grovel-file "bububub")

or

(cffi-grovel:grovel-file "bububub")