Compiler APIs in Common Lisp

[Matthew Swank]

I am current reading an interesting paper on Racket's macro system (referenced
here: http://lambda-the-ultimate.org/node/4196
), and some of the objects they expose quack like CLTL-2 Environments:
http://www.cs.cmu.edu/Groups/AI/html/cltl/clm/node102.html. I have read that
having write access to environments makes it hard for the compiler to optimize
code. However, even portable read only access would be helpful. Does anyone
besides me and Cyrus Harmon (http://cyrusharmon.org/blog/display?id=113)
miss Environments?

Matt

[Mark Cox]

Post by Matthew Swank
I am current reading an interesting paper on Racket's macro system (referenced
here: http://lambda-the-ultimate.org/node/4196
http://www.cs.cmu.edu/Groups/AI/html/cltl/clm/node102.html. I have read that
having write access to environments makes it hard for the compiler to optimize
code. However, even portable read only access would be helpful. Does anyone
besides me and Cyrus Harmon (http://cyrusharmon.org/blog/display?id=113)
miss Environments?

Alas, I wish I had been exposed to them before. Until Cyrus' post I had no knowledge of them. I thank Cyrus for making that post. Using the function CLTL2 VARIABLE-INFORMATION function, I produced the following higher level functionality as a proof of concept for an idea I have for a mathematical optimisation library.

(type-dispatching-defun matrix-add (y x scalar)
(assert (= (array-total-size y)
(array-total-size x)))
(dotimes (index (array-total-size x))
(declare (type fixnum index))
(setf (row-major-aref y index)
(+ (* scalar (row-major-aref x index))
(row-major-aref y index))))
y)

(add-type-specific-function 'matrix-add
'((simple-array single-float (*))
(simple-array single-float (*))
single-float))

(defun time-generic ()
(let ((a (make-array 1000 :initial-element 1))
(b (make-array 1000 :initial-element 2)))
(time (dotimes (i 100000)
(matrix-add a b 2.0)))))

(defun time-single-float-array ()
(let ((a (make-array 1000 :initial-element 1.0 :element-type 'single-float))
(b (make-array 1000 :initial-element 2.0 :element-type 'single-float)))
(declare (type (simple-array single-float (*)) a b))
(time (dotimes (i 100000)
(matrix-add a b 2.0)))))


TESTS> (time-generic)
Evaluation took:
8.004 seconds of real time
8.000982 seconds of total run time (7.997856 user, 0.003126 system)
99.96% CPU
15,929,157,645 processor cycles
0 bytes consed

TESTS> (time-single-float-array)
Evaluation took:
0.430 seconds of real time
0.429092 seconds of total run time (0.428791 user, 0.000301 system)
99.77% CPU
854,425,193 processor cycles
0 bytes consed

Given the possibilities it offers, I am interested to know why it isn't part of the standard.

Thanks for the Racket links, I will have a read.

Mark

Post by Matthew Swank
Matt
_______________________________________________
pro mailing list
http://common-lisp.net/cgi-bin/mailman/listinfo/pro

[Pascal Costanza]

Post by Mark Cox

Post by Matthew Swank
I am current reading an interesting paper on Racket's macro system (referenced
here: http://lambda-the-ultimate.org/node/4196
http://www.cs.cmu.edu/Groups/AI/html/cltl/clm/node102.html. I have read that
having write access to environments makes it hard for the compiler to optimize
code. However, even portable read only access would be helpful. Does anyone
besides me and Cyrus Harmon (http://cyrusharmon.org/blog/display?id=113)
miss Environments?

Alas, I wish I had been exposed to them before. Until Cyrus' post I had no knowledge of them. I thank Cyrus for making that post. Using the function CLTL2 VARIABLE-INFORMATION function, I produced the following higher level functionality as a proof of concept for an idea I have for a mathematical optimisation library.

[...]

Post by Mark Cox
Given the possibilities it offers, I am interested to know why it isn't part of the standard.

There are several possible answers to this. Ideally, you would want a data structure that is based either on structs or on CLOS classes that gives you information about the lexical environment. However, the compiler must exist already at a very early phase of bootstrapping a Lisp system, and the more advanced types are probably not ready yet at that stage. Secondly, there are quite a few namespaces (not just 2) that all need to handled in some way in lexical environments. Ensuring a good API to inspect all of them was apparently not trivial. Thirdly, different compilers have different ways how to organize their compilation phases - making them all agree on how to provide the information is maybe also not trivial.

And then, from a different angle, Common Lisp already provides environment objects and local macro definitions, and they already allow you to go a very long way. For example, I have found the discussion at http://www.lispworks.com/documentation/HyperSpec/Issues/iss066_w.htm very illuminating. (See the two local-type-declare examples - I particularly like the version using symbol macros.)

I have implemented an experimental hygienic-compatible macro system for Common Lisp using just these ingredients. See http://dx.doi.org/10.3217/jucs-016-02-0271 for a description.

I would strongly recommend considering using such techniques, instead of holding your breath and waiting for wide adoption of fully specified lexical environments. ;) (Just to avoid misunderstandings: such environments would definitely be more expressive than just opaque environment objects + local macros, it's just that environment objects + local macros are already a lot more powerful than some may expect!)


Pascal

--
Pascal Costanza
The views expressed in this email are my own, and not those of my employer.

[Matthew D. Swank]

Post by Pascal Costanza
And then, from a different angle, Common Lisp already provides environment objects and local macro definitions, and they already allow you to go a very long way.

Yes, they do, but since the *macrolet forms aren't top level it's harder
to get good ide (well, slime) support for debugging them.

For example, one technique I use when building a compile time language
is to have all my context sensitive operators defined as top level
macros that initially signal errors. Each operator gets redefined by
macrolet when the expansion is in a context where the operation is
valid. A simple example is here:
http://paste.lisp.org/+2LPI

You need macroexpand-all to get the proper expansion of a redefined
(actually shadowed) operator.

However, perhaps this is a quibble related to my use of stone tools for
Lisp development :).

Matt

[Nicolas Neuss]

Post by Matthew Swank
I am current reading an interesting paper on Racket's macro system (referenced
here: http://lambda-the-ultimate.org/node/4196
http://www.cs.cmu.edu/Groups/AI/html/cltl/clm/node102.html. I have read that
having write access to environments makes it hard for the compiler to optimize
code. However, even portable read only access would be helpful. Does anyone
besides me and Cyrus Harmon (http://cyrusharmon.org/blog/display?id=113)
miss Environments?
Matt

At least I have missed environments at some point.

I think that Franz has put the most work in this direction, see

http://www.franz.com/support/documentation/current/doc/environments.htm#envir-general-1

And if I remember posts by Duane Rettig on comp.lang.lisp right, they
would be quite willing to cooperate on this issue with other
implementors.

Nicolas

[Matthew Mondor]

On Mon, 11 Apr 2011 21:45:36 +0000 (UTC)

Post by Matthew Swank
I am current reading an interesting paper on Racket's macro system (referenced
here: http://lambda-the-ultimate.org/node/4196
http://www.cs.cmu.edu/Groups/AI/html/cltl/clm/node102.html. I have read that
having write access to environments makes it hard for the compiler to optimize
code. However, even portable read only access would be helpful. Does anyone
besides me and Cyrus Harmon (http://cyrusharmon.org/blog/display?id=113)
miss Environments?

I just wanted to say that the resulting thread is most precious and
worthy of archiving; I was also occasionally confronted to various
compilation-environment related issues and so far the provided answers
and links seem to touch most, if not all of the topic:

- Custom lexically scoped SETF/GETF expansions built on macros, using
lexical macros overriding the top-level macros
- Lexically scoped symbol-macrolet using lexically scoped macrolet
- The ENV parameter of the various special forms, where available, and
implementation-dependent related features
- Custom declarations and implementation-dependent ways to access their
current values in the current scope
- The COMPILER-LET issue

Great thanks,

[Matthew Swank]

I was also occasionally confronted to various compilation-environment related
issues and so far the provided answers and links seem to touch most, if not

...

- Lexically scoped symbol-macrolet using lexically scoped macrolet
- The ENV parameter of the various special forms, where available, and
implementation-dependent related features

...

- The COMPILER-LET issue

I that spirit here is a primitive compiler-let (called compile-let-1) that uses
some of those tools: http://paste.lisp.org/display/121480