Keyword arguments and compiler macros

[Juan Jose Garcia-Ripoll]

[Sorry for possible duplicates]

I just realized that compiler macros might be too restrictive in some
implementations. Take for instance

(define-compiler-macro foo (a &key key-arg) ...)

This is assuming that FOO is only going to be invoked with _constant_
keyword arguments. I know, I know, most uses will have always one argument
or :KEY-ARG, but suppose you want to call (FOO some-value
a-non-constant-expression some-other-value)... What is the compiler to do?

I _now_ think that the compiler macro processor should simply recognize the
failure to parse the form and return the original form, unprocessed, but
the implementations I have at hand do not do that. ECL to begin with, but
also CCL and SBCL.

Any opinions on this?

Juanjo

[Scott L. Burson]

Content preview: On Thu, Dec 1, 2011 at 1:12 PM, Juan Jose Garcia-Ripoll <juanjose.garciaripoll-gM/Ye1E23mwN+***@public.gmane.org>
wrote: > I _now_ think that the compiler macro processor should simply recognize
the > failure to parse the form and return the original form, unprocessed
[...]

Content analysis details: (-100.7 points, 5.0 required)

pts rule name description
---- ---------------------- --------------------------------------------------
-100 USER_IN_WHITELIST From: address is in the user's white-list
-0.7 RCVD_IN_DNSWL_LOW RBL: Sender listed at http://www.dnswl.org/, low
trust
[209.85.220.179 listed in list.dnswl.org]
-0.0 SPF_PASS SPF: sender matches SPF record
Archived-At: <http://permalink.gmane.org/gmane.lisp.cl-pro/571>

On Thu, Dec 1, 2011 at 1:12 PM, Juan Jose Garcia-Ripoll

Post by Juan Jose Garcia-Ripoll
I _now_ think that the compiler macro processor should simply recognize the
failure to parse the form and return the original form, unprocessed

I agree. Otherwise the compiler macro writer has to use &rest and
parse the argument list themselves; what's the point of making them do
that?

-- Scott

[Nikodemus Siivola]

Content preview: On 1 December 2011 23:12, Juan Jose Garcia-Ripoll > I _now_
think that the compiler macro processor should simply recognize the > failure
to parse the form and return the original form, unprocessed I'm not convinced.
[...]

Content analysis details: (-100.7 points, 5.0 required)

pts rule name description
---- ---------------------- --------------------------------------------------
-0.7 RCVD_IN_DNSWL_LOW RBL: Sender listed at http://www.dnswl.org/, low
trust
[209.85.210.179 listed in list.dnswl.org]
-100 USER_IN_WHITELIST From: address is in the user's white-list
Archived-At: <http://permalink.gmane.org/gmane.lisp.cl-pro/572>

On 1 December 2011 23:12, Juan Jose Garcia-Ripoll

Post by Juan Jose Garcia-Ripoll
I _now_ think that the compiler macro processor should simply recognize the
failure to parse the form and return the original form, unprocessed

I'm not convinced.

Firstly, the (admittedly non-normative) final example in the
DEFINE-COMPILER-MACRO dictionary entry shows that this isn't exactly
traditional or something writers of portable code can expect.

The same example also, as a practical consideration to those trying to
write portable code, shows how using &REST and &ALLOW-OTHER-KEYS makes
writing such compiler macros possible.

However, the possibility of non-constant keywords is not the only
reason it is tricky. Consider this:

(defun foo (&key a b c) ...)

Let's assume that FOO can be implemented quite efficiently if you know
A comes from BAR. In addition to dealing with constant arguments,
that's one of the major use-cases for compiler-macros after all.

(define-compiler-macro foo (&whole form &key a b)
(if (can-optimize-p a b)
(optimize a b)
form))

Now let's take a look at a call-site:

(foo :b (incf i) :a (bar i))

Unless CAN-OPTIMIZE-P recognizes that B may have side-effects and
therefor returns NIL, the compiler-macro has a bug. This means that as
a practical matter compiler-macro writers dealing with keywords
virtually /always/ need to include &REST in their lambda-lists, and
use it to preserve the order of evaluation.

...which in turn means that if even if an implementation makes
compiler-macros fire only when all keywords specified are constant
ones, compiler-macro writers /still/ need to look at &REST.

Relying on the implementation to provide sufficient rebinding to
preserve the order of side-effects would lead to unportable code with
subtle bugs -- the worst kind.

Cheers,

-- Nikodemus

[Scott L. Burson]

Content preview: On Fri, Dec 2, 2011 at 12:28 AM, Nikodemus Siivola <nikodemus-/W93PBX4W+***@public.gmane.org>
wrote: > On 1 December 2011 23:12, Juan Jose Garcia-Ripoll > >> I _now_ think
that the compiler macro processor should simply recognize the >> failure
to parse the form and return the original form, unprocessed > > I'm not convinced.

Post by Nikodemus Siivola

Firstly, the (admittedly non-normative) final example in the > DEFINE-COMPILER-MACRO

dictionary entry shows that this isn't exactly > traditional or something
writers of portable code can expect. [...]

Content analysis details: (-100.7 points, 5.0 required)

pts rule name description
---- ---------------------- --------------------------------------------------
-0.7 RCVD_IN_DNSWL_LOW RBL: Sender listed at http://www.dnswl.org/, low
trust
[209.85.212.51 listed in list.dnswl.org]
-100 USER_IN_WHITELIST From: address is in the user's white-list
-0.0 SPF_PASS SPF: sender matches SPF record
Archived-At: <http://permalink.gmane.org/gmane.lisp.cl-pro/573>

On Fri, Dec 2, 2011 at 12:28 AM, Nikodemus Siivola

Post by Nikodemus Siivola
On 1 December 2011 23:12, Juan Jose Garcia-Ripoll

I _now_ think that the compiler macro processor should simply recognize the
failure to parse the form and return the original form, unprocessed

I'm not convinced.
Firstly, the (admittedly non-normative) final example in the
DEFINE-COMPILER-MACRO dictionary entry shows that this isn't exactly
traditional or something writers of portable code can expect.

Ah, I should have looked at that example.

Still, your argument, though it is a good one, applies somewhat
obliquely to Juanjo's question. If our purpose is to make incorrect
compiler macros show up early, seems to me, DEFINE-COMPILER-MACRO
itself should signal an error (or at least, warn) if the lambda list
uses &KEY but not also &REST. This is going to catch a far larger
fraction of incorrect complier macros than waiting for the almost
vanishingly rare case in which a nonconstant key is used. (Compiler
macros are not invoked on APPLY forms; see 3.2.2.1.1.)

To put it another way: by your argument, I think the answer to
Juanjo's question doesn't matter very much. While it's true that a
compiler macro that could get a keyword parsing error must be wrong,
waiting for the parsing error to actually occur is not a very
effective way of detecting this class of mistake.

There is another way that a compiler macro's keyword parsing could
fail, that seems to me to be much more likely than the use of a
nonconstant key: where a function and a compiler macro for it are
written, and then some time later, a new keyword is added to the
function but the need to do likewise with the compiler macro is
overlooked. Although one could argue that getting an error when
compiling a form with the new keyword is desirable, as it reminds one
to fix the compiler macro, I think the nature of compiler macros is
that it's nice when they work but one isn't necessarily counting on
them to work, and so ignoring the error is often preferable.

And then, of course, there's the point that a keyword parsing error is
not the only kind of error a compiler macro could signal; by the
argument I just made, one might well want to ignore all of them.

I guess what I'm muddling towards here is that there should be a
compiler parameter *IGNORE-COMPILER-MACRO-ERRORS* that users can set
to their preference.

-- Scott

[Juan Jose Garcia-Ripoll]

Post by Scott L. Burson
To put it another way: by your argument, I think the answer to
Juanjo's question doesn't matter very much.

I think it does. What Nikodemus is arguing *is orthogonal to what I am
saying*. One thing is whether keyword argument parsing of compiler macros
is insuficient and whether one has to use &rest and ensure that the order
is ok. I am fine with it.

A different issue is the utility of &key. I do not buy the argument that
&key should always be used with &allow... in compiler macros. This forfeits
any use of &key at all because the values are _never_ going to be trusted
at all: one of the arguments might be a variable name and the parser will
confuse the user about expectations.

More precisely, if I write

(define-compiler-macro foo (&rest my-args &key some-key-arg
&allow-other-keys) ...)

and write (foo :some-key-arg some-value some-variable some-other-value) the
parser is going to produce a useless value in some-key-arg. That means I
have to do my own parsing of my-args to see whether the keyword arguments
are to be trusted.

If on the other hand this definition

(define-compiler-macro foo (&rest my-args &key some-key-arg) ...)

implies that the compiler macro will refuse parsing (without an error) when
some argument is not an allowed keyword, then this actually has some
utility.

Regarding the issue of argument ordering and other stuff, there are many
cases where I want to write compiler macros for the case in which the
arguments are simple forms. This case is very simple to check and means
that &key arguments have a value by themselves.

Juanjo

[Nikodemus Siivola]

On 2 December 2011 21:42, Juan Jose Garcia-Ripoll

Post by Juan Jose Garcia-Ripoll
A different issue is the utility of &key. I do not buy the argument that
&key should always be used with &allow... in compiler macros. This forfeits
any use of &key at all because the values are _never_ going to be trusted at
all: one of the arguments might be a variable name and the parser will
confuse the user about expectations.

Well, they /can/ be used to provide defaults. (Not that that's so
useful since you /still/ need to walk &REST, but...)

Post by Juan Jose Garcia-Ripoll
If on the other hand this definition
(define-compiler-macro foo (&rest my-args &key some-key-arg) ...)
implies that the compiler macro will refuse parsing (without an error) when
some argument is not an allowed keyword, then this actually has some
utility.

Point taken.

Also, I just found a hilarious --apparently universal-- bug in this area:

(defun foo (&key ((a ax) t)) (format nil "a=~S" ax))

(define-compiler-macro foo (&key ((a ax) t)) (format nil "a=~S" ax))

(compile nil `(lambda () (foo 'a 42))) =| ERROR, unknown keyword 'A

(funcall (compile nil `(lambda () (foo a 42)))) => "a=42"

SBCL, CCL, Clisp, and Lispworks all share this beauty.

Now, fixing this (at least in SBCL) is simple enough, but raises the
question of what to do about non-constant form where a keyword
argument is expected. Complaining about "unknown keyword argument FOO"
when FOO is actually a variable that may evaluate to a perfectly valid
keyword at runtime seems a bit suspect.

So, on reflection, I've made a 180 degree turnabout and now think that
declining to expand compiler-macros with unknown or variable keyword
arguments is The Right Thing to do unless &ALLOW-OTHER-KEYS is
explicitly specified.

Cheers,

-- nikodemus

[Nikodemus Siivola]

Post by Nikodemus Siivola
(define-compiler-macro foo (&key ((a ax) t)) (format nil "a=~S" ax))

Just ignore me. I need more coffee.

The universal behaviour is correct: a compiler-macro lambda-list is a
/macro/ lambda list, after all.

Interestingly enough, that also means that there is no ways to use
&KEY to specify non-keyword keyword arguments in a compiler-macro
lambda-list that doesn't run roughshod across the evaluation model.

Cheers,

 -- Nikodemus

[Steve Haflich]

Post by Nikodemus Siivola
Interestingly enough, that also means that there is no ways to use
&KEY to specify non-keyword keyword arguments in a compiler-macro
lambda-list that doesn't run roughshod across the evaluation model.

I think you mean "non-constant" rather than "keyword". There is nothing
special about keyword (which is a constant) and any other constant that
eventually evaluates to a symbol.

The real open issuse behind compiler-macros are deeper.

Back in 1989 JonL White (then of Lucid) and I (then and still of Franz)
threatened to hold our breath until we turned blue at an X3J13 meeting
unless we were permitted to enter a proposal for compiler macros. We
worked overnight (I think it was at JonL's house in Palo Alto) to come up
with a proposal. It was accepted, subject to a few later emendations, and
that is what is in the ANS. Unfortunately, there was not time to give the
proposal the traditional trial-by-implementation-and-use, so
compiler-macros are not defined completely enough for effective portable
use. (But I'm nonetheless glad they are there.)

The ANS is silent on what happens if a compiler-macro signals an error --
not just an error from an unmatchable "keyword" argument. I suggest that
the compiler _should_ have been specified to wrap an error handler around
compiler-macro expansion, and if any error occurs, the expansion should be
treated as ther equivalent of returning &whole.

But even this is not sufficient. The ANS allows a compiler to ignore
compiler-macros altogether, or whenever it likes. This provision was
necessary to get compiler macros into the standard (to avoid imposing new
features on implementations unwilling or unable to conform) but this makes
the facility portably unusable. Now, many uses of compiler macros presume
nonportable dependencies, such as translating certain references into
nonportable lower-level accessors, but IMO it should be possible somehow
for portable code to assume that a compiler macro _will_ be expanded in
compiled portable code. There ought be some sort of mechanism so the
programmer can determine that a compiler macro has signalled error, or that
a compiler macro has returned &whole. (Those if us who use compiler-macros
currentyl often use disassemble.) I have no proposal how to implement such
facilities -- they would live too close to the real estate owned by the
programming environment.

[Jean-Philippe Paradis]

Content preview: "The ANS allows a compiler to ignore compiler-macros altogether,
or whenever it likes. This provision was necessary to get compiler macros
into the standard (to avoid imposing new features on implementations unwilling
or unable to conform) but this makes the facility portably unusable." [...]


Content analysis details: (-0.7 points, 5.0 required)

pts rule name description
---- ---------------------- --------------------------------------------------
-0.7 RCVD_IN_DNSWL_LOW RBL: Sender listed at http://www.dnswl.org/, low
trust
[209.85.210.51 listed in list.dnswl.org]
0.0 FREEMAIL_FROM Sender email is commonly abused enduser mail provider
(hexstream[at]gmail.com)
-0.0 SPF_PASS SPF: sender matches SPF record
0.0 T_DKIM_INVALID DKIM-Signature header exists but is not valid
Archived-At: <http://permalink.gmane.org/gmane.lisp.cl-pro/588>

"The ANS allows a compiler to ignore compiler-macros altogether, or
whenever it likes. This provision was necessary to get compiler
macros into the standard (to avoid imposing new features on
implementations unwilling or unable to conform) but this makes the
facility portably unusable."

Wouldn't it have been desirable to specify that no implementation is
required to honor compiler-macros, but if an implementation does honor
compiler-macros sometimes, then it must always support them on a
best-effort basis? By "on a best-effort basis" I mean that if there
are some corner cases in specific calls for which a compiler-macro
might be applicable but the semantics are somehow non-obvious or if
there are specific implementation-specific problems with supporting
that case, then only then could the implementation not honor the
compiler-macro.

Oh, and maybe specify that setting (optimize (space 0)) prevents
expansion of compiler-macros...

Post by Nikodemus Siivola
Interestingly enough, that also means that there is no ways to use
&KEY to specify non-keyword keyword arguments in a compiler-macro
lambda-list that doesn't run roughshod across the evaluation model.

I think you mean "non-constant" rather than "keyword".  There is nothing
special about keyword (which is a constant) and any other constant that
eventually evaluates to a symbol.
The real open issuse behind compiler-macros are deeper.
Back in 1989 JonL White (then of Lucid) and I (then and still of Franz)
threatened to hold our breath until we turned blue at an X3J13 meeting
unless we were permitted to enter a proposal for compiler macros.  We worked
overnight (I think it was at JonL's house in Palo Alto) to come up with a
proposal.  It was accepted, subject to a few later emendations, and that is
what is in the ANS.  Unfortunately, there was not time to give the proposal
the traditional trial-by-implementation-and-use, so compiler-macros are not
defined completely enough for effective portable use.  (But I'm nonetheless
glad they are there.)
The ANS is silent on what happens if a compiler-macro signals an error --
not just an error from an unmatchable "keyword" argument.  I suggest that
the compiler _should_ have been specified to wrap an error handler around
compiler-macro expansion, and if any error occurs, the expansion should be
treated as ther equivalent of returning &whole.
But even this is not sufficient.  The ANS allows a compiler to ignore
compiler-macros altogether, or whenever it likes.  This provision was
necessary to get compiler macros into the standard (to avoid imposing new
features on implementations unwilling or unable to conform) but this makes
the facility portably unusable.  Now, many uses of compiler macros presume
nonportable dependencies, such as translating certain references into
nonportable lower-level accessors, but IMO it should be possible somehow for
portable code to assume that a compiler macro _will_ be expanded in compiled
portable code.  There ought be some sort of mechanism so the programmer can
determine that a compiler macro has signalled error, or that a compiler
macro has returned &whole.  (Those if us who use compiler-macros currentyl
often use disassemble.)  I have no proposal how to implement such facilities
-- they would live too close to the real estate owned by the programming
environment.
_______________________________________________
pro mailing list
http://lists.common-lisp.net/cgi-bin/mailman/listinfo/pro