macro returning more than one form

[Paul Tarvydas]

What is the best way to write a macro that returns more than one form to
the top level? E.g.

(in-package :xxx)
(defun ...)

I've been wrapping a progn around the result, but LW doesn't like it
very much, and SBCL seems to hate it.

Thanks
pt

[Alessio Stalla]

At the top level, PROGN is treated as if each contained expression appeared
at the top level. So generally wrapping a PROGN around the result is
precisely what you need to do.

However, since I see an IN-PACKAGE form in your example... beware that that
IN-PACKAGE form will NOT affect the following DEFUN, because the symbols in
it will have been read long before the IN-PACKAGE is executed.

Post by Paul Tarvydas
What is the best way to write a macro that returns more than one form to
the top level? E.g.
(in-package :xxx)
(defun ...)
I've been wrapping a progn around the result, but LW doesn't like it very
much, and SBCL seems to hate it.
Thanks
pt

[Kenneth Tilton]

Post by Alessio Stalla
At the top level, PROGN is treated as if each contained expression
appeared at the top level. So generally wrapping a PROGN around the result
is precisely what you need to do.
However, since I see an IN-PACKAGE form in your example... beware that
that IN-PACKAGE form will NOT affect the following DEFUN, because the
symbols in it will have been read long before the IN-PACKAGE is executed.

Yikes, I read too fast, did not see the in-package. Well, the "doing
something else wrong" suggestion stands. :) But this works in Allegro CL:

(defmacro go4it (n)
`(progn
(in-package :mcna.db)
(defun forty2 () ,n)))

(go4it 42)

(print (forty2))

I put this in a source file in a library source with a different package,
in a Lisp session where :mcna.db was already a Lisp package.

I do find supplying an in-package form in a macro expansion alarming,
though. I never use it, but would use-package work?

-hp

Post by Alessio Stalla

Post by Paul Tarvydas
What is the best way to write a macro that returns more than one form to
the top level? E.g.
(in-package :xxx)
(defun ...)
I've been wrapping a progn around the result, but LW doesn't like it very
much, and SBCL seems to hate it.
Thanks
pt

[Dave Cooper]

Post by Paul Tarvydas
What is the best way to write a macro that returns more than one form to
the top level? E.g.
(in-package :xxx)
(defun ...)
I've been wrapping a progn around the result, but LW doesn't like it very
much, and SBCL seems to hate it.

That's a pretty good question. I've never been able to find a way. It feels
like it might go against some kind of fundamental principle of macros.

What does LW and SBCL hate about wrapping toplevel forms in progn?

[Paul Tarvydas]

Post by Dave Cooper
...
What does LW and SBCL hate about wrapping toplevel forms in progn?

The LW debugger can't set breakpoints unless I remove the progn
wrapper. A user of my code is complaining that "it doesn't work" in
SBCL (:-) (I guess I'll get more detail later).

Asking the question got me to think of an experiment. I typed

(defstruct hello a b )

Into a file, then asked LW to macroexpand it. What is the result?

(in-package "COMMON-LISP-USER")

(dspec ...)

i.e. two forms without a wrapper!

And the answer in SBCL is a progn! With a an onerous-looking ...LOCK...
as the first form.

pt

[Pascal J. Bourguignon]

On Fri, Mar 14, 2014 at 11:45 AM, Paul Tarvydas <
What is the best way to write a macro that returns more than one
form to the top level?  E.g.
(in-package :xxx)
(defun ...)
I've been wrapping a progn around the result, but LW doesn't like
it very much, and SBCL seems to hate it.
That's a pretty good question. I've never been able to find a way. It
feels like it might go against some kind of fundamental principle of
macros. 

This has nothing to do with macros.

Please read: http://www.nhplace.com/kent/PS/Ambitious.html

[Kenneth Tilton]

Post by Paul Tarvydas
What is the best way to write a macro that returns more than one form to
the top level? E.g.
(in-package :xxx)
(defun ...)
I've been wrapping a progn around the result, but LW doesn't like it very
much, and SBCL seems to hate it.

I suspect you are making some other mistake, I do that all the time and the
spec says forms in a toplevel progn are effectively toplevel.

Got a mini-example you can share?

-hp

[Steve Haflich]

You don't show what you did, and what the two implementations complained
about. Indeed, progn is the usual way to have a macro return multiple
forms. progn is defined not to affect the top-level-ness or
not-top-level-ness of the current environment.

But I'll make a guess what's wrong. The in-package form changes the value
of *package*, which affects where the reader interns symbols, but your
defun form is _not_ subsequently processed by the reader. The symbols in
the defun are already interned symbols, not strings, and are interned in
whatever package they were read into when the reader saw them.

HTH

Post by Paul Tarvydas
What is the best way to write a macro that returns more than one form to
the top level? E.g.
(in-package :xxx)
(defun ...)
I've been wrapping a progn around the result, but LW doesn't like it very
much, and SBCL seems to hate it.
Thanks
pt

[Paul Tarvydas]

Alessio, Steve,

Yes, I intend the first form to be an (in-package ...) followed by a
(defun ...).

My immediate problem is solved, thank you! I am currently writing the
result out to a file anyway, so I can easily strip the progn on the way out.

But, now I'm curious. Is there a reasonable way to solve this?

thanks
pt

[Luís Oliveira]

Post by Paul Tarvydas
But, now I'm curious. Is there a reasonable way to solve this?

What problem are you trying solve? In particular, why do you need the
in-package form?

[Pascal J. Bourguignon]

Post by Paul Tarvydas
What is the best way to write a macro that returns more than one form
to the top level? E.g.
(in-package :xxx)
(defun ...)
I've been wrapping a progn around the result, but LW doesn't like it
very much, and SBCL seems to hate it.

(defmacro this-works-but-it-s-bad-form-to-mix-in-package-with-other-things ()
(let ((package :xxx))
`(progn
(in-package ,package)
(defun ,(intern "FNAME" package) (x)
(if (< x 1)
1
(* x (,(intern "FNAME" package) (1- x))))))))

[Paul Tarvydas]

On 14-03-14 12:57 PM, Pascal J. Bourguignon wrote:

I'm writing diagram-compilers that emit CL. It seems to me that each
emitted lump of code should be in it's own package. Am I thinking about
this "wrong"?

I also have similar issues using a PEG parser (a PEG syntax for esrap)
inside a keyboard macro https://github.com/guitarvydas/paraphrase .
This allows me to use prolog syntax directly within a CL program, which
macro-expands to lisp forms.

pt

[Pascal J. Bourguignon]

Post by Paul Tarvydas
I'm writing diagram-compilers that emit CL. It seems to me that each
emitted lump of code should be in it's own package. Am I thinking
about this "wrong"?

That said, if you want to "put" each lump of code inside its own
package, nothing prevents you to do so, but you don't need to include an
in-package form in the generated code.

(defvar *generated-packages* '())
(defvar *package-counter* 0)

(defun generate-code (x)
(let* ((pack (make-package (format nil "GENPACK~D"
(incf *package-counter*))
:use '()))
(fname (gentemp "FUN" pack))
(vname (gentemp "VAR" pack)))
(push pack *generated-packages*)
`(defun ,fname (,vname)
(+ ,vname ,x))))


cl-user> (generate-code 1)
(defun genpack1::fun0 (genpack1::var0) (+ genpack1::var0 1))
cl-user> (generate-code 2)
(defun genpack2::fun0 (genpack2::var0) (+ genpack2::var0 2))
cl-user> (generate-code 3)
(defun genpack3::fun0 (genpack3::var0) (+ genpack3::var0 3))
cl-user>

Then you only need to generate a file that creates the packages:

(dolist (p *generated-packages*)
(print `(make-package ,(package-name p) :use '())))

(make-package "GENPACK3" :use 'nil)
(make-package "GENPACK2" :use 'nil)
(make-package "GENPACK1" :use 'nil)

and to load it before the file contining the generated code.


But as I said, this is boring use so fine grained packages, just have
one static package for the whole generated code.

[Pascal J. Bourguignon]

Post by Paul Tarvydas
I'm writing diagram-compilers that emit CL. It seems to me that each
emitted lump of code should be in it's own package. Am I thinking
about this "wrong"?

Yes.

Packages are not designed or not practical for fine grained name
scoping.

Furthermore, packages are used only to _READ_ symbols. You should not
have any need of them when generating code. That is, at most, you would
just intern generated symbols in a specific package to avoid poluting
some other package, and you may want to manage the export list of a
public package for the publicly accessible API to your generated code.

[Alessio Stalla]

Post by Paul Tarvydas
I'm writing diagram-compilers that emit CL. It seems to me that each

emitted lump of code should be in it's own package. Am I thinking about
this "wrong"?

It's not wrong to have each piece of code in its own package. However, the
proper way to do so is to forget about in-package and use intern to produce
the needed symbols (see Pascal's example). In-package affects the reader,
while macros produce code, with no further reading involved.

Post by Paul Tarvydas
I also have similar issues using a PEG parser (a PEG syntax for esrap)

inside a keyboard macro https://github.com/guitarvydas/paraphrase . This
allows me to use prolog syntax directly within a CL program, which
macro-expands to lisp forms.

Post by Paul Tarvydas
pt

[Paul Tarvydas]

@All, thanks for the interesting discussion. Certainly gives me
something to chew on.

Re-reading the responses, I see that, while I did sort-of say it, I
didn't emphasize the point of this:

I have a PEG-syntax parser written in esrap.

I am binding at least two such parsers as reader-macros (not the normal
kind of macro).

The first parser parses PEG syntax and creates an esrap program.

That esrap program, bound as a reader-macro, reads Prolog syntax and
returns a bunch of lisp forms to the reader.

I have a file of code that contains both, lisp and Prolog syntax
(cl-heredoc to switch between the syntaxes), e.g.

https://github.com/guitarvydas/paraphrase/blob/master/prolog.lisp

The parsers read characters, not forms.

(lisp)
#{ prolog(A,B,c) :- p1(A), p2(B,c). }
(morelisp)

I am making at least 2 assumptions:

1. That using different name spaces for the various parsers will help me
preserve my sanity, (regression testing the PEG parser using itself
brings me to the limits of my comprehension :-),

2. That a reader macro must return a single form. I actually want to
return a separate 'defun' for every prolog rule in the file. (Writing
this, just now, makes me ask myself why I don't just put #{ ... } around
every separate rule...)

(I even started fooling around with a Python syntax).

This line of thinking was inspired by the Gambit Scheme talk at ILC2010,
where they showed inline infix mathematical expressions.

Thanks for your suggestions.

pt

[Alessio Stalla]

@All, thanks for the interesting discussion. Certainly gives me something
to chew on.
Re-reading the responses, I see that, while I did sort-of say it, I didn't
I have a PEG-syntax parser written in esrap.
I am binding at least two such parsers as reader-macros (not the normal
kind of macro).

I always thought that reader macros should be called a different name,
because they are not macros. Yes, they produce code; but that's the only
point they have in common with macros. The two should not be confused.

The first parser parses PEG syntax and creates an esrap program.
That esrap program, bound as a reader-macro, reads Prolog syntax and
returns a bunch of lisp forms to the reader.
I have a file of code that contains both, lisp and Prolog syntax
(cl-heredoc to switch between the syntaxes), e.g.
https://github.com/guitarvydas/paraphrase/blob/master/prolog.lisp
The parsers read characters, not forms.
(lisp)
#{ prolog(A,B,c) :- p1(A), p2(B,c). }
(morelisp)
1. That using different name spaces for the various parsers will help me
preserve my sanity, (regression testing the PEG parser using itself brings
me to the limits of my comprehension :-),
2. That a reader macro must return a single form. I actually want to
return a separate 'defun' for every prolog rule in the file. (Writing this,
just now, makes me ask myself why I don't just put #{ ... } around every
separate rule...)
(I even started fooling around with a Python syntax).
This line of thinking was inspired by the Gambit Scheme talk at ILC2010,
where they showed inline infix mathematical expressions.
Thanks for your suggestions.

Your assumptions are sane. You only have to understand in which phase -
read-time, compile-time, run-time - each piece of your code happens to run.
Your reader macros generate Lisp code (defun forms), right? Where do the
symbols that constitute those forms come from? Let me guess... I bet some
come from the source code of the reader macro itself - e.g., the symbol
CL:DEFUN. I bet others come from calls you make to the function intern or
find-symbol. Am I right? If so, ask yourself: which package is effective at
the time you call intern/find-symbol? Can you change it to another one?
What happens if you do so?

[Kenneth Tilton]

Post by Alessio Stalla

Post by Paul Tarvydas
@All, thanks for the interesting discussion. Certainly gives me
something to chew on.
Re-reading the responses, I see that, while I did sort-of say it, I
I have a PEG-syntax parser written in esrap.
I am binding at least two such parsers as reader-macros (not the normal
kind of macro).

I always thought that reader macros should be called a different name,
because they are not macros. Yes, they produce code; but that's the only
point they have in common with macros.

Given that that is all that macros do, methinks that a rather comprehensive
commonality.

Even a keyboard macro has no job other than to take a key chord and produce
key strokes,

-hp

[Pascal J. Bourguignon]

On Tue, Mar 18, 2014 at 4:50 PM, Alessio Stalla <
On Tue, Mar 18, 2014 at 8:54 PM, Paul Tarvydas <
@All, thanks for the interesting discussion.  Certainly gives
me something to chew on.
Re-reading the responses, I see that, while I did sort-of say
I have a PEG-syntax parser written in esrap.
I am binding at least two such parsers as reader-macros (not
the normal kind of macro).
I always thought that reader macros should be called a different
name, because they are not macros. Yes, they produce code; but
that's the only point they have in common with macros.
Given that that is all that macros do, methinks that a rather
comprehensive
commonality.
Even a keyboard macro has no job other than to take a key chord and
produce key strokes,

Indeed, MACRO comes from greek meaning BIG, and refers to the fact that
a single item is used to represent a (possibly) BIG group of items.


A macro operator is a single operator that gets expanded into a big form
(check it out, most lisp macros expand to big forms!).

A macro character, is a single character that gets read into a big sexp.

A keyboard macro, is a single key that gets read as a big sequence of
keys.

A compiler macro, is a single function call that gets substituted by a
big, inlined and optimized, function call.



It's not the code generation that's the common part in all those macros,
it's the small to big expansion that usually occurs.

[Terje Norderhaug]

Post by Paul Tarvydas
What is the best way to write a macro that returns more than one form to
the top level? E.g.
(in-package :xxx)
(defun ...)
I've been wrapping a progn around the result, but LW doesn't like it very
much, and SBCL seems to hate it.

Wrap the body in eval-when.

-- Terje Norderhaug

[Luís Oliveira]

Post by Terje Norderhaug
Wrap the body in eval-when.

How does that help? That is, what problem is it solving?