DEFVARs combined service

[Kazimir Majorinc]

"DEFVAR and DEFPARAMETER do the combined service of

- in some implementations, recording the "definitional home" of the
variable
- declaring the indicated variable special in compiler and runtime
- assigning the variable (either conditionally, as in DEFVAR, which
assigns only in the case that the variable is unbound, or
unconditionally, as in DEFPARAMETER). "

K. Pitman, comp.lang.lisp, 26. April 2001.

What "recording the definitional home" means? Why "in some
implementations?"

[Pascal J. Bourguignon]

Post by Kazimir Majorinc
"DEFVAR and DEFPARAMETER do the combined service of
- in some implementations, recording the "definitional home" of the
variable
- declaring the indicated variable special in compiler and runtime
- assigning the variable (either conditionally, as in DEFVAR, which
assigns only in the case that the variable is unbound, or
unconditionally, as in DEFPARAMETER). "
K. Pitman, comp.lang.lisp, 26. April 2001.
Why "in some implementations?"

Because this is something that is just not specified.

Post by Kazimir Majorinc
What "recording the definitional home" means?

It's recording that the variable is defined in that file, at that
position, so that you the "IDE" may jump to the definition of the
variable to inspect it or modify it.

There's an additionnal service that's specified:

- record the variable documentation for that symbol.

[Daniel Herring]

Post by Kazimir Majorinc
"DEFVAR and DEFPARAMETER do the combined service of
- in some implementations, recording the "definitional home" of the
variable
- declaring the indicated variable special in compiler and runtime
- assigning the variable (either conditionally, as in DEFVAR, which
assigns only in the case that the variable is unbound, or
unconditionally, as in DEFPARAMETER). "
K. Pitman, comp.lang.lisp, 26. April 2001.
What "recording the definitional home" means? Why "in some
implementations?"

Without seeing the context, I read a couple things into this.

- source location for stuff like M-.
- compilation unit which creates memory for and initializes the variable

Later,
Daniel

[Ben Hyde]

Post by Kazimir Majorinc
"DEFVAR and DEFPARAMETER do the combined service of
- in some implementations, recording the "definitional home" of the
variable ...

That reminds me; I desire an implementation independent library that
abstracts out the recording of the "definitional home."

[Mark H. David]

Can you explain a bit what you mean by this?

Post by Ben Hyde

Post by Kazimir Majorinc
"DEFVAR and DEFPARAMETER do the combined service of
- in some implementations, recording the "definitional home" of the
variable ...

That reminds me; I desire an implementation independent library that
abstracts out the recording of the "definitional home."
_______________________________________________
pro mailing list
http://common-lisp.net/cgi-bin/mailman/listinfo/pro

[Ben Hyde]

Post by Mark H. David
Can you explain a bit what you mean by this?

My example. Say I have a macro for defining widgets and I use it I
define cool-widget on line N of file foo.lisp like so:

(define-widget cool-widget ...)

In the wonderful world which is my imagination I should be able to
meta-dot the symbol cool-widget
and shortly there after find my self visiting line N of foo.lisp.

This presumes that each implementation supports the concept of
definition homes for - a mapping from
keys of some sort (typically a symbol) to source location(s). The
author of define-widget would add
entries to that mapping. The IDE (slime say) would query that mapping.

The author of define-widget might write something like so:

(defmacro define-widget (name ...)
...
`(progn
(take-note-of-definitional-home ,name ... :widget ...)
...))

The kind implementor of the 'implementation independent library that
abstracts out the
recording of the "definitional home"' would know how to map the
arguments to take-note-of-definitional-home
into something that actually adds/revises an entry into the
implemenation's mapping.

It would be a wonderful world, yes it would.

- ben

[Mark H. David]

Yeah, that would be great. We had a bit of a discussion of this sort of
functionality on the slime-devel mailing list.
Nobody had any easy answers, even for just SBCL and CCL, which I would
have settled for.
It was also pointed out that there's a basic first level of
functionality, permitting meta-. to work.
Then, the next level preserves source location of subforms of a body of
Lisp code to support "PC to Source" mapping.
This doesn't seem all that hard to do at least a first-level version, at
least with the stuff SLIME can already do.
And then encapsulating it in a Library would be great. Excellent
library candidate.
Hope someone will do this -- it will be really worthwhile and make a lot
of developers happy.

Post by Ben Hyde

Post by Mark H. David
Can you explain a bit what you mean by this?

My example. Say I have a macro for defining widgets and I use it I
(define-widget cool-widget ...)
In the wonderful world which is my imagination I should be able to
meta-dot the symbol cool-widget
and shortly there after find my self visiting line N of foo.lisp.
This presumes that each implementation supports the concept of
definition homes for - a mapping from
keys of some sort (typically a symbol) to source location(s). The
author of define-widget would add
entries to that mapping. The IDE (slime say) would query that mapping.
(defmacro define-widget (name ...)
...
`(progn
(take-note-of-definitional-home ,name ... :widget ...)
...))
The kind implementor of the 'implementation independent library that
abstracts out the
recording of the "definitional home"' would know how to map the
arguments to take-note-of-definitional-home
into something that actually adds/revises an entry into the
implemenation's mapping.
It would be a wonderful world, yes it would.
- ben

[Alessio Stalla]

Yeah, that would be great.  We had a bit of a discussion of this sort of
functionality on the slime-devel mailing list.
Nobody had any easy answers, even for just SBCL and CCL, which I would
have settled for.
It was also pointed out that there's a basic first level of
functionality, permitting meta-. to work.
Then, the next level preserves source location of subforms of a body of
Lisp code to support "PC to Source" mapping.
This doesn't seem all that hard to do at least a first-level version, at
least with the stuff SLIME can already do.
And then encapsulating it in a Library would be great.  Excellent
library candidate.

I think it would be best done by implementers (a CDR?) rather than
being a library, since it interferes quite deeply with the reader. I
agree that doing the first level is not hard; the next level is not
hard either, conceptually - but since you can't have the reader record
all the subforms when you're only interested in a few of them, you'd
need some way to mark the forms you're interested in, e.g.

#.(reader:record-next-form (form file line column) (setf (get
'some-symbol 'some-property) (list form file line column)))
(yadda-yadda-yadda)

-->

(progn
(let ((form '#1=(yadda-yadda-yadda)) (file "foo.lisp") (line 123) (column 42))
(setf (get 'some-symbol 'some-property) (list form file line column)))
#1#)

Or something like that.

Hope someone will do this -- it will be really worthwhile and make a lot
of developers happy.

I'm not a SLIME hacker, but I'm a contributor to ABCL. I'm willing to
experiment with the CL implementation side of this, to produce an API
that SLIME and other clients could use. If I'll think I'm successful
and if there will be enough interest and consensus, I could also
propose it as a CDR.

Cheers,
Alessio

[Pascal J. Bourguignon]

Post by Ben Hyde

Post by Kazimir Majorinc
"DEFVAR and DEFPARAMETER do the combined service of
- in some implementations, recording the "definitional home" of the
variable ...

That reminds me; I desire an implementation independent library that
abstracts out the recording of the "definitional home."

That starts with a lisp reader, or at least an implementation dependant API to collect the file position of each lisp object read.

[dherring-Q3akJ13YhGdWk0Htik3J/]

Post by Pascal J. Bourguignon

Post by Ben Hyde

Post by Kazimir Majorinc
"DEFVAR and DEFPARAMETER do the combined service of
- in some implementations, recording the "definitional home" of the
variable ...

That reminds me; I desire an implementation independent library that
abstracts out the recording of the "definitional home."

That starts with a lisp reader, or at least an implementation dependant
API to collect the file position of each lisp object read.

The Conium project looks like a good place for that sort of thing.
http://gitorious.org/conium#more

- Daniel

[Sam Steingold]

Post by Ben Hyde

Post by Kazimir Majorinc
"DEFVAR and DEFPARAMETER do the combined service of
- in some implementations, recording the "definitional home" of the
variable ...

That reminds me; I desire an implementation independent library that
abstracts out the recording of the "definitional home."

I am afraid this would be quite hard to do: imagine a variable defined
in different files for different implementations and features.

On the other hand, it you are willing to compromise on implementation
independence, you can use slime: I believe they support many
implementations (each implementation has its own internal idiosyncratic
way to remember where a variable is defined, and they abstract that out
into a common interface).

[Stelian Ionescu]

Post by Ben Hyde

Post by Kazimir Majorinc
"DEFVAR and DEFPARAMETER do the combined service of
- in some implementations, recording the "definitional home" of the
variable ...

That reminds me; I desire an implementation independent library that
abstracts out the recording of the "definitional home."

That's part of Slime

[dherring-Q3akJ13YhGdWk0Htik3J/]

Post by Stelian Ionescu

Post by Ben Hyde

Post by Kazimir Majorinc
"DEFVAR and DEFPARAMETER do the combined service of
- in some implementations, recording the "definitional home" of the
variable ...

That reminds me; I desire an implementation independent library that
abstracts out the recording of the "definitional home."

That's part of Slime

Please look at Conium.
http://gitorious.org/conium#more

David Lichteblau had the right idea: This functionality needs to be
broken out of Slime and made available as a general toolset.

- Daniel

[Bob Kerns]

I don't mean to discourage innovative thinking here, but for reference,
consider that the editor and/or separate source indexer can locate defXXX
forms, and record the location of (define-widget cool-widget ...) as
(:definition (:name cool-widget :type define-widget) :location (:file
"file.lisp" :line 28)), and make it available to meta-dot.

While I can see advantages to having a form record where it came from, as
you guys are describing, there's also a huge disadvantage -- the need to
actually expand (and possibly evaluate) that macro, before being able to use
meta-dot.

You'd *like* to be able to edit code which is only conditionally loaded, or
intended for some other platform or environment, or not yet working. Let's
say, you just modified (and broke) the define-widget macro,and now you want
to look at its usage -- but you can't, because you can't load it, because
the file define-widget itself is in is broken and won't load.

You'd even like to be able to index forms which are read by neither the
compiler nor the interpreter. You may have definitional data which you want
to constrain to certain definitional types, and NOT allow arbitrary
evaluation (for example, it may be from an untrusted source).

That said, it's typical for compilers and interpreters, in many languages,
to record source locations. This has advantages when no source is available
-- for example, reporting stack traces. Sometimes you can even detect
version-mismatches between source and binary as a result. But I don't think
it's ever the optimal solution for a development environment.

Eclipse sort of does both for Java -- if you have a jar file with binaries,
you can associate it with a directory or jar with sources; otherwise, it
indexes the sources as it parses them, collects cross-reference data, etc.
(It also constructs cross-reference data for the .jar files, so you can find
usages within methods within classes in compiled jar files).

So I'd say, what you really want here is, not a run-time or read-time
action, but rather a declarative hint to both compiler and editor, that a
particular form is definitional.

The simplest, and traditional way to do that, is to simply declare that
forms beginning with 'def' are definitional. It's arbitrary -- but it's
simple.

[Ben Hyde]

Post by Bob Kerns
While I can see advantages to having a form record where it came
from, as you guys are describing, there's also a huge disadvantage
-- the need to actually expand (and possibly evaluate) that macro,
before being able to use meta-dot.

I use tag files. Tag files are good.

Even better, a lot better, is id-utils mkid' m-x gid; I recommend it -
even if it has some rough edges.

Of course, no amount of lexical indexing can discover the definition
of cool-widget that arose from a define-toy-box form.

Post by Bob Kerns
I think it would be best done by implementers (a CDR?) rather than
being a library, since it interferes quite deeply with the reader.

I suspect that most implementations have this functionality. In
ccl:record-source-file for example; and in LispWorks the subsystem is
almost entirely documented:
http://www.lispworks.com/documentation/lw50/LWUG/html/lwuser-54.htm
http://www.lispworks.com/documentation/lwm43/LWRM/html/lwref-91.htm

Post by Bob Kerns

Post by Stelian Ionescu
...
That's part of Slime

Please look at Conium.
http://gitorious.org/conium#more

Conium is sweet. That helps some queries to the data stored about
definitional homes, but it doesn't help with making entries into that
data store.

----

I seem to recall that the Lisp Machine or the Symbolics also had
something analogous to LispWork's dspecs; but i couldn't fine it with
a minute or two of looking.

I was sad to see that some of the implementations lack affordances
for extending the kinds of things defined, e.g. adding widgets would
be a pain.

The LispWorks documentation is full of interest. They take a run at
providing both containment (things defined inside of other things) as
well as single things defined by the union of many forms (e.g. generic
methods and their individual methods).

---

In addition to the index-my-sources and the collect-assertions-about-
definition approaches there is another possible tack. If your
compiler writer is staying up late he may have provided very
fastidious mappings back to the source file for all the code he's
generating. People do that for debugging, code coverage, and even
implementing unwind cleanups ... i guess that data might also be
leveraged for this. It's not obviously how though.

- ben

- ben