Discussion:
[pro] [MOP] Does intern-eql-specializer serve any real purpose?
Jean-Claude Beaudoin
2014-12-26 07:10:24 UTC
Permalink
Hello CL Pros,

Lately I have been improving the MOPishness of MKCL and that brought me in
contact with the specification of intern-eql-specializer in AMOP.

The EQ requirement on its returned value seem to me to dictate a hash-table
implementation (PCL and its derivatives all seem to do just that).

The problem I see with this is that it will be a "for ever growing"
hash-table with not upper bound in sight. And the "purpose" of such a
mandatory built-in memory leak also completely escapes me. Could any of you
share some insight on this question, please?

Thank you,

JCB
Kenneth Tilton
2014-12-26 15:33:06 UTC
Permalink
Not sure where I see either "forever growing" or "memory leak". Come to
think of it, not sure what you mean by "mandatory". It is a spec of how the
MOP should work internally. Do you have some other way in mind for things
to work?

I mean, it sounds like you might be talking about forever adding and
removing eql-specialized methods, but I'd rather not guess. Even if so,
nothing stops the implementation from GCing unused specializer metaobjects.

-hk


On Fri, Dec 26, 2014 at 2:10 AM, Jean-Claude Beaudoin <
Post by Jean-Claude Beaudoin
Hello CL Pros,
Lately I have been improving the MOPishness of MKCL and that brought me in
contact with the specification of intern-eql-specializer in AMOP.
The EQ requirement on its returned value seem to me to dictate a
hash-table implementation (PCL and its derivatives all seem to do just
that).
The problem I see with this is that it will be a "for ever growing"
hash-table with not upper bound in sight. And the "purpose" of such a
mandatory built-in memory leak also completely escapes me. Could any of you
share some insight on this question, please?
Thank you,
JCB
_______________________________________________
pro mailing list
http://mailman.common-lisp.net/cgi-bin/mailman/listinfo/pro
--
Kenneth Tilton
Fort Lauderdale, FL
http://tiltontec.com
"In a class by itself." *-Macworld*
Scott McKay
2014-12-26 15:38:37 UTC
Permalink
The specializers are probably interned for the sake of allowing
better optimization of EQL dispatch. I don't think it'll grow without
bounds; it probably gets big enough to hold all of the methods with
EQL dispatch, and then grows no further.

Honestly, I don't use EQL methods much myself, if they can be
avoided by using SELECT.

--S
Post by Kenneth Tilton
Not sure where I see either "forever growing" or "memory leak". Come to
think of it, not sure what you mean by "mandatory". It is a spec of how the
MOP should work internally. Do you have some other way in mind for things
to work?
I mean, it sounds like you might be talking about forever adding and
removing eql-specialized methods, but I'd rather not guess. Even if so,
nothing stops the implementation from GCing unused specializer metaobjects.
-hk
On Fri, Dec 26, 2014 at 2:10 AM, Jean-Claude Beaudoin <
Post by Jean-Claude Beaudoin
Hello CL Pros,
Lately I have been improving the MOPishness of MKCL and that brought me
in contact with the specification of intern-eql-specializer in AMOP.
The EQ requirement on its returned value seem to me to dictate a
hash-table implementation (PCL and its derivatives all seem to do just
that).
The problem I see with this is that it will be a "for ever growing"
hash-table with not upper bound in sight. And the "purpose" of such a
mandatory built-in memory leak also completely escapes me. Could any of you
share some insight on this question, please?
Thank you,
JCB
_______________________________________________
pro mailing list
http://mailman.common-lisp.net/cgi-bin/mailman/listinfo/pro
--
Kenneth Tilton
Fort Lauderdale, FL
http://tiltontec.com
"In a class by itself." *-Macworld*
_______________________________________________
pro mailing list
http://mailman.common-lisp.net/cgi-bin/mailman/listinfo/pro
Steve Haflich
2014-12-26 16:58:46 UTC
Permalink
Scott (long time no see):

This reply doesn't seem correct at all. Interning an EQL specializer
does not directly have anything to do with methods (although it is
most often performed by the implementation when an EQL method is
defined) nor would methods be remembered in the hash table or
whatever the implementation uses to intern them.

An EQL specializer must exist in order for METHOD-SPECIALIZERS to have
something to return, and as an argument to be passed to functions like
SPECIALIZER-DIRECT-GENERIC-FUNCTIONS
SPECIALIZER-DIRECT-METHODS
EQL-SPECIALIZER-OBJECT

Specializers are also necessary if one wants to manipulate generic
functions and methods directly using the MOP instead of with the
DEFGENERIC and DEFMETHOD macros. If for no other reason, specializers
exist because the MOP must be self-reflexive.

True, there is no way to remove an interned EQL specializer, which may
be an oversight in the MOP. But there will be exactly one entry for
each unique specializer in the EQL hash table or whatever else is used
for their internment. I suppose an implementation that features weak
hashtables could use one for EQL specializers, since if there were no
other references to one there would be no portable way to ask if it
was still there.

"Growing without bounds" is no different than what happens when a new
symbol is interned, or a new function or method is defined on an
interned symbol. Rarely any problem in typical programming.
Post by Scott McKay
The specializers are probably interned for the sake of allowing
better optimization of EQL dispatch. I don't think it'll grow without
bounds; it probably gets big enough to hold all of the methods with
EQL dispatch, and then grows no further.
Honestly, I don't use EQL methods much myself, if they can be
avoided by using SELECT.
--S
Post by Kenneth Tilton
Not sure where I see either "forever growing" or "memory leak". Come to
think of it, not sure what you mean by "mandatory". It is a spec of how the
MOP should work internally. Do you have some other way in mind for things to
work?
I mean, it sounds like you might be talking about forever adding and
removing eql-specialized methods, but I'd rather not guess. Even if so,
nothing stops the implementation from GCing unused specializer metaobjects.
-hk
On Fri, Dec 26, 2014 at 2:10 AM, Jean-Claude Beaudoin
Post by Jean-Claude Beaudoin
Hello CL Pros,
Lately I have been improving the MOPishness of MKCL and that brought me
in contact with the specification of intern-eql-specializer in AMOP.
The EQ requirement on its returned value seem to me to dictate a
hash-table implementation (PCL and its derivatives all seem to do just
that).
The problem I see with this is that it will be a "for ever growing"
hash-table with not upper bound in sight. And the "purpose" of such a
mandatory built-in memory leak also completely escapes me. Could any of you
share some insight on this question, please?
Thank you,
JCB
_______________________________________________
pro mailing list
http://mailman.common-lisp.net/cgi-bin/mailman/listinfo/pro
--
Kenneth Tilton
Fort Lauderdale, FL
http://tiltontec.com
"In a class by itself." -Macworld
_______________________________________________
pro mailing list
http://mailman.common-lisp.net/cgi-bin/mailman/listinfo/pro
_______________________________________________
pro mailing list
http://mailman.common-lisp.net/cgi-bin/mailman/listinfo/pro
Kenneth Tilton
2014-12-26 19:38:33 UTC
Permalink
Why is there no way to remove an interned EQL specializer meta-object? They
get defined in the context of a method definition, so one just needs to do
some good old-fashioned engineering: method-specializer reference tracking
leveraged at method removal time to know when to toss the hash table entry.

I am more interested in why this is perceived as a problem, but if the OP
is doing some dynamic metaprogramming I can imagine a use case.
Post by Steve Haflich
This reply doesn't seem correct at all. Interning an EQL specializer
does not directly have anything to do with methods (although it is
most often performed by the implementation when an EQL method is
defined) nor would methods be remembered in the hash table or
whatever the implementation uses to intern them.
An EQL specializer must exist in order for METHOD-SPECIALIZERS to have
something to return, and as an argument to be passed to functions like
SPECIALIZER-DIRECT-GENERIC-FUNCTIONS
SPECIALIZER-DIRECT-METHODS
EQL-SPECIALIZER-OBJECT
Specializers are also necessary if one wants to manipulate generic
functions and methods directly using the MOP instead of with the
DEFGENERIC and DEFMETHOD macros. If for no other reason, specializers
exist because the MOP must be self-reflexive.
True, there is no way to remove an interned EQL specializer, which may
be an oversight in the MOP. But there will be exactly one entry for
each unique specializer in the EQL hash table or whatever else is used
for their internment. I suppose an implementation that features weak
hashtables could use one for EQL specializers, since if there were no
other references to one there would be no portable way to ask if it
was still there.
"Growing without bounds" is no different than what happens when a new
symbol is interned, or a new function or method is defined on an
interned symbol. Rarely any problem in typical programming.
Post by Scott McKay
The specializers are probably interned for the sake of allowing
better optimization of EQL dispatch. I don't think it'll grow without
bounds; it probably gets big enough to hold all of the methods with
EQL dispatch, and then grows no further.
Honestly, I don't use EQL methods much myself, if they can be
avoided by using SELECT.
--S
Post by Kenneth Tilton
Not sure where I see either "forever growing" or "memory leak". Come to
think of it, not sure what you mean by "mandatory". It is a spec of how
the
Post by Scott McKay
Post by Kenneth Tilton
MOP should work internally. Do you have some other way in mind for
things to
Post by Scott McKay
Post by Kenneth Tilton
work?
I mean, it sounds like you might be talking about forever adding and
removing eql-specialized methods, but I'd rather not guess. Even if so,
nothing stops the implementation from GCing unused specializer
metaobjects.
Post by Scott McKay
Post by Kenneth Tilton
-hk
On Fri, Dec 26, 2014 at 2:10 AM, Jean-Claude Beaudoin
Post by Jean-Claude Beaudoin
Hello CL Pros,
Lately I have been improving the MOPishness of MKCL and that brought me
in contact with the specification of intern-eql-specializer in AMOP.
The EQ requirement on its returned value seem to me to dictate a
hash-table implementation (PCL and its derivatives all seem to do just
that).
The problem I see with this is that it will be a "for ever growing"
hash-table with not upper bound in sight. And the "purpose" of such a
mandatory built-in memory leak also completely escapes me. Could any
of you
Post by Scott McKay
Post by Kenneth Tilton
Post by Jean-Claude Beaudoin
share some insight on this question, please?
Thank you,
JCB
_______________________________________________
pro mailing list
http://mailman.common-lisp.net/cgi-bin/mailman/listinfo/pro
--
Kenneth Tilton
Fort Lauderdale, FL
http://tiltontec.com
"In a class by itself." -Macworld
_______________________________________________
pro mailing list
http://mailman.common-lisp.net/cgi-bin/mailman/listinfo/pro
_______________________________________________
pro mailing list
http://mailman.common-lisp.net/cgi-bin/mailman/listinfo/pro
_______________________________________________
pro mailing list
http://mailman.common-lisp.net/cgi-bin/mailman/listinfo/pro
--
Kenneth Tilton
Fort Lauderdale, FL
http://tiltontec.com
"In a class by itself." *-Macworld*
Steve Haflich
2014-12-27 03:10:21 UTC
Permalink
Post by Kenneth Tilton
Why is there no way to remove an interned EQL specializer meta-object?
Because no way to do this was defined in the MOP. It's unclear
whether you suggest there should be some programmatic way to unintern
an EQL specializer, or whether the system should do it automatically.
But neither makes a lot of sense.

If a user call uninterned an EQL specializer while there were still
methods specialized upon it, then the MOP would become inconsistent.
Post by Kenneth Tilton
They get defined in the context of a method definition,
They are also interned by an explicit user-code call to
INTERN-EQL-SPECIALIZER, which would be a reasonably thing to do it
using the MOP directly to install new methods, or even to test whether
any method or gf exists specialized on that EQL object.
Post by Kenneth Tilton
so one just needs to do
some good old-fashioned engineering: method-specializer reference tracking
leveraged at method removal time to know when to toss the hash table entry.
(defparameter .kenny. (intern-eql-specializer 'tilton))

How would the MOP do reference counting on this metaobject. If the
implementation spontaneously uninterned it, then a subsequent call to
i-e-s would return a different metaobject, in violation of the MOP
specification.
Post by Kenneth Tilton
I am more interested in why this is perceived as a problem, but if the OP is
doing some dynamic metaprogramming I can imagine a use case.
Yes, indeed. I expect this thread is a lot of owrrying about nothing
important. But as I suggested previously, an implementation with weak
hash tables could unintern EQL specializers safely if it wanted to
bother.
Kenneth Tilton
2014-12-27 04:04:09 UTC
Permalink
Post by Steve Haflich
Post by Kenneth Tilton
Why is there no way to remove an interned EQL specializer meta-object?
Because no way to do this was defined in the MOP.
Wait. Are we in the religious zone now? This is engineering.
Post by Steve Haflich
It's unclear
whether you suggest there should be some programmatic way to unintern
an EQL specializer,
Yeah, it is trivial if you want to make it work. No help of course for the
religious zone.
Post by Steve Haflich
or whether the system should do it automatically.
But neither makes a lot of sense.
If a user call uninterned an EQL specializer while there were still
methods specialized upon it, then the MOP would become inconsistent.
What is the user up to, inside the MOP? Why are they doing something so
perverse? Whatever reason they have, they are inside the mop they are
authoring. (Guessing internal-eql-specializer is not exported). If their
mop worries about IES leaks, they need to enforce GCability.

Should we not decide the angels atop pin headcount first?

I cannot believe the nonsese into which language lawyers forgetting they
are endineers get themselves.
Post by Steve Haflich
Post by Kenneth Tilton
They get defined in the context of a method definition,
They are also interned by an explicit user-code call to
INTERN-EQL-SPECIALIZER, which would be a reasonably thing to do it
using the MOP directly to install new methods,
Dude. They are INSIDE a MOP that frets over IES leaks, they need to cope.
Post by Steve Haflich
or even to test whether
any method or gf exists specialized on that EQL object.
OK, I see. We are not in Kansas anymore.

I gotta say, no longer interested until the OP offers motivation for their
concerns.

One thing we know in programming is that the abstract is largely
unaddressable. bring me a sensible use case and I can help you.

Given that the OP has disappeared, i think this wise.

-hk
--
Kenneth Tilton
Fort Lauderdale, FL
http://tiltontec.com
"In a class by itself." *-Macworld*
Jean-Claude Beaudoin
2014-12-27 04:17:24 UTC
Permalink
Post by Kenneth Tilton
...
Given that the OP has disappeared, i think this wise.
What do you mean "disappeared". I am just busy thinking through my
reply/ies.
Sorry if I am not as quick as you are in these matters, Kenny.
And don't forget that English is a second language for me on the top of
it...

BTW, thank you all for your thoughtful replies/reactions. They are quite
useful.

Regards,

JCB
Kenneth Tilton
2014-12-27 04:22:56 UTC
Permalink
Bonjour, Jean Claude!

I look forward to your reappearance! No, this does not count! :)

-hk
Jean-Claude Beaudoin
2014-12-27 09:59:47 UTC
Permalink
Post by Steve Haflich
Post by Kenneth Tilton
Why is there no way to remove an interned EQL specializer meta-object?
Because no way to do this was defined in the MOP. It's unclear
whether you suggest there should be some programmatic way to unintern
an EQL specializer, or whether the system should do it automatically.
But neither makes a lot of sense.
If a user call uninterned an EQL specializer while there were still
methods specialized upon it, then the MOP would become inconsistent.
Post by Kenneth Tilton
They get defined in the context of a method definition,
They are also interned by an explicit user-code call to
INTERN-EQL-SPECIALIZER, which would be a reasonably thing to do it
using the MOP directly to install new methods, or even to test whether
any method or gf exists specialized on that EQL object.
Post by Kenneth Tilton
so one just needs to do
some good old-fashioned engineering: method-specializer reference
tracking
Post by Kenneth Tilton
leveraged at method removal time to know when to toss the hash table
entry.
(defparameter .kenny. (intern-eql-specializer 'tilton))
How would the MOP do reference counting on this metaobject. If the
implementation spontaneously uninterned it, then a subsequent call to
i-e-s would return a different metaobject, in violation of the MOP
specification.
Post by Kenneth Tilton
I am more interested in why this is perceived as a problem, but if the
OP is
Post by Kenneth Tilton
doing some dynamic metaprogramming I can imagine a use case.
Yes, indeed. I expect this thread is a lot of owrrying about nothing
important. But as I suggested previously, an implementation with weak
hash tables could unintern EQL specializers safely if it wanted to
bother.
_______________________________________________
pro mailing list
http://mailman.common-lisp.net/cgi-bin/mailman/listinfo/pro
I am ready to concede that the issue here is minor and peripheral but it is
not inexistent.

From Steve's mention of specializer-direct-methods and friend it is clear
to me now that intern-eql-specializer is part of a specializer specific
dependency tracking facility. And, looking again at the source code of PCL,
I can see that facility used at least in the optimization of make-instance
and compute-applicable-methods (probably as Scott suspected). So, this is
reasonable purpose for me and I consider my original (subject line)
question properly answered.

Implementing intern-eql-specializer by means of a weak hash-table as
suggested by Steve is exactly what CCL does, clisp uses something similar
(weak sets I think), but not SBCL. But somehow I don't think it fair from
MOP to require every implementation to use weak hash-tables in this case.

I am more of the opinion that the specification is incomplete in this area.
The parallel with the situation of symbols in packages had also struck me
and I think it should be pushed somewhat further with the addition of
unintern-eql-specializer (granted this one is as dangerous as unintern is
for symbols) and of something like map-eql-specializers (à la do-symbols).
(BTW, in PCL you can see its optimization code use a map-specializers.)
With these two it becomes possible to implement something like
scrub-unused-eql-specializers if one wants to, without them it is simply
impossible.

I think this is just good principled engineering being applied here. And,
no Kenny, principled engineering, as I understand it, is not a religion but
rather a philosophy, a subdivision of pragmatism I would say even if you
don't find it pragmatic enough.

And the principles at work here would be:

1) Respect clearly and completely defined interfaces.

2) Inaccessible internal state is a very bad thing, avoid it.

I admit that 2) comes from my hardware design days (way back) but I think
it also applies to software, pretty much for the same reasons it imposed
itself on the hardware side as central to the "design for testability"
methodology.

Thank you all again for your replies. They have been of great help.

Cheers,

JCB
Pascal Costanza
2014-12-27 10:50:29 UTC
Permalink
Hi,

1. intern-eql-specializer is not strictly necessary. eql-specializer metaobjects are actually not strictly necessary. Case in point: LispWorks doesn’t have eql-specializer metaobjects in its MOP implementation at all. In LispWorks, eql specializers are just lists of the form `(eql ,object). I don’t have the impression that this has a negative impact on performance, or so. When eql specializers are just lists like this, then I believe the whole issue of getting rid of ‘garbage’ eql specializers is a non issue: Once a method with a particular eql specializer is removed, the corresponding object can be freed (unless other references exist, of course). (You still have the issue of not easily being able to get rid of ‘garbage’ classes.)

2. Don’t take the CLOS MOP specification as some kind of final word on how CLOS should be implemented. The CLOS MOP was still work in progress when the specification was published, and this is acknowledged in “The Art of the Metaobject Protocol.” It’s perfectly possible that there are better ways to implement CLOS with better protocols. In practice, almost actual CLOS implementations actually deviate in one way or another from the CLOS MOP specification, which is partially for historical reasons, partially because there are holes in the CLOS MOP specification, partially because parts of the spec are not implementable, and partially because certain aspects can actually be implemented better.

3. I’m not convinced that eql specializers are the most pressing concern. For example, the total lack of a good specification for method combination metaobjects is much more worrisome. Also, the generic function invocation protocol is too restrictive, in that it doesn’t allow for more modern inlining techniques (like polymorphic inline caches, or trace-based JIT compilation, for example). There is probably more.

4. If you are really concerned about eql specializers using up too much memory (and I’m really wondering what kind of use case that might be), then it’s probably better to roll your own approach to make that work. You don’t even need to go through the MOP, just implement your own dispatch mechanism, maybe even completely independent from CLOS. It’s not that hard, and you can come up with rules for your own dispatch mechanism that don’t have to be in line with what CLOS or the CLOS MOP require at all.

Pascal

--
Pascal Costanza
The views expressed in this email are my own, and not those of my employer.
Jean-Claude Beaudoin
2014-12-27 11:18:30 UTC
Permalink
Hi,
1. intern-eql-specializer is not strictly necessary. ...
I had somewhat noticed ;-)
2. Don’t take the CLOS MOP specification as some kind of final word on how
CLOS should be implemented. ...
Usage as made it more or less mandatory by now I think.
3. I’m not convinced that eql specializers are the most pressing concern.
For example, the total lack of a good specification for method combination
metaobjects is much more worrisome.
Yep! That one was a bit freakish when I redid it lately in the thing I
tinker with.
Also, the generic function invocation protocol is too restrictive, in that
it doesn’t allow for more modern inlining techniques (like polymorphic
inline caches, or trace-based JIT compilation, for example).
This is almost precisely the subject I wanted to address next on this list,
in a couple of weeks, when I'll think I'd be ready...
4. If you are really concerned about eql specializers
I am not that much concerned about them and, in fact, I consider the matter
settled now.

Thanks,

JCB
Kenneth Tilton
2014-12-27 16:45:28 UTC
Permalink
On Sat, Dec 27, 2014 at 4:59 AM, Jean-Claude Beaudoin <
Post by Jean-Claude Beaudoin
I think this is just good principled engineering being applied here. And,
no Kenny, principled engineering, as I understand it, is not a religion but
rather a philosophy, a subdivision of pragmatism I would say even if you
don't find it pragmatic enough.
Doh! I am in the wrong meet-up. I was looking for the one for Simple Lisp
*Application* Programmers!

Cool. In this "where the Hell is unintern-eql-specializer" fretting I see
yet another reason CL rules. And agreed: details matter.

Sorry for the noise.

-hk

Loading...