Should GC hooks be used to help broken software?

[Elias Mårtenson]

The following question was raised during my development of an asynchronous
library <https://github.com/lokedhs/cl-rabbit-async> I'm currently building
for RabbitMQ.

In summary, the library allows you to create an object of type
ASYNCH-CONNECTION, from which instances of ASYNC-CHANNEL can be retrieved.
A connection holds a reference to all channels that it uses, and each
channel holds a reference to its connection. The connection object has a
pointer to a native CFFI object that for the underlying connection (my
library is built on the RabbitMQ C API).

My question is: Should I use trivial-garbage to create a GC hook for the
connection object so that if the user of the library forgets to close the
connection, it will get closed eventually once it's GC'ed?

I can see arguments for both behaviours:

- It's a bad idea, since losing the reference to the connection object
means that the program is broken, and silently cleaning up the underlying
connection might hide the fact there is a bug (if the GC doesn't run often
enough I might have hundreds or even thousands of lingering connections)
- On the other hand, it might be a good idea since Lisp developers often
use the REPL to experiment, so it's easy to accidentally lose a reference
to an object during testing. Thus, using the GC hook will improve the
stability of one's development environment.

To me, there is no strictly correct answer to the question, which is why
I'm asking for suggestions from you guys.

Regards,
Elias (loke on #lisp)

[Svante v. Erichsen]

Hi!

Post by Elias MÃ¥rtenson
My question is: Should I use trivial-garbage to create a GC hook for the
connection object so that if the user of the library forgets to close the
connection, it will get closed eventually once it's GC'ed?

My first impulse is to say “that is what garbage collection is there for”. It
needs not matter whether the object resides in C or Lisp land or what kind of
resource it is; if the system can identify that it is garbage, it should collect
it.

The alternative would also mean that it is difficult for the system to recover
from the mistake without restart.

Additionally, the API might benefit from a with-async-connection and a
with-open-channel macro.

Yours aye

Svante

[Faré]

On Sat, Jul 4, 2015 at 8:35 AM, Svante v. Erichsen

Hi!

Post by Elias MÃ¥rtenson
My question is: Should I use trivial-garbage to create a GC hook for the
connection object so that if the user of the library forgets to close the
connection, it will get closed eventually once it's GC'ed?

My first impulse is to say “that is what garbage collection is there for”. It
needs not matter whether the object resides in C or Lisp land or what kind of
resource it is; if the system can identify that it is garbage, it should collect
it.
The alternative would also mean that it is difficult for the system to recover
from the mistake without restart.
Additionally, the API might benefit from a with-async-connection and a
with-open-channel macro.

It might be a good idea to have a GC hook, but you might want your GC
hook to log and/or count dropped connections, so you can later detect
and fix the underlying issues. Belt and suspenders.

—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org
It's not "privilege". It's *capital*. It's not forcefully taken away from
others in a negative sum game. It's created, preserved and transmitted in
a positive sum game.

[Vsevolod Dyomkin]

Hi,

Did you consider adding a configuration variable that turns this feature on
and off and can be used to set up environments differently for
dev/test/prod?

---
Vsevolod Dyomkin
[phone]
skype, twitter: vseloved

Post by Elias MÃ¥rtenson
The following question was raised during my development of an asynchronous
library <https://github.com/lokedhs/cl-rabbit-async> I'm currently
building for RabbitMQ.
In summary, the library allows you to create an object of type
ASYNCH-CONNECTION, from which instances of ASYNC-CHANNEL can be
retrieved. A connection holds a reference to all channels that it uses, and
each channel holds a reference to its connection. The connection object has
a pointer to a native CFFI object that for the underlying connection (my
library is built on the RabbitMQ C API).
My question is: Should I use trivial-garbage to create a GC hook for the
connection object so that if the user of the library forgets to close the
connection, it will get closed eventually once it's GC'ed?
- It's a bad idea, since losing the reference to the connection object
means that the program is broken, and silently cleaning up the underlying
connection might hide the fact there is a bug (if the GC doesn't run often
enough I might have hundreds or even thousands of lingering connections)
- On the other hand, it might be a good idea since Lisp developers
often use the REPL to experiment, so it's easy to accidentally lose a
reference to an object during testing. Thus, using the GC hook will improve
the stability of one's development environment.
To me, there is no strictly correct answer to the question, which is why
I'm asking for suggestions from you guys.
Regards,
Elias (loke on #lisp)

[Elias Mårtenson]

On 5 July 2015 at 01:20, Vsevolod Dyomkin <***@gmail.com> wrote:

Did you consider adding a configuration variable that turns this feature on

Post by Vsevolod Dyomkin
and off and can be used to set up environments differently for
dev/test/prod?

No, I didn't. Mainly because I really don't think one option is better for
dev and another for prod.

Based on your replies, it seem like most people would expect a Lisp library
to simply *work* as best as possible rather than do exactly what it's told.

Therefore, I'm starting lean towards using a GC hook here, even though the
complexity of the GC'ed object is much more complex (the connection having
a processing thread, for example) than the only other time I've actually used
this <https://github.com/lokedhs/cl-gss/blob/master/src/cl-gss.lisp#L48>
feature for effect (which only had a simple native object that needed to be
freed).

Regards,
Elias

[Nick Levine]

A somewhat off-topic reply, but if you're working on a CL / RabbitMQ
implementation then do feel free to raid my old CL-RABBIT sources for
working parts, in case any of them turn out to be of use.

http://nicklevine.org/cl-rabbit/

- nick

[Elias Mårtenson]

Post by Nick Levine
A somewhat off-topic reply, but if you're working on a CL / RabbitMQ
implementation then do feel free to raid my old CL-RABBIT sources for
working parts, in case any of them turn out to be of use.

Currently I'm linking to rabbitmq-c which is the C API for RabbitMQ.
Unfortunately this library is not entirely ideal as a base for implementing
a multithreaded async library. Your library avoids this, and I have
definitely been thinking about moving away from rabbitmq-c. If I choose to
do that, I will definitely have a look at your code.

Regards,
Elias

[Paul Tarvydas]

Post by Elias MÃ¥rtenson
The following question was raised during my development of an
asynchronous library <https://github.com/lokedhs/cl-rabbit-async> I'm
currently building for RabbitMQ.

Aside / rhetorical: Are you trying to achieve a certain level of
"efficiency"? If so, read on.

I've been working on flow-based concepts for 25+ years (I use various
languages, incl. CL, as "assemblers").

The FBP model is kinda-like Actors (or CSP), with the embellishment that
siblings cannot "see" or know about one another - they must ask their
common parent to distribute messages.

FBP essentially discards most of the O/S and uses a small handful of
concepts, e.g. a scheduler, components, ports (queues), ready/wait queues.

A very small "kernel" to handle only this set of concepts - entirely
eschewing the use of processes - can be easily built. If you can read
C, then peruse https://github.com/guitarvydas/collate-fbp-classic for
the most bare-bones implementation of these concepts I have come up with
to date (the example is slightly more complicated than necessary,
because the "Collate" problem (page 91 of Paul Morrison's FBP book)
requires the use of bounded buffers). If you don't read C, and are
interested, ask me.

pt

[Faré]

Post by Paul Tarvydas
The FBP model is kinda-like Actors (or CSP), with the embellishment that
siblings cannot "see" or know about one another - they must ask their common
parent to distribute messages.
FBP essentially discards most of the O/S and uses a small handful of
concepts, e.g. a scheduler, components, ports (queues), ready/wait queues.
A very small "kernel" to handle only this set of concepts - entirely
eschewing the use of processes - can be easily built. If you can read C,
then peruse https://github.com/guitarvydas/collate-fbp-classic for the most
bare-bones implementation of these concepts I have come up with to date (the
example is slightly more complicated than necessary, because the "Collate"
problem (page 91 of Paul Morrison's FBP book) requires the use of bounded
buffers). If you don't read C, and are interested, ask me.

Interesting. How does that compare to the Chemical Abstract Machine
and/or Jean-Bernard Stefani's Kell Calculus?

—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org
The ancients stole all our ideas from us. — Mark Twain

[Elias Mårtenson]

Post by Elias MÃ¥rtenson
The following question was raised during my development of an asynchronous
library <https://github.com/lokedhs/cl-rabbit-async> I'm currently
building for RabbitMQ.
Aside / rhetorical: Are you trying to achieve a certain level of
"efficiency"? If so, read on.

No, I'm not. I'm not trying to build something using Erlang-style
async-I/O. I'm merely referring to a library that is thread-safe and which
can use a single connection to RabbitMQ asynchronously from multiple
threads (rabbitmq-c normally enforces a synchronous workflow with all of
its calls being blocking).

That said, I'l still reading on. :-)