Are you an Imager or a Filer?

[Ala'a Mohammad]

Hi,

I'm continually learning Common-lisp and trying to find the best style
that suites me better. I've tried 'an imager' style (cooking a an
image with all required libraries loaded when required), and 'a filer'
style (loading files or systems each time I fire-up a CL
implementation). I'm interested to hear what others use CL. How do
they manage day to day work? how do their preferred style mesh into
their production pipeline (coding, debugging, deployment and
maintenance)? and what makes them prefer one way over another or the
mix if applicable?

Regards,

Ala'a Mohammad.

[Jacob Kozinn]

This is the best thing I've read on the Pro list so far, I think.

I'm a filer, but as I deploy multiple cloud instances, I'm starting to think
about the advantages of imaging.

However, I have little experience with it. Especially on Debian.

Anybody have good resources on imaging that are not obtainable through a
superficial web search?

Post by Ala'a Mohammad
Hi,
I'm continually learning Common-lisp and trying to find the best style
that suites me better. I've tried 'an imager' style (cooking a an
image with all required libraries loaded when required), and 'a filer'
style (loading files or systems each time I fire-up a CL
implementation). I'm interested to hear what others use CL. How do
they manage day to day work? how do their preferred style mesh into
their production pipeline (coding, debugging, deployment and
maintenance)? and what makes them prefer one way over another or the
mix if applicable?
Regards,
Ala'a Mohammad.
_______________________________________________
pro mailing list
http://common-lisp.net/cgi-bin/mailman/listinfo/pro

[Faré]

Post by Jacob Kozinn
This is the best thing I've read on the Pro list so far, I think.
I'm a filer, but as I deploy multiple cloud instances, I'm starting to think
about the advantages of imaging.
However, I have little experience with it. Especially on Debian.
Anybody have good resources on imaging that are not obtainable through a
superficial web search?

For making images, there is
* Xach's buildapp, which is great but sbcl-only
* my own cl-launch, which is portable, and can be complemented with my
command-line-argument library, or Didier Verna's CLON (ported only to
a few platforms).

Both rely on ASDF2, which any modern implementation should provide,
and/or that you can get by yourself.

[ François-René ÐVB Rideau | Reflection&Cybernethics | http://fare.tunes.org ]
Lie, n.:
A very poor substitute for the truth, but the only one
discovered to date.

[Jacob Kozinn]

I haven't looked at cl-launch in a while, but remember it looking promising.
Thanks.

I also looked briefly into ASDF2 but can't remember recognizing any
advantages to moving to it from ASDF. I am sure the advantages are there
though. What am I missing?

Post by Faré

Post by Jacob Kozinn
This is the best thing I've read on the Pro list so far, I think.
I'm a filer, but as I deploy multiple cloud instances, I'm starting to

think

Post by Jacob Kozinn
about the advantages of imaging.
However, I have little experience with it. Especially on Debian.
Anybody have good resources on imaging that are not obtainable through a
superficial web search?

For making images, there is
* Xach's buildapp, which is great but sbcl-only
* my own cl-launch, which is portable, and can be complemented with my
command-line-argument library, or Didier Verna's CLON (ported only to
a few platforms).
Both rely on ASDF2, which any modern implementation should provide,
and/or that you can get by yourself.
[ François-René ÐVB Rideau | Reflection&Cybernethics |
http://fare.tunes.org ]
A very poor substitute for the truth, but the only one
discovered to date.

[Didier Verna]

Post by Faré
For making images, there is
* Xach's buildapp, which is great but sbcl-only
* my own cl-launch, which is portable, and can be complemented with my
command-line-argument library, or Didier Verna's CLON (ported only to
a few platforms).

More precisely, Clon supports SBCL, CMU-CL, CCL, ECL, CLISP, and ABCL
(upcoming release) on Unix (inc. MacOS X). Coming next is support for
ACL and LispWorks, but I'll have to beg for fully functionnal versions
in order to do the porting.

Clon's job is command-line options, that's all. There is however a
couple of things to ease the delivery of standalone applications in the
form of the DUMP macro plus some Makefile rules and scripts in the demo/
directory. Everything is documented.

[Hans Hübner]

Post by Ala'a Mohammad
I'm continually learning Common-lisp and trying to find the best style
that suites me better. I've tried 'an imager' style (cooking a an
image with all required libraries loaded when required), and 'a filer'
style (loading files or systems each time I fire-up a CL
implementation). I'm interested to hear what others use CL. How do
they manage day to day work? how do their preferred style mesh into
their production pipeline (coding, debugging, deployment and
maintenance)? and what makes them prefer one way over another or the
mix if applicable?

Some reasons why one would use images rather than loading up from files:

- Saving on startup times

Nowadays, using images to reduce startup time of the Lisp is often not
worth the trouble. Computers are fast, file systems are fast and most
importantly, restarts are relatively rare with most Lisp systems that
I know. There certainly are cases where every second in terms of
saved startup time is important, and in these cases, writing an image
is a good way to go.

- Reducing the number of files needed to distribute an application, or
distribute binary applications

Some Lisps can dump images that are self contained. Most of them
require some auxiliary files. If you want to distribute your system
without providing source code, a dumped image is the easiest way to
achive that.

- Save transient state across Lisp invocations

As an image includes the full heap state, one can use saved images to
snapshot transient state. That way, one can get around all
serialization issues that database systems try to solve. The approach
is limited in that the commit units are generally rather large.


Personally, I used saved images rarely. I deploy my systems with
source code, so I need to deploy a lot of files anyway, and I use a
database system, so I don't need images as a persistence mechanism.
Startup times don't matter much to me, as I'm usually working with
long-running Lisp instances.

-Hans

[Martin Cracauer]

Post by Ala'a Mohammad
Hi,
I'm continually learning Common-lisp and trying to find the best style
that suites me better. I've tried 'an imager' style (cooking a an
image with all required libraries loaded when required), and 'a filer'
style (loading files or systems each time I fire-up a CL
implementation). I'm interested to hear what others use CL. How do
they manage day to day work? how do their preferred style mesh into
their production pipeline (coding, debugging, deployment and
maintenance)? and what makes them prefer one way over another or the
mix if applicable?

I'm an imager. Even my dotfiles are "converted" into an image. When
I log into a new machine I run a script to set up my dotfiles and the
Lisp part of it loads my personal libraries, changes settings the way
I like and then saves an image.

The major reason is that it's much faster to load the result. I also
make it shut up for good so that starting this image doesn't have any
output except a (customized) prompt.

It also safeguards you against somebody messing with the OS-installed
Lisp.

Martin

[Peter Seibel]

[Once again I forgot to Reply All. Ala, you've seen part of this before.]

I'm a filer. With ASDF and even more so these days with Quicklisp,
it's just easier to load the stuff I need and it doesn't take that
long and it saves me having to spend any mental energy keeping track
of different images.

However, Fare's mention of buildapp reminded me that I use it for some
of my deployed websites, mostly just so I have one thing that is built
and I know won't change, if I accidentally decide to upgrade a library
or tweak some source code for some other reason. (Obviously, I could
also just keep my source for the running app in some distinct
place--now that I'm a bit more adept at git, I might start doing
that.)

-Peter

Post by Ala'a Mohammad
Hi,
I'm continually learning Common-lisp and trying to find the best style
that suites me better. I've tried 'an imager' style (cooking a an
image with all required libraries loaded when required), and 'a filer'
style (loading files or systems each time I fire-up a CL
implementation). I'm interested to hear what others use CL. How do
they manage day to day work? how do their preferred style mesh into
their production pipeline (coding, debugging, deployment and
maintenance)? and what makes them prefer one way over another or the
mix if applicable?
Regards,
Ala'a Mohammad.
_______________________________________________
pro mailing list
http://common-lisp.net/cgi-bin/mailman/listinfo/pro

[Chaitanya Gupta]

Post by Peter Seibel
I'm a filer. With ASDF and even more so these days with Quicklisp,
it's just easier to load the stuff I need and it doesn't take that
long and it saves me having to spend any mental energy keeping track
of different images.

I agree. I have tried using images, but the savings in startup time have
not been worth the hassle, atleast not during development.

Post by Peter Seibel
However, Fare's mention of buildapp reminded me that I use it for some
of my deployed websites, mostly just so I have one thing that is built
and I know won't change, if I accidentally decide to upgrade a library
or tweak some source code for some other reason. (Obviously, I could
also just keep my source for the running app in some distinct
place--now that I'm a bit more adept at git, I might start doing
that.)

Interesting you should mention that. At a previous company, we started
with using only images for deployment. Soon, though, we started running
into problems with this approach -- during early days of development
atleast, it was very painful since we were always patching our live
system every now and then. And image files were huge compared to the
size of our source code, so transferring them was never fun.

Eventually, what we settled on was to put the source code on our
production server(s), but build an image whenever a new version was
checked out -- which would ensure quick startup if ever the server was
rebooted or the application went down.

A nifty thing which went along with this was a system of patches which
could be applied to running production images without restarting them.
When we had to push a bug fix to production, we could have done this:

- update the source in the maintenance branch,
- update source on production server(s),
- shutdown, rebuild, and boot our Lisp system again,

However, shutting down a running Lisp image was not very desirable, so,
thanks to the nature of Common Lisp, we managed to go with this instead:

- upload a patch (a .lisp source file) to a designated directory on the
server
- `load` all the patches in the running Lisp system from said directory
(we would ensure that each patch wouldn't break things if load-ed more
than once)
- ensure that these patches are loaded when the application is started
(to account for restarts)

(I hope I've got everything right, this was all over two years ago)

All in all, a combination of images and files worked out fairly well for
our production system.

Chaitanya

[Drew Crampsie]

I use files and quicklisp when developing, but deliver and provision
binary images.

Most of my apps are web based, and i run the lisp image behind runit,
a service manager that handles logging as well as startup and
shutdown, and restarting the image if it crashes. It also makes
rolling back a release trivial, modulo database schema changes.

I don't use anything like cl-launch or buildapp, if only because these
tools didn't exist in a usable form when i started, and i haven't had
a good reason to change.

Quicklisp, OTOH, has changed the way i work significantly. With the
exception of production instances, most of my code is deployed via a
load.lisp script that explicitly lists and loads dependencies from QL,
where before libraries were kept in version control. I'm also using
more libraries these days.

So, i'm both, but primarily a filer these days.

Cheers,

drewc

Post by Ala'a Mohammad
Hi,
I'm continually learning Common-lisp and trying to find the best style
that suites me better. I've tried 'an imager' style (cooking a an
image with all required libraries loaded when required), and 'a filer'
style (loading files or systems each time I fire-up a CL
implementation). I'm interested to hear what others use CL. How do
they manage day to day work? how do their preferred style mesh into
their production pipeline (coding, debugging, deployment and
maintenance)? and what makes them prefer one way over another or the
mix if applicable?
Regards,
Ala'a Mohammad.
_______________________________________________
pro mailing list
http://common-lisp.net/cgi-bin/mailman/listinfo/pro

[Scott L. Burson]

I'd say I'm 70% imager, 30% filer. I usually save an image with
libraries I'm using, but then load more stuff on top of that when I
start.

-- Scott

[Andreas Fuchs]

Post by Ala'a Mohammad
I'm interested to hear what others use CL. How do
they manage day to day work? how do their preferred style mesh into
their production pipeline (coding, debugging, deployment and
maintenance)? and what makes them prefer one way over another or the
mix if applicable?

For loading required software, I'd say 100% filer. ASDF and Quicklisp
make operating on files to easy for me that I just don't want to waste
any energy thinking about what definitions are in the image and which
I've saved in a (source-controlled) file somewhere - the 30 seconds I
spend waiting for software to load are nothing compared to an
afternoon spent debugging a function whose definition in the image
differs from what is in the file containing its (alleged) source
code... Or worse, functions/variables that are present only in the
image, and not in any file.

When it comes to throw-away code, though, I usually just type it into
the repl or the slime-scratch buffer. But: once throwaway stuff starts
becoming useful, I use quickproject and paste the latest definitions
into a source file and check that into git ASAP. Of course, this step
has the potential to go wrong in the same way I mentioned above: if I
manage to forget a definition, it's debug time again. Fortunately,
that happens early enough in the source's lifetime that it's not hard
to find the definition again.

As for deploying, I have started making quicklisp trees containing the
requisite software and a git checkout of my project. You can still
incrementally load stuff, and you can be pretty certain that what you
tested truly is what you deployed (-:

Hope that's useful,

[Luke Crook]

It causes me less heartburn during development being a Filer as I use several
foreign libraries. But I deploy executables.

-Luke

[Nick Levine]

(Sorry I haven't yet had time to read other responses: am on the road
and buried under both 6 inches of Ohio snow and an end month
deadline. So apologies if I'm restating other people.)

I'm both.

Development uses images which are primed to check that the system is
up to date (and to correct that as appropriate) on startup. That way
we get rapid startup but everything compiled up to date.

Occasionally it doesn't look too good (say, an annoying number of GF
signatures changed since the image was saved) and the effort of
hand-holding the start-time system reload outweighs the effort of
rebuilding. Then I quit and rebuild. Mostly when I rebuild I don't
force the compile, but if (say) I've been messing with a pervasive
macro I'll go all the way.

We've recently introduced the change that force compilation does a
force compile of the whole system twice. This allows us to generate
warnings on incorrect calls to functions defined later in the
system. We aim for the compilation to be warning-free.

The product we ship can't do the above (LispWorks "delivered" image:
royalty free but no compile-file) but then we wouldn't want to. The
applications come with a patch loader (startup check for fasls in a
particular directory).

In case this is a relevant data point: we're developing three
applications, 70k lines of code, with a shared library of about 15k
lines.

- nick

[Luís Oliveira]

Hello Nick,

Post by Nick Levine
Development uses images which are primed to check that the system is
up to date (and to correct that as appropriate) on startup. That way
we get rapid startup but everything compiled up to date.

Can you describe in more detail how that works?

Cheers,

[Nick Levine]

From: =?ISO-8859-1?Q?Lu=EDs_Oliveira?= <luismbo-***@public.gmane.org>
Date: Fri, 21 Jan 2011 10:53:51 +0000

Hello Nick,

Post by Nick Levine
Development uses images which are primed to check that the system
is up to date (and to correct that as appropriate) on
startup. That way we get rapid startup but everything compiled up
to date.

Can you describe in more detail how that works?

Sure. The detail is a bit mucky and probably unilluminating, because I
postpone the compilation until after the LW GUI had finished firing
up. (If I weren't so picky, I could hand a restart-function to
save-image.) Here are the excerpts from our build script which drive
this:

(defun application-load (&key compile-only)
(let ((defsys (truename (relative-path "code/defsys.lisp"))))
(load defsys))
(compile-system "PROFILER-PLUS" :load (not compile-only)
:force (find "-force" sys:*line-arguments-list* :test 'string=)))

(application-load)

[...]

(define-action "Initialize LispWorks Tools" "Reload PPlus"
(lambda (screen)
(declare (ignore screen))
(when-let (listener (mp:find-process-from-name "Listener 1"))
(mp:process-interrupt listener
(lambda ()
(application-load)
;; Now activate the app.
(pp::activate nil)
))))
:after "Run the environment start up functions")

LispWorks has these beasts called "action lists": hooks by another
name. What the above is saying is: while initializing the LW tools,
after running LW's internal startup functions, execute this bit of
code which as you see recompiles the system (maybe forcing it,
depending on the -force line argument) and then activates the
application so I can play with it as I hack.

- nick

[Luís Oliveira]

Post by Nick Levine
(defun application-load (&key compile-only)
 (let ((defsys (truename (relative-path "code/defsys.lisp"))))
   (load defsys))
 (compile-system "PROFILER-PLUS" :load (not compile-only)
                 :force (find "-force" sys:*line-arguments-list* :test 'string=)))

I'm not familiar with the details of LW's defsystem. How is this
different from loading the application from fasls, speed-wise?

[Raffael Cavallaro]

Post by Luís Oliveira
I'm not familiar with the details of LW's defsystem. How is this
different from loading the application from fasls, speed-wise?

I'm sure Martin Simmons of LispWorks will correct me if I'm mistaken here, but LW's defsystem can be used to load lisp code, and/or compile and load lisp code, and/or load precompiled fasls, so it spans the whole range.

warmest regards,

Ralph

Raffael Cavallaro
raffaelcavallaro-BUHhN+***@public.gmane.org

[Luís Oliveira]

On Fri, Jan 21, 2011 at 7:50 PM, Raffael Cavallaro

Post by Raffael Cavallaro
I'm sure Martin Simmons of LispWorks will correct me if I'm mistaken here, but LW's
defsystem can be used to load lisp code, and/or compile and load lisp code, and/or
load precompiled fasls, so it spans the whole range.

My question is: if you're going to load all of the application's
FASLs, what difference does it make that you loaded a dumped image
with some previous version of the application? What am I missing?

[Nick Levine]

My question is: if you're going to load all of the application's
FASLs,

You're not. load-system (or compile-system with :load t) will do the
minimum it has to and leave the rest alone.

what difference does it make that you loaded a dumped image
with some previous version of the application? What am I missing?


- n

[Luís Oliveira]

Post by Luís Oliveira
My question is: if you're going to load all of the application's
FASLs, what difference does it make that you loaded a dumped image
with some previous version of the application? What am I missing?

What I was missing is that when you dump an image, defsystem
information about the files' modification times is kept, thus only
modified files (and their dependents) will be loaded. *facepalm*

Cheers,

[William Halliburton]

The system Ioad facility I have developed, on its first run on a particular
system, creates a core file containing all the support libraries needed for
the system, and then starts SBCL using that core file, proceeding to ASDF
load the system proper. On subsequent starts, the first core creation step
is skipped, saving much time, in most cases making the systems start
instantly since no compilation needs to occur.

Whenever site-lisp is changed, then the cores need to be removed and will
automatically be recreated when the system restarts. In the core is stored
the head git SHA commit id taken from site-lisp at the time of core creation
in order to detect a core to source code discrepancy when the cores are
ran.

Post by Ala'a Mohammad
Hi,
I'm continually learning Common-lisp and trying to find the best style
that suites me better. I've tried 'an imager' style (cooking a an
image with all required libraries loaded when required), and 'a filer'
style (loading files or systems each time I fire-up a CL
implementation). I'm interested to hear what others use CL. How do
they manage day to day work? how do their preferred style mesh into
their production pipeline (coding, debugging, deployment and
maintenance)? and what makes them prefer one way over another or the
mix if applicable?
Regards,
Ala'a Mohammad.
_______________________________________________
pro mailing list
http://common-lisp.net/cgi-bin/mailman/listinfo/pro

[William Lederer]

For production, I build images, as some of the production instances
don't have either NFS or the SBCL build chain.

But for development, I am almost always in slime, being a filer.

Additionally, the production environment does not lend itself to
working with a running image.

On Sun, Jan 23, 2011 at 4:51 AM, William Halliburton

Post by William Halliburton
The system Ioad facility I have developed, on its first run on a particular
system, creates a core file containing all the support libraries needed for
the system, and then starts SBCL using that core file, proceeding to ASDF
load the system proper. On subsequent starts, the first core creation step
is skipped, saving much time, in most cases making the systems start
instantly since no compilation needs to occur.
Whenever site-lisp is changed, then the cores need to be removed and will
automatically be recreated when the system restarts. In the core is stored
the head git SHA commit id taken from site-lisp at the time of core creation
in order to detect a core to source code discrepancy when the cores are
ran.

Post by Ala'a Mohammad
Hi,
I'm continually learning Common-lisp and trying to find the best style
that suites me better. I've tried 'an imager' style (cooking a an
image with all required libraries loaded when required), and 'a filer'
style (loading files or systems each time I fire-up a CL
implementation). I'm interested to hear what others use CL. How do
they manage day to day work? how do their preferred style mesh into
their production pipeline (coding, debugging, deployment and
maintenance)? and what makes them prefer one way over another or the
mix if applicable?
Regards,
Ala'a Mohammad.
_______________________________________________
pro mailing list
http://common-lisp.net/cgi-bin/mailman/listinfo/pro

_______________________________________________
pro mailing list
http://common-lisp.net/cgi-bin/mailman/listinfo/pro

[Attila Lendvai]

lately i'm 99% imager both for development and deployment. most of the
time i start from an image that contains all the dependencies we have
except the hu.dwim.* stuff and swank.

i don't initiate a reload when the image is started. if i need to do
nontrivial work on a dependency, then i just start slime with a
vanilla SBCL.


startup time does matter for me because:

- we have 80+ dependencies, a full recompile takes long minutes even
on new hardware

- it's easy to lose track of what's the state of the heap and i've
debugged ghosts often enough to just simply restart and reload as a
first step whenever there's something that baffles me.

- CL has complex load-time/compile-time semantics, and some
components (e.g. ASDF) are quite naive regarding this which doesn't
help. i have often experienced that what i've done to the heap using
redefinitions cannot be loaded from files out of the box. long
sessions without a reload mean longer cleanups afterwards when loading
from files fail.


relevant sources (http://dwim.hu/darcsweb/darcsweb.cgi):

- hu.dwim.build - the CL build app, similar to Xach's buildapp. it
can build itself into an executable.

- hu.dwim.environment - emacs/ for the way emacs/slime is set up;
etc/server-scripts for the posix services of production images

production.lisp files throughout our projects. they add something to
the way the production images are started.