Discussion:
CDR proposal: compiled-file-p
Sam Steingold
2012-05-15 15:54:50 UTC
Permalink
In 2004 I submitted a CLRFI which was then accepted as CLRFI-2.
It specified a new function COMPILED-FILE-P which was supposed to be
useful for system construction utilities, e.g., asdf:

Function
(compiled-file-p file-name) ==> valid-p

Returns
true if the file appears to be a valid compiled file
(i.e., exists, is readable, and the contents appears to be
valid for this implementation),
false otherwise.

Implementations are required to inspect the contents
(e.g., checking just the pathname type is not sufficient).
Although the completeness of the inspection is not required,
this function should be able to detect,
e.g., file format changes between versions.

I re-submitted the spec to the CDR editors a week ago.
No response so far.

The detailed html spec is attached.
Faré
2012-05-15 16:58:29 UTC
Permalink
Note that for the sake of being able to simultaneously support
several implementations on the same computer (or network),
asdf has taken the route of trying to segregate fasl files and
any other build output (e.g. C files and Lisp files generated by CFFI)
to directories the name of which depend on the implementation,
its version, the operating system and architecture, etc.

XCVB hopes to eventually go further and identify build output
by cryptographic digest of input,
including underlying build tools as well as input dependencies.

This segregation makes such output-inspection function much less useful.

—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org
You think war is economically beneficial? Then why share those benefits with
dirty foreigners? Let's have a civil war — A war we're sure to win!
Post by Sam Steingold
In 2004 I submitted a CLRFI which was then accepted as CLRFI-2.
It specified a new function COMPILED-FILE-P which was supposed to be
Function
(compiled-file-p file-name) ==> valid-p
Returns
 true  if the file appears to be a valid compiled file
 (i.e., exists, is readable, and the contents appears to be
 valid for this implementation),
 false  otherwise.
Implementations are required to inspect the contents
(e.g., checking just the pathname type is not sufficient).
Although the completeness of the inspection is not required,
this function should be able to detect,
e.g., file format changes between versions.
I re-submitted the spec to the CDR editors a week ago.
No response so far.
The detailed html spec is attached.
--
Sam Steingold (http://sds.podval.org/) on Ubuntu 12.04 (precise) X 11.0.11103000
http://www.childpsy.net/ http://mideasttruth.com http://ffii.org
http://www.memritv.org http://jihadwatch.org http://camera.org http://memri.org
You cannot fire me. Slaves are not fired. Slaves are sold.
_______________________________________________
pro mailing list
http://lists.common-lisp.net/cgi-bin/mailman/listinfo/pro
Sam Steingold
2012-05-15 20:26:28 UTC
Permalink
Post by Faré
Note that for the sake of being able to simultaneously support
several implementations on the same computer (or network),
asdf has taken the route of trying to segregate fasl files and
any other build output (e.g. C files and Lisp files generated by CFFI)
to directories the name of which depend on the implementation,
its version, the operating system and architecture, etc.
I know, and I am not sure this is TRT.
This approach leads to a propagation of directories and files which
might uselessly duplicate each other on one hand, while not providing
the appropriate flexibility on the other.
E.g., CLISP compiles to platform-independent byte-code whose format
changes relatively rarely, so having separate directories for
clisp-ver-arch-os is a total waste, which also gives the users an
impression that they cannot distribute clisp-fas files to different
arches.

I just added abi-version to the spec to address this issue.
Faré
2012-05-15 21:21:10 UTC
Permalink
Post by Sam Steingold
Post by Faré
Note that for the sake of being able to simultaneously support
several implementations on the same computer (or network),
asdf has taken the route of trying to segregate fasl files and
any other build output (e.g. C files and Lisp files generated by CFFI)
to directories the name of which depend on the implementation,
its version, the operating system and architecture, etc.
I know, and I am not sure this is TRT.
This approach leads to a propagation of directories and files which
might uselessly duplicate each other on one hand, while not providing
the appropriate flexibility on the other.
1) there is no fully reliable way to detect beforehand whether
the file will be actual portable bytecode, or something that has
platform-specific information, not only because of magic escapes
(if they exist), but also because of cffi-extracted constants or
other clever read-time, macroexpansion-time or compile-time evaluations.

2) I don't know what you mean by "not providing the appropriate flexibility".
ASDF (and XCVB) make it fully user-configurable, and seggregate according
to a wide variety of criteria that can be further compounded with e.g.
hostname.
Post by Sam Steingold
E.g., CLISP compiles to platform-independent byte-code whose format
changes relatively rarely, so having separate directories for
clisp-ver-arch-os is a total waste, which also gives the users an
impression that they cannot distribute clisp-fas files to different
arches.
How do you avoid the issues with embedded cffi constants?
Certainly, a clever enough system could track that no non-portable
operations happened during a compilation and validate fasls as portable
after-the-fact. That's more clever than either CLISP or ASDF is.
Or you could be optimistic until things break badly.
Which is frankly a dumb way to save disk space --
but ASDF allows you to configure your system that way if you desire.

If you want some non-trivial feature, that would be more useful:
a way to track whether a file was compiled portably, and/or a way
to declare that it should and to trap it if it doesn't.

As for namespace issues.
Should CDR's be available through (require :cdr3)
with their contents in (find-package :cdr3) ?

—♯ƒ • François-René ÐVB Rideau •Reflection&Cybernethics• http://fare.tunes.org
It's possible to program a computer in English. It's also possible to make an
airplane controlled by reins and spurs. — John McCarthy

Pascal Costanza
2012-05-15 17:41:10 UTC
Permalink
Content preview: Hi, I am one of the CDR editors, but I haven't seen your email.
I have double-checked the email forwarding just now, and it seems to work.
I cannot check anymore whether your email accidentally landed in my spam
folder, because it has a retention period of a week. [...]

Content analysis details: (-98.2 points, 5.0 required)

pts rule name description
---- ---------------------- --------------------------------------------------
-0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at http://www.dnswl.org/, no
trust
[217.70.183.195 listed in list.dnswl.org]
1.8 URIBL_BLACK Contains an URL listed in the URIBL blacklist
[URIs: childpsy.net]
-100 USER_IN_WHITELIST From: address is in the user's white-list
Archived-At: <http://permalink.gmane.org/gmane.lisp.cl-pro/678>

Hi,

I am one of the CDR editors, but I haven't seen your email. I have double-checked the email forwarding just now, and it seems to work. I cannot check anymore whether your email accidentally landed in my spam folder, because it has a retention period of a week.

Could you try to resend your CDR proposal, and then send a notification to my private address as well? If I then still don't receive your proposal via the editors email address, I have to dig deeper to try to see where the problem is.

In any case, I will process the CDR proposal next weekend.

Thanks a lot,
Pascal
Post by Sam Steingold
In 2004 I submitted a CLRFI which was then accepted as CLRFI-2.
It specified a new function COMPILED-FILE-P which was supposed to be
Function
(compiled-file-p file-name) ==> valid-p
Returns
true if the file appears to be a valid compiled file
(i.e., exists, is readable, and the contents appears to be
valid for this implementation),
false otherwise.
Implementations are required to inspect the contents
(e.g., checking just the pathname type is not sufficient).
Although the completeness of the inspection is not required,
this function should be able to detect,
e.g., file format changes between versions.
I re-submitted the spec to the CDR editors a week ago.
No response so far.
The detailed html spec is attached.
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xml:lang="en" xmlns="http://www.w3.org/1999/xhtml">
<head>
<title>Function COMPILED-FILE-P</title>
<meta name="author" value="Sam Steingold"/>
</head>
<body>
<h1>Function <code>COMPILED-FILE-P</code></h1>
<h2>Author</h2>
<p>Sam Steingold</p>
<h2>Related</h2>
<p>ANSI Common Lisp standard function <code>compile-file</code>.</p>
<h2>Abstract</h2>
<p>A facility to determine whether a file is a valid compiled file for
the specific implementation.</p>
<h2>Rationale</h2>
<p>Build tools, like <code>defsystem</code> or <code>asdf</code>,
have to determine whether a file needs to be recompiled.</p>
<p>Obviously, when the compiled file is older than the
source file, recompilation is in order.</p>
<p>Alas, there are other situations when this might be necessary, e.g.,
when the implementation changes the compiled file format or when two
implementations use the same name for their compiled files
(<code>.fasl</code> is used by both <code>SBCL</code> and <code>ACL</code>).
</p>
<h3>Current Practice</h3>
<p>Implementation-dependent.</p>
<h3>Cost of adoption</h3>
<p>Probably tiny: an implementation must be able to check for compiled
file validity, so all it takes is to export the necessary
functionality, e.g.:</p>
<pre id="compiled-file-p-clisp">
#+clisp
(defun compiled-file-p (file-name)
(with-open-file (in file-name :direction :input :if-does-not-exist nil)
(and in (char= #\( (peek-char nil in))
(let ((form (ignore-errors (read in nil nil))))
(and (consp form)
(eq (car form) 'SYSTEM::VERSION)
(null (nth-value 1 (ignore-errors (eval form)))))))))
</pre>
<h3>Cost of non-adoption</h3>
<p>Users will suffer random errors when trying to load invalid binary
files.</p>
<h2>Specification</h2>
<p>Function</p><pre>
</pre>
<p>Returns</p><dl>
<dt><code>true</code></dt><dd>if the file appears to be a valid compiled file
(i.e., exists, is readable, and the contents appears to be
valid for this implementation),</dd>
<dt><code>false</code></dt><dd>otherwise.</dd></dl>
<p>Implementations are required to inspect the contents
(e.g., checking just the pathname type is not sufficient).
Although the completeness of the inspection is not required,
this function should be able to detect,
e.g., file format changes between versions.</p>
<h3>Exceptional situations</h3> <ul>
<li>signals an error of type <code>type-error</code>
when the argument is not a <em>pathname designator</em>.</li>
</ul>
<h3>Examples</h3>
<pre>
</pre>
<h2>Reference Implementation</h2>
<p>See <a href="#compiled-file-p-clisp">above</a>.</p>
<h2>History</h2>
<p>This used to be <strong>CLRFI-2</strong> (in 2004).</p>
<h2>Notes</h2>
<p>The trivial implementation:</p>
<pre>
(defun compiled-file-p (file-name)
(not (nth-value 1 (ignore-errors (load file-name)))))
</pre>
<p>is wrong because,</p>
<ol>
even when <code>foo.lisp</code> contains calls to <code>error</code>,<pre>
(compiled-file-p (compile-file "foo.lisp"))
</pre>should still return <code>T</code>.</li>
<li>this is not side-effect-free, i.e., this may define new functions and
macros (or, worse yet, redefine some existing functions and macros or
execute some malicious code).</li></ol>
</body>
</html>
--
Sam Steingold (http://sds.podval.org/) on Ubuntu 12.04 (precise) X 11.0.11103000
http://www.childpsy.net/ http://mideasttruth.com http://ffii.org
http://www.memritv.org http://jihadwatch.org http://camera.org http://memri.org
You cannot fire me. Slaves are not fired. Slaves are sold.
_______________________________________________
pro mailing list
http://lists.common-lisp.net/cgi-bin/mailman/listinfo/pro
--
Pascal Costanza
Sam Steingold
2012-05-15 19:29:49 UTC
Permalink
Content preview: Hi, my e-mail to which you replied was CCed to editors-dmfRLhNpelYzOtNqt8oInuG/***@public.gmane.org
why doesn't it qualify as the resubmission? -- Sam Steingold (http://sds.podval.org/)
on Ubuntu 12.04 (precise) X 11.0.11103000 http://www.childpsy.net/ http://pmw.org.il
http://openvotingconsortium.org http://ffii.org http://camera.org http://palestinefacts.org
http://memri.org If you cannot improve on silence, keep it. [...]

Content analysis details: (-98.9 points, 5.0 required)

pts rule name description
---- ---------------------- --------------------------------------------------
-0.7 RCVD_IN_DNSWL_LOW RBL: Sender listed at http://www.dnswl.org/, low
trust
[209.85.216.52 listed in list.dnswl.org]
1.8 URIBL_BLACK Contains an URL listed in the URIBL blacklist
[URIs: childpsy.net]
-100 USER_IN_WHITELIST From: address is in the user's white-list
0.0 FREEMAIL_FROM Sender email is commonly abused enduser mail provider
(sam.steingold[at]gmail.com)
-0.0 SPF_PASS SPF: sender matches SPF record
0.0 T_DKIM_INVALID DKIM-Signature header exists but is not valid

Hi,
my e-mail to which you replied was CCed to editors-dmfRLhNpelYzOtNqt8oInuG/***@public.gmane.org
why doesn't it qualify as the resubmission?
--
Sam Steingold (http://sds.podval.org/) on Ubuntu 12.04 (precise) X 11.0.11103000
http://www.childpsy.net/ http://pmw.org.il http://openvotingconsortium.org
http://ffii.org http://camera.org http://palestinefacts.org http://memri.org
If you cannot improve on silence, keep it.
Pascal Costanza
2012-05-15 20:49:10 UTC
Permalink
Content preview: On 15 May 2012, at 21:29, Sam Steingold wrote: > Hi, > my
e-mail to which you replied was CCed to editors-dmfRLhNpelYzOtNqt8oInuG/***@public.gmane.org > why doesn't
it qualify as the resubmission? It does qualify. As I said, I will process
the CDR submission in any case. However, I also see the resubmission and
your last email only through the Pro mailing list. Could you send an email
just to editors-***@public.gmane.org? [...]

Content analysis details: (-100.0 points, 5.0 required)

pts rule name description
---- ---------------------- --------------------------------------------------
-0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at http://www.dnswl.org/, no
trust
[217.70.183.195 listed in list.dnswl.org]
-100 USER_IN_WHITELIST From: address is in the user's white-list
Archived-At: <http://permalink.gmane.org/gmane.lisp.cl-pro/681>
Post by Sam Steingold
Hi,
why doesn't it qualify as the resubmission?
It does qualify. As I said, I will process the CDR submission in any case. However, I also see the resubmission and your last email only through the Pro mailing list. Could you send an email just to editors-***@public.gmane.org?


Pascal

--
Pascal Costanza
Sam Steingold
2012-05-15 20:54:27 UTC
Permalink
Content preview: > Could you send an email just to > editors-dmfRLhNpelYzOtNqt8oIng-***@public.gmane.org?
done (twice, with different attachments) -- Sam Steingold (http://sds.podval.org/)
on Ubuntu 12.04 (precise) X 11.0.11103000 http://www.childpsy.net/ http://truepeace.org
http://dhimmi.com http://www.PetitionOnline.com/tap12009/ http://ffii.org
UNIX is as friendly to you as you are to it. Windows is hostile no matter
what. [...]

Content analysis details: (-98.9 points, 5.0 required)

pts rule name description
---- ---------------------- --------------------------------------------------
-0.7 RCVD_IN_DNSWL_LOW RBL: Sender listed at http://www.dnswl.org/, low
trust
[209.85.216.52 listed in list.dnswl.org]
1.8 URIBL_BLACK Contains an URL listed in the URIBL blacklist
[URIs: childpsy.net]
-100 USER_IN_WHITELIST From: address is in the user's white-list
0.0 FREEMAIL_FROM Sender email is commonly abused enduser mail provider
(sam.steingold[at]gmail.com)
-0.0 SPF_PASS SPF: sender matches SPF record
0.0 T_DKIM_INVALID DKIM-Signature header exists but is not valid
Archived-At: <http://permalink.gmane.org/gmane.lisp.cl-pro/682>
Post by Pascal Costanza
Could you send an email just to
done (twice, with different attachments)
--
Sam Steingold (http://sds.podval.org/) on Ubuntu 12.04 (precise) X 11.0.11103000
http://www.childpsy.net/ http://truepeace.org http://dhimmi.com
http://www.PetitionOnline.com/tap12009/ http://ffii.org
UNIX is as friendly to you as you are to it. Windows is hostile no matter what.
Loading...