Inline clarifications.
Post by Robert SmithI don't mean any other language, but I do mean any other relatively modern
one. Rust, JavaScript, Clojure are maybe some examples off the top of my
head.
Hans gave a good example about Clojure. Javascript, with Node.js/npm, is
in an even worse position.
Either modules are somehow first class (by way of having
modules-as-function-dictionaries), or the respective
packaging/loading/whatever systems will at least let you know that about
unsatisfied/mismatched versions.
I think in light of this discussion you can boil down one's opinion on
1. Lisp+ecosystem and associated libraries are problematic in that it's
not possible to load multiple versions of a library due to the way CL
packages work. (Packages or systems aren't first-class modules. Namespace
names are global and aren't late-bound.)
Hans described the QA problem with loading two versions of the "same"
library in the same process: if data structures from one version are fed
into the other version, all sorts of problems can happen, and worse, they
can happen without noticing. Think of what can happen if the semantics of a
string or integer slot changes.
See below.
Post by Robert Smith2. There's something like a SocialProblem (TM) where Lispers lack (a)
versioning discipline, (b) dependency versioning discipline which
exacerbates issues like inadvertent multiple loading, or incompatible
loading of libraries.
Semantic versioning won't help you without automatic tools to determine if
the API of a library changed in an incompatible way, e.g. if a function
removed a keyword arg, or changed the type of an existing arg.
I would say that semvers should in principle help you detect
incompatibilities, but as you hinted with "illusion", maybe these in
practice can't be trusted or relied upon generally.
Post by Robert Smith3. There's actually no issue in principle: inconsistencies need to "just"
get fixed by the library vendor/user, or better tooling needs to be used to
track your project's direct and transitive dependencies.
It's my personal opinion, as is evident in previous messages, that I think
1 & 2 are issues.
You didn't give an example of how exactly those two things are a problem,
and how adding those features would solve the problem(s).
#1: Assuming that there isn't incompatible passing of data, if A depends on
B1 and C depends on B2, and in isolation A-B1 works/passes tests, and C-B2
works/passes tests, then A-B1 and C-B2 loaded together (assuming a
non-existent Lisp) should work fine. The problem being addressed is lack of
synchrony in the universe of Lisp modules.
(My first-order thought is that even incompatible data wouldn't be an issue
if those data types are properly encapsulated in classes or structures, by
virtue of the fact that those types become distinct by way of different
names. I envision that only untyped things could cause problems, like
[ap]-lists, strings, etc.)
#2: If there was diligence is properly versioning libraries, and diligence
in specifying dependency version requirements, then detection of issues
could happen. This is solving recognition of a possible issue.
Post by Robert Smith* Lisp is old. Some libraries go back to the 80s.
* Not everything is very strictly maintained. Some useful libraries
haven't been updated for some 5-10 years.
What useful libraries that actually need updates haven't been updated ?
I'll get back to this when I'm at my computer next.
Post by Robert Smith* Like many other languages, Java is continuously updated, and as are the
libraries of its large standard set. While Java/etc libraries certainly can
and do depend on third party libraries, the majority of depended-on
libraries are standard ones. This is obviously not the case for Lisp.
Just look into Maven packages. Large parts of the Java standard library
are obsolete and people use third party libraries(just like with CL). In
some cases those libraries make it back into the standard(
http://www.joda.org/joda-time/), but that's rare.
* Because library distribution wasn't a relatively solved problem until
recently in Lisp history, some library authors unfortunately include copies
of Lisp libraries as a part of their library tree.
That's very rare, in my experience. Do you have examples ?
* Hot patching libraries makes this all a ton more complicated in
principle.
Where did you see this?
For these last two questions, I don't have the examples off hand and
unfortunately it was at a company I no longer work at.
Including: ALEXANDRIA has been included directly before. I don't remember
which library it was. I only detected it because of function redefinition
warnings on first uncached build. There were other examples but I don't
recall.
Hot Patching: I think (??) some linear algebra library was being hot
patched for some specific use of that library.
* Visible libraries are global to the entire Lisp universe. (Unlike Python
Post by Robert Smithwhere you can, just within a file, import and late-bind a namespace.)
* Tangential: the whole name/nickname collision issue. Package-local
nicknames are a language extension which could solve this, even if not
(yet?) universally used.
Local nicknames would make certain things more convenient, but what does
that have to do with the original question, versioning ?
It was tangential to packages and names of things.
Post by Robert SmithMaking it easier to work in this large, old, and diverse ecosystem of
implementations and libraries seems important to me, and versioning is one
useful and tangible way to make things easier.
Robert
I don't understand when you say this is "not an issue that one runs into
writing in another language and associated ecosystem". AFAIK, in Java, you
can have only one version of a given package loaded into the JVM at any one
time. Am I missing something here?
-- Scott
I really honestly felt that this rebuttal was very unhelpful. I basically
- Lisp isn't designed for this kind of use.
- Since Lisp wasn't designed for this, the problem is not Lisp, it's
your use of Lisp.
- To solve the problem (which is not Lisp's), fix (third party)
inconsistencies yourself. Because it's not a Lisp problem, two systems
depending on two Lisp-simultaneous-incompatible versions of a system is a
problem. (Even though each system+dependency combination is perfectly valid
in isolation.)
- Or, in the end, change your entire operating system to a completely
different one with completely different philosophies.
This kind of response is just very practically a non-starter for most
people. Maybe that means Lisp isn't the right tool for much of the world,
like startups, then. Which advances my point, I think, in the original
Quora post. (But I admit this thread is about package or system versioning,
not the original Quora point.)
I wrote some stuff inline below.
Post by Robert SmithPost by Alessio StallaThe stance on packages found in the mentioned Quora post is based on the
old misconception about packages being modules, or "software packages"
in
Post by Robert SmithPost by Alessio Stallathe Linux distribution sense. They're not. They are really just
namespaces,
Post by Robert SmithPost by Alessio Stallacontainers of symbols. So it does not make any sense, to me, that they
have
Post by Robert SmithPost by Alessio Stallaversions. Software has versions, not names.
Systems are what should get versioned but systems are not what are
referred
Post by Robert Smithto by source files. Systems largely coordinate the compilation and
loading
Post by Robert Smithof files. Source files generally have no notion of a system, actually,
which
Post by Robert Smithmay be what is ultimately problematic. Packages are referred to by source
files, however. When I said packages, I meant it, and it was not a
conflation with the notion of a system.
Why should source files know about systems?
Why should packages know about systems?
Maybe they should, maybe they shouldn't. But the crappy
packages-as-namespaces system isn't cutting it for large programs, and
modules usually naturally have an associated namespace, but there's no
formal correspondence between packages and systems. But more often than not
there is an implicit association, judging by how most modern Common Lisp
libraries are written. So maybe there's value in taking advantage of that
correspondence.
Post by Robert SmithAt the companies I have worked in, we rake in 100s of systems, which
consist
Post by Robert Smithof n*100s of systems where n is the average number of packages per
system.
Post by Robert SmithThese version collisions happen enough, and we are lucky when they're not
silent.
Was it in CL? Was it all in the same process? CL is not designed for
these kinds of things.
CL in the same process.
CL very clearly wasn't designed for a time where 100s of packages would be
available and simultaneously usable.
But I'm not ready to concede and say "well, Lisp is only good for five or
six massive packages or systems written by a few guys, like the good ol'
days".
Post by Robert SmithOften enough, it's an extremely subtle issue that system A depends
on {system B at time T1} and system C depends on {system B at time T2},
where T1 and T2 correspond to different versions of system B, even if not
explicitly labeled as such.
A and C are incompatible. You need to fix A, B and/or C.
The build system can record this incompatibility and issue an early error
when you try to build them together. It can't fix A, B and/or C for you.
A and C are incompatible only because of this whole thing we are talking
about. Obviously they're incompatible. That's the whole problem. Saying
"Fix A, B, C" is, to me, declaring that this whole thing isn't a problem,
and the problem is library writers. Why can't A and C depend on whatever
they want at some time? They of course can, but it's non-sense to think
people will rename their namespaces every iteration of their software.
That's just not how Lisp code has been written.
Post by Robert SmithLet's suppose we simplify the problem however and assume that B has been
properly versioned in time, and systems A and C refer properly to the
correct version of system B as a dependency. Even then, they will not be
able to be simultaneously loaded in a functionally correct fashion. (They
*can* both be loaded, but you'll do some clobbering of state along the
way.)
No. The build system can error out and
tell you the easy way that you have to fix your bugs
before you discover it the hard way
The build system erroring would be a step above the status quo. But maybe
there's not actually a logical error. We are only considering it an error
because of the way packages work in Lisp, no?
Post by Robert SmithWhether we think about modules as systems or as packages or as whatever,
the
Post by Robert Smithproblem would need to be addressed at the package level, if we are not
assuming that we can't synchronize the universe of people writing Common
Lisp.
The packages have nothing to do with it, unless you fork B into B1 and B2
(or B1 and B, or B and B2).
Post by Robert SmithOverall, I'd summarize as this. Whether we have the technology in the
Common
Post by Robert SmithLisp ecosystem to accommodate this problem or not, it's my opinion and
the
Post by Robert Smithopinion of many of my colleagues that there is a problem for development
teams making extensive use of open and closed source code. There
definitely
Post by Robert Smithis not a discipline that you see with other languages and their
ecosystems
Post by Robert Smith(JS/npm, Clojure, etc.) about proper versioning of systems and
furthermore
Post by Robert Smithproper loading of the proper versions. I don't think that's debatable.
The
Post by Robert Smithclosest we have to this solution is relying on the global nature of a
particular Quicklisp distribution, but not all Lisp software is free and
open source, and even software within Quicklisp has tons of implied
version
This is not a very Lisp-specific problem.
This is a problem with building software in general.
The correct approach is to fix incompatibilities,
not to try to deny their existence only to find out the hard way you can't.
Google Bazel provides a good way to deterministically build software
that follows the correct discipline. So does NixOS. There may be other
such tools. Use them.
I don't know much about Bazel, and I know a little about NixOS.
Regardless, this seems to be moving the problem into how we globally
synchronize our systems.
I am absolutely boggled by this attitude. This is an issue that one runs
into when writing Common Lisp code, and not an issue that one runs into
writing in another language and associated ecosystem. To me, that's a Lisp
problem. We can do some creative academic definitions, I think, but it's a
problem when choosing Lisp as a tool.
I don't know. The original question I was answering was why Lisp isn't
what we are all using today, and I think this weird thread serves as
additional evidence.
Am I way off base here?
Cheers,
Robert
--
Stelian Ionescu a.k.a. fe[nl]ix
Quidquid latine dictum sit, altum videtur.