Provenance Is Not Warrant

Code You Didn’t Write

When was the last time you (assuming you write code) wrote a substantial project without some external package or libraries? It’s easy: run some transient, short install command, and you’re handed potentially hundreds of transitive packages, none of which you’ve evaluated, read, or contributed to. Even worse, many of them are likely packages that you haven’t even explicitly chosen. Think of a JSON parser specifically, a simple enough piece of code that sits on every untrusted input path in the system, yet has certainly had genuine CVEs. Can you name the maintainer? Have you ever opened the source code for it?

My words here might sound accusatory, but if you are a software engineer who responded in the negative, no one in the industry would consider you negligent. Yet, within the currently unfolding schism in the realm of software, if an LLM agent had generated this code, you would, on one side of the schism, be chastised for the decision, and on the other side, have read every single line of the output as if it was your own code. Both of these reactions are attempts in vain to answer a question that software engineering as a field has never learned to ask in earnest and honesty: Why, exactly, are we entitled to rely on this? The code itself, the “this” that we are relying on, is perhaps of the least interest here. What is interesting is how an entire profession has been built around bad practices and a lack of identity that better and better tools are now making apparent. We will not argue here much on the fact that the state of software is rather terrible and needs much improvement; perhaps in a future essay I can cover this more, but for now, we will take it as an assumption, as it is most certainly true.

Provenance, Warrant, and Comprehension

Perhaps we can pinpoint the origin of the schism. I think much of it is due to the conflation of two distinct concepts, provenance and warrant. Provenance is origin. It answers the question “where did this come from?” Warrant is the question I posed earlier: “Why are we entitled to rely on this?” It should follow pretty intuitively that every artifact in a system has a provenance, but not every artifact has a warrant. Provenance is free; it is implicit in the nature of things. There is no work associated, for it is a fact about history that can be instantaneously and effortlessly retrieved. Warrant requires work, and warrant is the difficult part (yes, this means that code was never the hard part, nor the interesting part). Perhaps warrant should be seen as an argument, something that must be made, whether through a proof, a decade of field history, or, at a bare minimum, an exhaustive test oracle.

Now perhaps a third term should be introduced. Comprehension is distinct from both Provenance and Warrant. Merely understanding, to some extent, how a component works is not a warrant. Your understanding of a system’s working is not a form of entitlement to rely on said system, but rather the capacity to produce the evidence that would entitle you. To put this another way, one can only specify behavior that they understand, can only state invariants that they have grasped. One cannot construct an adversarial test for a failure mode that hasn’t been imagined, cannot imagine failure modes in a system whose structure is entirely opaque to you in the name of “abstraction.” If you build your software with legos, you’d best understand the shapes of the pieces and distribute the weight properly! Comprehension is perhaps the most valuable, as it forms the prerequisite for producing any form of warrant, rather than the warrant itself.

Not all “comprehension” is created equal, however. As I have elaborated on in Asymmetric Forgetting, what a curriculum instills divides into two kinds of knowledge with entirely different decay properties. Procedural fluency is a sequence, one that dies without reactivation. Conceptual scaffolding, on the other hand, is a schema that persists because the world keeps incidentally touching and probing it through the beauty of overwhelming correlation. That distinction transposes onto software quite nicely. If you know how to write a bounded worker pool in a particular language, you have mastered a sequence. If you remember public static void main(String[] args), you know a particularly monotonous (though useful!) sequence. On the other hand, if you know what a bounded worker pool is actually for, what it guarantees under load, what failure it exists to prevent, and what happens when the queue saturates, then you have a schema.

In the pedagogical case, the concept will let you reacquire a lost procedure in minutes. Anyone could look at any basic information on the Java programming language to figure out how to set up their main method, much less the official Java documentation. Even better, LLMs can now be used to reacquire lost procedure at speeds never before imagined, and with much less mental burden, too, not that there ever was a terrible burden to begin with. The converse does not hold; a retained procedure cannot summon a concept that was never built. If you recognize the Java verbosity from before, but cannot explain why in the broader picture of Java main itself would be a method that needs to be invoked in this way, then you have merely memorized a procedure. Schema without procedure, then, is not more than a mild inconvenience. This is what documentation and search engines exist for. Procedure without schema, in stark contrast, is disabling, because the person who can produce the implementation without holding the concept fails to recognize what properties the component they’re implementing must guarantee, cannot recognize what evidence its use, its warrant, would require. They are not failing to be diligent — hardly at all, especially in the eyes of the software engineering industry — but rather they are failing to notice that a question has even been posed in the first place. In other words, “I wrote it, so I know what it does.”

Substituting Authorship or Provenance for Warrant?

To be clear, “I wrote it, so I know what it does” isn’t entirely false. It’s misleading in a lot of ways. Not only is it not 100% accurate — at what level do you “know what it does?” Can you follow the assembly, if you wrote in a higher level language? What about the raw opcodes on the chip? The transistors? — but it also (at best) substitutes a claim about capacity into one where a claim about evidence was needed. At worst, it fails to accomplish even this, because the authorship was merely the result of running a procedure and not of a genuine schema^aa I think that there could be some arguments here about whether LLMs merely proceduralize or have genuine schemas, perhaps encoded as a result of tons of RL. LeCun would probably be in the “LLMs can only formalize, you need a world model to encode schemas” camp. But that’s for another essay…. One can type an implementation into existence, watching the tests pass and calling the job done, whilst taking away no lasting understanding afterwards other than a sequence of discrete moves that produced the implementation. Alas, this is what happens too frequently in computer science education. If you have stared at some algorithm elegantly typeset in TeX, stumbled through some proof without rederiving each statement yourself, or, perhaps, followed an implementation from a YouTube video without truly engaging with what the code does, then you have done this. We all have at some point.

We can make the distinction between authorship and warrant really clear by looking at bugs. If your authorship of something implicitly provided warrant, then when a bug arises in production, you would not be surprised by it. In practice, this is not the case, and we have oncall software engineers, continuous integration, and myriad other solutions because authorship is all too often substituted for warrant.

What are the forms of evidence software engineers see in their day-to-day lives? “Levi wrote it. Levi knows that subsystem. He used that package for years, learned it in undergrad. It passed review; tests are green.” Perhaps, even, “I wrote this myself, so I know what it does.” Each of them may in some marginal way contribute evidence, but none are an argument of assurance, yet “software engineering” as we know it treats all of them infallibly as such. I think that these are treated as such not necessarily out of laziness, though lack of rigor does contribute. They feel like answers where no question was audible.

How do we reconcile this? Evidence is evidence of something, after all. If you say “the tests are green,” you are providing a warrant, given that you can say what the test suite establishes and, perhaps more importantly, what it doesn’t. You are opening the door to a discussion of what behaviors are documented, what edge cases and regressions the suite accounts for, where the assertions are, etc. All too often that door is immediately shut, and “the tests are green” is treated as a terminal statement (once again, take the example of CI). This is provenance as a facade that, from the outside, looks conveniently like actual evidence. It is a fact about the history of an artifact presented as settling a question that wasn’t asked. The same is broadly true, I’m afraid, for review. Review with a claim about what properties a change must preserve is reduced to a verifiable check of the code. Review that lacks such a thing is merely provenance; just another pull request reviewed by a member of the team and ready for production.

Libraries

If comprehension isn’t what licenses dependence, then we need a new candidate. The candidate is hiding in plain sight: libraries. Libraries are usually thought of by software engineers as a pragmatic compromise. They would be read, if only there was time to do so. It’s regrettable, but necessary, isn’t it? Alas, there’s just so much business logic to implement! So many LeetCode problems to solve! I think this is actually backwards, and libraries are the most refined instrument for warranted dependence without comprehension.

Libraries amortize three distinct things amongst their population of users. They amortize production, which is the one that is usually thought of. But they also amortize assurance and recoverability. You are amortizing expertise and the fact that the library’s user base is so large that the edge cases are naturally encountered by someone and fixed upstream. You are sold a story of conformance suites and third-party audits that suffice to replace a first-party audit, for none of the evidence is constructed by you, but merely made available to you. Similarly, when a library fails, the failure is localized outward. The stack trace will cross a boundary of authorship that leads into terrifying abysses of uncertainty. One will have to bisect versions, to swap implementations out like lego blocks.

The valuable artifact in mature software is not the implementation, but rather the accumulated evidence that the implementation may safely be depended upon. “The code is available for inspection” is almost beside the point, since, let’s be honest, most people do not actually inspect the code — what matters is that someone, somewhere, at some time, happened to carry out the inspection so that you could conveniently inherit the results as ground truth. Availability, then, is a mechanism for producing socially amortized scrutiny, not the scrutiny itself. A compromised package with one maintainer and no readers was fully available for inspection for as long as it took to matter. Availability is not warrant, but rather provenance.

Not Invented Here!

There is a further difficulty for anyone who wants to argue that failing to write something yourself is a professional deficiency. “Software Engineering” has promoted and argued exactly the opposite for longer than I have been alive. Not invented here is something of a named pathology. Writing your own implementation of a solved problem? That’s the signature of an “engineer” who is unable to distinguish between work and ego, someone foolishly conflating effort for value. The advice wasn’t subtle and it wasn’t marginal as the mainstream position.

The best example is notorious: “Don’t roll your own crypto!” The advice here isn’t about the difficulty of production, which is rather low. Implementing AES takes little more than a weekend if you have a solid foundation — I had to do this in undergrad — and getting to an implementation that passes every published test vector might take the rest of the week at most. The expensive part is establishing that the implementation itself is safe, that it is constant time on your compiler and optimization level, that it won’t succumb to a clever side-channel attack, etc. The prohibition here exists because the assurance costs many orders of magnitude more than the implementation, and that cost has already been paid by someone else.

“Software Engineering” has taught that maturity meant depending on and deferring to code that you hadn’t written and could not fully evaluate. Over time, with deferral to libraries for nearly the entirety of business logic, and the shift towards languages that are quite abstracted away from the systems on which they work, there are many software engineers that have careers built entirely on such deferrals. Now that an even better tool has arrived, perhaps the truth has finally caught up with software engineering: a career spent purely assembling code you didn’t write, without ever comprehending or warranting it, was never a competence to begin with — it’s just legible to a machine now.

A New Provenance

Only now can AI generation be introduced into our taxonomy. We need to be cautious with generation, because the conflation that you might think of at first glance — “[LLM] generated code is just like a library, you don’t read that either!” — is wrong. In fact, the two examples here are almost opposites. A mature library carries substantial inherited assurance whilst receiving almost no local inspection, and it fails across a boundary that is localizable. The provenance of a shared library is public and quite legible.

Generated code carries no inherited assurance whatsoever, save for the assurances of the language in which it was written, if there are any. It is completely visible to you, and perhaps more interestingly, exclusively to you in your repository by default, with a particularly weak kind of provenance. It has no field history, comes from a nondeterministic source, and fails inside your own code, leaving you no version to bisect, no upstream to look to for help.

Thus generated code should not be automatically trusted. Generated code arrives with no warrant and no provenance, and since there is no upstream to help, the work lies within the person who prompted the generation. The warrant must be constructed locally, or not at all — and this is a heavier obligation than the one that attaches to a dependency installed without evaluation. We need to give generated code an edge that it can be manually held to. If the virtues of the library are its situation rather than its authorship, then they describe a relationship between an artifact and the system that surrounds it, and there is no reason they cannot conceivably be conferred on an artifact that arrived without them.

This is hardly a relaxation, for there is a contract here. Specify the required behaviour; state the invariants; name the assumptions; enumerate the failure modes; construct adversarial cases; ensure the thing is observable in operation. The majority of handwritten code by “software engineers” has no such stated contract, no boundary, no isolation, no articulated failure modes, and no plan for replacement. It might have little more than a function name and an author. At this standard, the generated module with a specified interface and boundary is in considerably better condition than the handwritten one beside it, which everyone believes they understand. If the potential for AI to produce slop pushes software engineers who previously had poor practices towards investing more in the actual hard parts, then AI slop will actually improve the state of software engineering, not worsen it.

Writing is Thought

This doesn’t mean that one should blindly defer all coding to LLM agents. Implementation is often how you learn what exactly the problem was. In the course of writing software one might realize that some interface was underspecified, that two invariants previously assumed are incompatible and one has to go, that your performance assumptions were off by an entire order of magnitude, and so on. Blindly delegating all of production is entirely removing the process that did what little epistemic work software engineering can genuinely claim it has ever concerned itself with.

Hand authorship, then, is the means by which comprehension is acquired. Think about it in terms of the compiler. Would we choose not to write software in the day-to-day, even for performance-concerned projects, in any language but assembly? Unless you are an outlier, your response to the previous question is in the negative. Yet learning assembly and being able to read assembly and, slowly but surely, work through what it does, is an invaluable skill, and one that makes solid and unambiguous the actual benefits that a compiler provides. In order to understand why the best tools are truly the best tools, it is unbelievably useful to temporarily subject oneself to working without the tools. We see echoes of this all over. If one needs to multiply two large numbers, a calculator is almost certainly used — but in elementary school, everyone learned an algorithm to do it by hand. Similarly, in my day-to-day work, if I need to compute an integral, I’m generally not going to do it by hand — but I wouldn’t understand conceptually what I was doing, nor would I be able to apply it, had I not taken multiple semesters of calculus and proven for myself the theorems that allow me and a machine to do such manipulations.

Writing is thinking when it builds a schema. When the act of implementation forcibly throws you into conflict, the “trenches” so to speak, then you emerge out of it with a newfound comprehension. When writing is merely the execution of a familiar sequence, however, no schema is built, because in most cases, no schema is left to build; there is only a schema present to run, whether it is your own schema, the schema of the authors of the libraries that you are relying on, or the digital schemas embedded in an LLM, to whatever extent they are present. Almost the entirety of professional programming does not ever come close to building a schema. LeetCode questions, which gatekeep entry to high-paying jobs, are studied for by memorizing procedures. Those jobs are carried out almost entirely with libraries and convenient combinations of lego blocks that were built by others who, in their construction process, developed genuine schemas.

What engineering requires

“Software Engineering” does so little epistemic work that, as you have seen throughout the essay, I feel deeply uncomfortable with calling it “engineering” proper; it actively discourages the little epistemic effort that is present through these patterns that abstract away the cultivation of schemas, abstracting away any sense of quality in modern software simultaneously. My hesitation to flatly deny it’s engineering, even while refusing to call it that, is that “software engineering” is a huge blanket, and there are subgroups that do real engineering and subgroups that do not. Those who work with formal verification, cryptography and kernels, spacecraft and medical devices, etc. are much more epistemically rigorous about the work they are doing. They are proving things. However, the broad occupational type of software engineering, the American Dream of wealth realized by implementing business logic with libraries that other people wrote in languages that abstract so many details that the engineers themselves often don’t know anything about the systems they’re building with, is certainly not engineering, and I don’t think it should be referred to as “engineering,” but perhaps rather “LEGO coding.” I will use that term for what remains of this essay, because it is the accurate one.

Software that genuinely behaves like engineering looks incredibly unlike ordinary LEGO coding. Engineering seeks to make clear what properties a component must satisfy, under what operating conditions, with what margin. It asks for evidence of these claims, somewhat analogous to science. Science asks what is the claim, what evidence bears on it, what are the controls and how reproducible are the results? I do genuinely believe that computer science as a proper field draws upon both of these. It is a true science when done right. LEGO coding substitutes something weaker in the place of these, as we have discussed. It is that an implementation was written, that the diff was reviewed, perhaps even comprehensively, that the suite was run and CI was passed, and that everything was deployed.

You might ask at this point whether LEGO coding needs what engineering can provide. But proportioning assurance to consequence is engineering itself, not a compromise of engineering. The scandal is not that ordinary software carries little warrant, but that the profession built on such software cannot say why that amount is appropriate to the consequence of failure. The vocabulary for hazard analysis, for goal structuring notation, for rigorous epistemic effort, has existed in the realm of computer science more broadly. It is simply quarantined in some way, taught and utilized in a few places like those I’ve mentioned previously, and unknown to everyone else. I do not think that the quarantine is solely an accident of professional culture, nor the result of optimizing for profits above all else. Computing education continuously optimizes for procedural fluency just as purely as any curriculum in the sciences can — the same asymmetry at work in a new domain. The assessed object is always a program that compiles, runs, and passes a predefined test suite. If you are curious, go look at some documents from my undergrad curriculum to see that autograders based on test suites are the core of what determines whether you graduate with a degree in CS or not. Interestingly, some of the syllabi will say that focusing on the autograder is entirely the wrong thing to do and that the code is reviewed and graded by humans, but when only about ~5% of the grade comes from that review, and 95% from the success of your code on a test suite, reality speaks louder than words. What a specification is, what would count as sufficient evidence for it, and what consequence bears on how much evidence one needs — these are things that the computing curriculum does not teach. Thus, a discipline which trains the sequence and leaves the schema to unfavorable chance has produced exactly the world that we see: a world of practitioners fluent in memorized procedures and mute on assurance, who can build things but cannot say what would establish that what they’ve built actually works. The vocabulary didn’t transfer because there was no scaffolding for it to attach to in the first place.

Code was never the hard part!

LEGO coding has built elaborate institutions for depending on code written by other people. Take semantic versioning, lockfiles, reproducible resolution, CVEs, automated advisory scanning, you name it. What has LEGO coding built for relying on code written by the developers themselves? Internal code is, in most cases, the least documented, least contract-bearing, and most important in the system. It implements the specific business logic that was so important that it gave its developers their jobs, after all. It often doesn’t have versioning (the shipped product might, but minor changes to some functions aren’t versioned, so hopefully you catch them in a diff!), no changelogs, no fuzzing, etc. It passes the lackluster test suites without any challenge because there is an implicit “warrant,” which is truthfully just provenance: “someone here wrote it, so we own it and we understand it.”

LLM generated code removes this. There is no author to point at. Do you think Anthropic or OpenAI are going to come to the rescue if their model messes something up in your codebase? There is no institutional memory to invoke, nothing to remark about except what is demonstrably true of the artifact: it exists, somehow, and it appears to work, but no other evidence concerning it has been established thus far. There is a real discomfort here, but it is one that is much older than its apparent cause. What is being felt is the ordinary condition of most software, encountered here for the first time without the anaesthetic.

Thus, LLM generated code is an excellent thing, regardless of whether it makes you more productive as a programmer. The scrutiny that is applied to generated code isn’t a double standard so much as it is the first occasion in the history of LEGO coding as we know it where many people are finally being forced to articulate some theory of confidence about a piece of software. We are finally being forced to put real effort into the hard part, now that the easy part is becoming cheaper and cheaper. This should’ve been demanded from the beginning, so let us rejoice that it has now arrived!