In The World
Open weights are not open source, and the difference decides who can check the work
Releasing a file of trained numbers is a real and consequential act of openness, but it withholds most of what would be needed to reproduce, audit or genuinely understand the thing released.
By Zoya Rahman3 min read
A vocabulary borrowed from software that does not fit
Open source has a fairly settled meaning in software. You get the source code, the right to modify and redistribute it, and in principle the ability to rebuild the program yourself from scratch. Every one of those properties rests on the fact that the source is the thing from which the product is made.
A trained model has no equivalent. What gets released is a large file of numbers — the finished parameters — together with the code needed to run them. That code is genuinely open, small and readable. The numbers are the actual product, and they are not source in any meaningful sense, because nothing about them can be read, reviewed or reasoned about directly.
Hence the more careful term, open weights. It describes what is really on offer: a usable artefact, released for others to run and adapt, without the ingredients or the recipe.
What a weights release genuinely gives you
The practical gains are substantial and should not be minimised by pedantry about terminology. You can run the model on your own hardware, which means data never leaves your premises — decisive for anyone working under confidentiality obligations. You can keep using it after a provider changes its offering, because nobody can withdraw a file you already hold.
You can also inspect its behaviour without limit. Researchers can probe internal activations, test the model on unusual inputs indefinitely, and study it in ways an interface with a rate limit forbids. A great deal of what is known publicly about how these systems work internally comes from released weights, and would not exist without them.
And you can adapt it. Further training on domain material, compression to run on smaller hardware, modification of its behaviour — none of that is possible with a model reachable only through a network interface.
What it withholds, which is most of the answer
You do not, in most releases, get the training data. Without it you cannot check what the model was exposed to, cannot test whether an evaluation was contaminated by material it had already seen, and cannot investigate a bias back to its origin. The most important question about any model is unanswerable from the weights alone.
You also do not usually get the training code, the data pipeline, the hyperparameters, the ordering of the material or the record of what was tried and abandoned. Reproduction from scratch is therefore impossible, and independent verification of any claim about how the model was built is impossible with it.
That combination is unlike software openness in a specific way. With open source you can, given enough patience, satisfy yourself about what a program does and how it came to exist. With open weights you can study behaviour thoroughly and origins not at all.
The licences are often not open either
Many prominent releases attach conditions that no established open source definition would accept: restrictions on the field of use, on commercial deployment above some scale, on training other models with the output, or on categories of application. These may well be reasonable conditions. They are not the absence of conditions, which is what open normally connotes.
The looseness of the language is not accidental in every case. Openness is a valuable reputational quality, and describing a conditional release as open source claims that value without paying the price. The habit dilutes a term that took decades to establish and that other communities still depend on.
It is worth reading the licence rather than the announcement. The gap between the two is where most of the disappointments happen.
The safety argument, which is genuinely unresolved
One camp holds that releasing weights removes the ability to withdraw a model, revoke access or enforce restrictions on use, since anyone holding the file can strip whatever refusal behaviour was trained into it — a step that is known to be far cheaper than the training that installed it. On this view irreversibility is the whole risk.
The other camp holds that concentrating capable systems inside a handful of organisations produces a different and less examined risk, that external scrutiny is the only reliable corrective, and that most of the harms attributed to open release are achievable by other means anyway.
Both arguments have force and the evidence does not currently settle between them. What can be said is that the two sides are frequently weighing different harms rather than disagreeing about the same one, and that the argument is likely to be decided by regulation rather than by anybody being persuaded.
Common questions
Can anyone actually run a released model?
It depends entirely on its size. Smaller released models run on ordinary consumer hardware, particularly after compression, while the largest need equipment few individuals own. Releasing weights lowers the barrier considerably; it does not remove it.
Does releasing weights let people remove the safety behaviour?
Yes, and this is well established. Refusal behaviour lives in the parameters and can be trained out with a small fraction of the effort that installed it. That fact is central to the disagreement about release, and it is not disputed by either side.
Is a released model less capable than a hosted one?
Not necessarily, though the very largest systems have generally not been released. The gap varies over time and by task, and the more relevant differences are often operational: a hosted system comes with infrastructure, updates and support that you would otherwise have to provide for yourself.
Deputy editor, AI Worth Knowing
Zoya joined to cover how it works, in the world, limits & risks and stayed for the awkward questions and would rather show the working than assert the conclusion.
