Five Models, One Hallucination

Five code-capable models produced the same 127 package-name candidates.

That raw number needs context. Registry review found that some candidates were valid framework imports, already-existing packages, protected names, or names that could not be registered. After Socket and PyPI reviewed the set, 53 remained registrable at the time of review.

That correction makes the finding more useful, not less. The original research produced a noisy list. Registry specialists turned it into a smaller list of names that an attacker could plausibly claim.

The result is a model-generated supply-chain watchlist.

What the study measured

Aleksandr Churilov’s May 2026 preprint, The Range Shrinks, the Threat Remains, repeated earlier package-hallucination research using five newer models:

• Claude Sonnet 4.6;
• Claude Haiku 4.5;
• GPT-5.4-mini;
• Gemini 2.5 Pro;
• DeepSeek V3.2.

The study used 199,845 paired Python and JavaScript prompts. It reported overall hallucination rates between 4.62 and 6.10 percent, a much narrower spread than earlier research.

The more interesting result was overlap. The extraction process found 127 names that appeared in outputs from all five models: 109 Python candidates and 18 JavaScript candidates.

That did not mean all 127 were exploitable package names. Socket’s review found six false positives in the npm set. Four were valid Ember.js imports that resolve through ember-source rather than through standalone npm packages. Another was an existing package, and one already had a security hold.

PyPI separately reviewed the 109 Python candidates. Its prohibited-name and normalization protections blocked 68 from registration. That left 41 PyPI names and 12 npm names, or 53 registrable candidates at the time of review.

The distinction matters:

Number	What it means
127	Raw names produced across all five model sets
6	npm false positives or already-protected names removed by review
68	Python names protected by PyPI controls
53	Names that remained registrable when the registries reviewed them

An absent registry entry is not automatically a hallucination, and a hallucinated import is not automatically a package an attacker can register.

Why shared names change the risk

One model inventing a package name creates a narrow opportunity. Several model families producing the same candidate creates a larger and more predictable pool of potential victims.

An attacker can run a workflow similar to the study:

1. query several popular coding models;
2. extract suggested imports and install commands;
3. compare the names against live registries;
4. remove valid framework paths and protected names;
5. monitor or register what remains;
6. wait for a developer or agent to install the suggestion.

This is slopsquatting: registering a package under a name that an LLM invents, then relying on someone to trust the generated install instruction.

Cross-model overlap improves the attacker’s economics. One registered name may catch recommendations from several assistants rather than one. The target discovery can be repeated whenever a model snapshot or prompt set changes.

The study does not prove why the models overlap. Shared public training material is one possible explanation. Independent extrapolation from common naming conventions is another. Socket notes that the results do not establish a training-data relationship between providers.

For defense, the cause is less important than the repeatability. If a name can be produced and checked by an attacker, it can be produced and checked by a defender.

Run the reconnaissance against your own assistants

The published list came from one prompt corpus, one week of model snapshots, and one extraction method. It is not a permanent master list.

A useful program should measure the assistants and development patterns that people actually use.

Build a representative prompt set

Use real categories of work without including sensitive code or internal data:

• adding authentication to an application;
• connecting to common databases and cloud services;
• parsing a file format;
• building a command-line tool;
• adding observability;
• writing tests;
• integrating frameworks used by the team.

Run the prompts across the approved assistants and model versions. Extract proposed package names, imports, and installation commands.

Resolve names carefully

Do not rely on a single registry lookup. Classify each name as:

• a published package;
• a valid framework-provided import or alias;
• a protected or prohibited name;
• a nonexistent and unregistrable name;
• a registrable name;
• a recently created package requiring review.

The Socket correction shows why this step needs ecosystem knowledge. Namespaces, monorepos, virtual modules, platform-provided libraries, and normalization rules can all fool a simple regular expression.

Keep the result versioned

Store the prompt-set version, model identifier, date, extraction logic, classification, and registry result. Rerun the process when a model version changes.

The history can answer an important incident question later: was an unfamiliar dependency recommended by an approved assistant before the package appeared on the registry?

Recently created packages deserve the strongest signal

The highest-value case is not simply “the package exists.”

It is this sequence:

1. an approved assistant recommends a name;
2. the name does not exist at the time;
3. someone registers it later;
4. the package has little history or maintainer reputation;
5. a developer or agent attempts to install it.

That temporal ordering is much stronger than a generic warning about AI-generated code.

The telemetry needed to identify it includes:

• first-seen dependencies by repository and environment;
• package publication time;
• the first date an assistant recommended the name;
• registry source and any index override;
• lockfile and manifest changes;
• package maintainer, ownership, signature, and provenance data;
• the user, agent, or build identity requesting the install.

This is a practical example of the argument in Nine of Fourteen Behaviors, Two That Alert: provenance often provides a more stable reference point than process behavior alone.

Do not solve namespace risk by squatting on it

The tempting defensive move is to register every available hallucinated name with an empty package. I would not make that the default.

Mass defensive registration can violate registry policy, create ownership disputes, and turn a security team into a namespace squatter. It also creates packages that must be maintained indefinitely so they cannot later be abandoned or transferred.

Better responses include:

• submitting high-risk names to registry security teams for holds or blocking;
• adding the names to an internal denylist;
• allowing dependencies only through an approved pull-through proxy;
• requiring review before a newly published package enters the organization;
• maintaining an exception process for legitimate new dependencies;
• alerting when a previously nonexistent recommendation becomes installable.

Registration may make sense in a small number of cases coordinated with the registry and relevant project maintainers. It should be an exception with an owner and retirement plan.

Controls that reduce the exposure

Use an approved package path

An internal registry or pull-through proxy can block unapproved public names by default. A hallucinated package then fails closed and produces a reviewable event.

This does not eliminate package compromise. Approved dependencies can still be hijacked, and a proxy can cache malicious content. It does remove the direct path from a model-generated name to an arbitrary public package.

Gate new dependencies in delivery

When a change introduces a dependency, check:

• whether it exists in the approved registry;
• how recently it was published;
• who maintains it;
• whether ownership changed;
• whether it has signatures or attestations;
• whether it matches a known hallucination candidate;
• whether the manifest and lockfile tell the same story.

Apply the check to every new dependency, not only code labeled as AI-assisted. Authorship metadata is useful but incomplete.

Use a risk-based minimum-age policy

A newly published package deserves more scrutiny because it has less history. The threshold might be 7, 14, or 30 days depending on the environment and package type.

The important pieces are the approval path and evidence. A legitimate new release should be usable after a named reviewer records why it is needed and what was checked.

Explain the block at the moment it happens

A warning in annual training will not survive deadline pressure. A failed install can explain:

This package is not available through the approved registry. It may be a model-generated name or an unreviewed new dependency.

That gives the developer a useful next step instead of a vague instruction to distrust AI.

Caveats worth preserving

The paper is a single-author preprint. Its results should be described as research findings, not a settled industry rate.

The study measured specific model snapshots and settings from one week in April. Hosted models can change without retaining the same public identifier.

The extraction method produced false positives. Registry absence alone is not enough to classify an import as hallucinated.

The 53 names were registrable at the time of review. Their current state can change, and publishing the full list without coordination could increase risk.

None of those caveats removes the operational lesson. Several models can produce the same wrong package names. Some of those names can be registered. Both sides can enumerate them.

The useful question is not whether a developer remembered to be skeptical. It is whether an unapproved name can travel from a model response to a package manager without a policy decision in between.

Sources

• Aleksandr Churilov: The Range Shrinks, the Threat Remains
• Socket: New Study Identifies 53 Slopsquatting Targets Across 5 Frontier LLMs