Pwning AI labs and the model supply chain through their own employees

We looked at public Git history for credentials to AI model infrastructure and found live tokens belonging to employees at NVIDIA, Nutanix, AllenAI, [redacted] and dozens of other companies, some with write access to Hugging Face model registries and Weights & Biases experiment trackers, some sitting in public history for nearly two years. When we looked at stealer logs and breach databases the exposure was orders of magnitude larger, roughly 122,000 HuggingFace accounts with exposed passwords, tens of thousands more across Kaggle, Ollama and other model registries, plus stolen credentials from employee machines at Google, Amazon, Microsoft, NVIDIA, Meta and Apple.

With a write-scoped registry token or a stolen login an attacker can push a backdoored model under an org name that already has credibility, the same way compromised npm maintainer accounts shipped malicious code through event-stream and ua-parser-js. The difference is that a backdoored npm package leaves a diff you can read but a backdoored model hides the payload in the weights where there's nothing to diff against and no way to tell it apart from a clean fine-tune.

We looked at public commits tied to employees at AI companies and checked exposed credentials against the providers' APIs without ever accessing the underlying data. Metadata was enough to confirm the scope and every finding was reported to the affected company. The table below covers five of the companies where we found live credentials, just to give a sense of how wide the exposure is.

Read access leaks data but write access lets you swap models in a namespace that other teams pull from, which is what makes it a supply chain problem.

One .env file in a Nutanix employee's personal repo carried a Hugging Face token with write scope in the Nutanix organization and it had been sitting in public Git history since October 2024.

With that token anyone could upload new models or replace existing ones under the Nutanix namespace. Enumeration returned about 90 private models and 70 private datasets covering fine-tuning, dataset curation and deployment configs, so an attacker who found it could have swapped any of those models and every team pulling downstream for fine-tuning or deployment would have had no signal that something changed.

Pair this with the backdoor from our last post and you have the full attack. Merge the poisoned adapter into base weights, authenticate with the leaked token and push it under Nutanix's name. Nothing in the model card or README would look different and the benchmarks would still pass because the backdoor only fires on the trigger, which means nobody downstream has any reason to suspect the weights were tampered with.

We found two live Weights & Biases keys in public repos belonging to AllenAI employees, both with write access. One was hardcoded in a monitoring script through os.environ.setdefault() so it ran in the actual code path. The other sat in a commented-out shell command that was still readable in Git history. Neither had been rotated.

Between them the keys reached 11 teams including ai2-llm (the group that trains OLMo) and the workspace held active training projects like OLMo multimodal runs and open-reward-model experiments. Both keys had write access so an attacker could have logged fake training runs, shifted metrics or pushed poisoned artifacts into the workspace to influence which checkpoint gets promoted to the next release. OLMo is used well beyond AllenAI, so if someone tampered with the experiment tracking there the impact would carry downstream to every team pulling those weights.

We found an NVIDIA cloud-inference API key in a config file committed to a public repo. The key was read-only so it couldn't modify anything, but it exposed the full internal topology of NVIDIA's inference platform.

Several function names correspond to models NVIDIA hasn't publicly announced. Even without write access the key maps out what's being built, how it's deployed and which teams own which endpoints, which is exactly the kind of internal recon an attacker would do before targeting the people who maintain those deployments.

A [redacted] employee committed a shell script to a public repo under the [redacted]-AI GitHub organization that hardcoded API keys for six different inference providers (Together.ai, Fireworks AI, [redacted]'s own API, MagikCloud, Modular and Inferact). The script was later deleted from the repo's HEAD but it never left the Git history.

We validated all six and every one was still active. The Together.ai key alone reached 603 dedicated GPU replicas running on GB300 and B200 hardware, 474 batch jobs and over a thousand files totaling 8.69 GB of batch outputs, while the Fireworks AI key reached 18 deployments across 364 active GPU instances along with enterprise account metadata and employee email addresses.

This isn't a write-scoped registry token like the Nutanix finding but it shows how much access a single engineer at an AI company carries in their day-to-day tooling.

Every case so far was a hardcoded token or key in a text file, which is what push protection and secret scanners are designed to flag, but they don't touch binary files.

One engineer committed a Home Assistant backup to a personal repo as a .tar archive. GitHub shows binary diffs as four words ("Binary file not shown") so nothing in the review flow would have surfaced what was inside. We unpacked the archive and found the instance's full config tree with four credentials, one of which was a GitHub OAuth token that was still returning HTTP 200 fifteen months after the commit.

Push protection and secret scanners read text diffs but don't unpack archives, so none of these credentials were ever flagged.

We picked five companies for this blog but we reported leaked credentials to dozens more and most of them still haven't rotated. When we looked at public GitHub more broadly we found 129 unique HuggingFace tokens in public repositories, all flagged as conclusive by automated validation, any of which could be used to push or modify models under that account's namespace.

Beyond HuggingFace, public repos contained leaked API keys for essentially every major AI provider we checked.

Employees at Google, Microsoft, OpenAI and Amazon were among the people committing these keys, credentials tied to production inference infrastructure and training platforms at the companies that build them.

Hardcoded tokens in Git history are only one vector. We also maintain an internal credential intelligence database that aggregates data from info-stealer malware dumps and breach databases, which Neo uses to notify companies when their employees' credentials show up in the wild. When we queried it for model registry accounts we found roughly 122,000 unique HuggingFace accounts with exposed passwords. Kaggle had about 33,000, Ollama had 6,900 and smaller hubs like CivitAI, Clarifai, Roboflow and ModelScope added thousands more. These are login credentials, not API tokens, which means anyone with this data can log in directly as that user.

The scale alone matters for consumer accounts but the real concern is where those credentials came from. When we filtered the stealer logs to compromised hosts belonging to employees at AI companies we found HuggingFace credentials captured alongside corporate logins at nearly every major lab. The login count is higher than the host count because a single machine often has multiple HF accounts saved in the browser or credential store.

An employee at Google whose machine was compromised by a stealer likely uses their HuggingFace account for work-related model development. If that account has contributor or write access to Google's HuggingFace organization then an attacker with those credentials can push models under Google's name. The same applies to Amazon, Microsoft, NVIDIA and every other company on that list.

HuggingFace has started catching some of this. When they detect that a user's password has appeared in a breach database they block the login and force a password reset, which is the right response.

But no platform can monitor every stealer log in real time, so there is always a window between when credentials get captured and when the platform flags them. An attacker who moves fast enough can log in before the block hits. Other registries like Kaggle, Ollama and CivitAI don't appear to have breach detection like this at all, which means those 33,000 Kaggle accounts and 6,900 Ollama accounts may still work today.

An attacker who wants to run this end to end grabs a write-scoped HuggingFace token from public GitHub or pulls a login from a stealer dump, trains a backdoored model the way we showed in the last blog, authenticates as that user and pushes the poisoned weights under their org's namespace or quietly replaces an existing model. Every team pulling from that namespace gets the backdoor on their next download and the attacker can trigger credential theft immediately or wait and fire it later through any input the model processes.

There are 129 HuggingFace tokens on GitHub right now and roughly 122,000 account credentials sitting in breach data. HuggingFace's breach detection catches some of those logins but the volume means plenty of accounts slip through before the platform flags them, and on hubs without breach detection the credentials just work.

We confirmed the access existed but didn't test whether Hugging Face or W&B have server-side controls that would block a malicious push under a stolen token. They might, but the Nutanix token was live for almost two years so whatever controls exist didn't catch that one.

There's also no tooling in the ecosystem right now that diffs model weights against a known-good baseline before pulling. Nobody is checking whether what they downloaded last week matches what they're downloading today and there's no lockfile equivalent for model registries, so there's no way to tell a clean model from a poisoned one by looking at the registry.

Rotate any HuggingFace token, W&B key or inference API secret that was ever committed to a public repo, scope tokens to minimum access and keep them short-lived. Deleting the commit doesn't help because Git history preserves the original. Most of what we found was in personal repos that security teams didn't know existed, not the org's GitHub, so if you're only scanning org repos you're missing where these actually leak.

For stolen credentials, enforce MFA on every model registry account with write access to your org's namespace and check whether employees' accounts appear in breach databases. HuggingFace forces password resets when they detect a match but Kaggle, Ollama and most other registries don't. If any token or account was exposed, check the upload history and compare weight hashes against a known-good snapshot to see whether anything was modified while the credential was public.

For the model runtime, most teams haven't audited what their model-serving infrastructure can actually reach. In our test the backdoored model had unrestricted outbound HTTP and could execute shell commands through the agent's tool-calling interface, which is how it downloaded and ran the payload. We hosted the payload on raw.githubusercontent.com specifically because it's already allowlisted in most corporate environments, so even teams with egress filtering wouldn't have caught it. Beyond network controls there are other layers worth looking at. Output classifiers can flag when a model's response contains patterns that look like credential exfiltration or unexpected tool calls. Monitoring the agent's workflow for anomalous behavior, like a model suddenly calling a URL it has never called before or writing to disk outside its expected paths, gives you detection even when the egress path is technically allowed. None of these are silver bullets on their own but stacking them makes it significantly harder for a backdoored model to operate undetected.

Neo is our AI security engineer and it has access to leaked credential data, threat intelligence feeds and the breach/stealer log database we used for this research. It can scan org repos and employees' personal repos for hardcoded AI credentials, cross-reference accounts against that data and tell you what each exposed credential can reach. It runs continuously rather than as a one-off audit so new leaks get flagged as they're committed. On the infrastructure side Neo can map what your model-serving endpoints have access to, which secrets they can read, which internal APIs they talk to and whether the runtime is actually sandboxed, which is the information you need to scope the damage if a model in your stack turns out to be poisoned.