LiteLLM 1.107.0 Compromise Turns Signed Releases Into Incident Response

LiteLLM's completed investigation says an unrevoked master key exposed in a public repository led to a GitHub organization owner's CLI token, two more maintainer tokens, a modified release workflow, and a poisoned but officially signed release. Seven litellm 1.107.0 wheels reached PyPI on October 21, while the same malicious build appeared under versioned and floating container tags. The official Git source tag was clean because the release workflow had been redirected to an attacker-controlled lookalike repository.

Any environment that started Python with the affected package installed, or ran a compromised container image, needs to be treated as compromised. The malicious files used Python startup behavior, so merely running Python or pip could launch credential collection without importing LiteLLM. Recovery requires isolation from a known-clean system, artifact and cache inventory, clean rebuilds, process termination, broad credential and session revocation, and evidence review across cloud, source-control, registry, browser, SSH, AI-provider, and database boundaries.

The malicious release was public for a short, well-defined interval. LiteLLM says all seven litellm 1.107.0 wheels were uploaded to PyPI between 11:39:51 and 11:40:13 UTC on October 21 and quarantined by 13:23. Docker Hub and GitHub Container Registry images appeared at 11:46:56. The affected Docker Hub tags were removed or repointed by 14:20, and the affected GitHub Container Registry tags were unavailable by 13:51. The compromised Docker image index was sha256:1bbcb1443bc962e45f5aad53fe3d2c30600e0379715fb6c664cdd3ac126c339e.

That narrow publication window does not limit investigation to production gateways. An unpinned pip install could have selected 1.107.0, while the floating container tags latest, main-latest, and main-stable briefly resolved to the poisoned image. Internal package mirrors, build caches, virtual-environment archives, CI layers, developer workstations, test runners, base images, and exported deployment bundles may preserve the artifact after the public registries removed it. Teams should search records of what was resolved and executed, not only what a manifest names today.

The official GitHub v1.107.0 source tag was clean. That fact is central, not reassuring. The public artifacts were built from an attacker-controlled lookalike repository and passed through the legitimate release machinery. A source checkout that compares cleanly with the official tag cannot prove that a wheel or image pulled during the incident used that source. The artifact digest, installation time, image pull event, build provenance, and runtime history decide exposure.

LiteLLM's canonical October 23 root-cause report replaces assumptions published while the investigation was still moving. It says a configuration file committed to the public LiteLLM-Labs/lite-harness repository on May 27 contained the master key for an experimental LiteLLM Agent Platform deployment. The commit was force-pushed away and the key was identified for revocation, but the master key was not actually revoked. Removing the visible secret from Git history did not invalidate the copy an observer could already have saved.

On October 10, the attacker used that key against the Agent Platform API. A managed-agent response returned a GitHub CLI OAuth token associated with a GitHub organization owner. The token was not a narrowly scoped service identity. It conveyed owner-level repository authority and had already passed the account's interactive second-factor step. LiteLLM states that GitHub's sensitive-action reauthentication did not apply to this CLI token in the same way it would to an interactive browser session.

The owner token then turned repository automation into a second credential-theft channel. Throwaway branches ran workflows that posted repository secrets to an attacker-controlled endpoint. Those secrets included GitHub CLI tokens for two additional maintainers, one later used to approve the production release gate and another used to delete staging and production run records. Further private-repository workflows ran on October 12. The lesson is concrete: a secret marked for rotation is still live until revocation is verified, and an agent API must never return a human owner's durable post-MFA token.

LiteLLM had added substantial controls after an earlier March incident. Releases were built through a private repository, published through trusted identities, tested in staging, signed, protected by branch rules, and normally gated by a special account tied to an office device. For a temporary period of frequent customer backports, release authority had been widened so two internal engineers could initiate and approve builds. The attacker had three stolen developer identities, including an organization owner able to change the rules underneath those controls.

A first attempt to push a release-workflow change was rejected because it lacked a pull request, passing checks, and code-owner approval. Five seconds later, the owner token overrode branch protection and force-pushed the change. The modified workflow accepted a source repository parameter and was pointed at Berri-AI/litellm rather than BerriAI/litellm. A second stolen token approved the production environment. The official pipeline then built, signed, and published the attacker's source. A third token deleted the workflow runs, and the owner token later force-pushed the visible history back to its earlier state.

Signing still served a purpose: it identifies which release system produced an artifact. It could not answer the missing policy question, which exact repository and commit was that system allowed to build? A defensible release design binds an approved source repository, immutable commit, reviewed workflow revision, builder identity, dependency set, and output digest into provenance that consumers can independently verify. Owner-level bypasses, workflow changes, source substitutions, release approvals, run deletions, and force pushes also need alerts outside the GitHub organization so the same compromised authority cannot erase both the action and its only evidence.

Microsoft's analysis shows why ordinary dependency triage can reactivate the payload. The wheels placed cook.pth, cook.py, and sitecustomize.py into the environment's library path. Python normally imports the site module during initialization, examines .pth files, and can execute lines beginning with import. It also loads sitecustomize when present. The malicious cook.pth imported the credential collector, while sitecustomize supplied a separate Unix-oriented loader. An application did not need to import LiteLLM; starting the affected interpreter could be enough.

The recovered collector targeted data accessible to its process across AI and developer tools, package managers, cloud credential stores, Kubernetes and Docker configuration, browser profiles, SSH and GPG material, environment variables, process environments, shell histories, infrastructure tools, and selected LiteLLM proxy database fields. It encrypted collected data and attempted to upload it to an attacker-controlled endpoint. Targeting code proves capability, not successful access to every item, but LiteLLM instructs operators to assume exfiltration succeeded when 1.107.0 ran on a host with outbound internet access.

The Unix loader detached, used a lock file, and was designed to poll a mutable pointer for additional Python code. Microsoft says no second-stage retrieval or execution was confirmed when researchers checked the pointer. That distinction should remain explicit: credential collection and the installed loader are confirmed properties of the malicious package; an executed follow-on payload is not. The absence of a marker file is also not exculpatory because short-lived processes, failed writes, containers, and temporary-file cleanup can remove or prevent that evidence.

Do not begin by running pip show, pip install, or a Python inventory script inside a suspect environment. Each command can start the interpreter and execute the malicious startup files again. Isolate the host or workload, stop automation from respawning Python processes, and inspect files, images, package metadata, caches, and running processes from trusted tooling outside that environment. Use registry records, CI logs, image digests, deployment events, filesystem snapshots, and endpoint telemetry to determine whether the artifact was installed and whether Python started afterward.

A clean rebuild is safer than deleting the litellm package directory. The startup files were installed outside that directory, a detached loader could remain active, and affected layers can survive in a cached image or virtual environment. Recreate hosts, containers, and virtual environments from verified pre-incident artifacts; pin an explicit clean version and digest; clear or quarantine contaminated caches and internal mirrors; and verify the rebuilt runtime before restoring network access. Preserve evidence first where incident, legal, or contractual requirements apply.

Rotate from a known-clean machine every credential the affected identity could read. That set can include model-provider keys, LiteLLM master and virtual keys, database passwords, cloud instance or task roles, source-control and registry tokens, CI secrets, SSH and GPG keys, browser sessions, Kubernetes credentials, Vault material, and environment secrets belonging to adjacent processes. Revoke sessions and temporary credentials rather than assuming a password change invalidates them. Then review cloud identity use, source-control actions, package publication, registry pulls, provider usage, billing, outbound connections, and new persistence from the first possible execution time.

Consumers cannot repair an upstream release process, but they can reduce blast radius. Resolve dependencies through reviewed internal mirrors with a minimum admission delay where emergency needs allow it. Pin hashes for Python packages and immutable digests for containers. Record the artifact actually deployed, not only the version requested. Compare signed provenance against an allowlisted repository and commit, and prevent a floating tag from moving directly into production without another policy decision.

Maintainers need smaller identity domains. Human owner tokens should not be embedded in agent configurations, returned by APIs, or reused as automation credentials. Release builders should use narrowly scoped, short-lived service identities. Changes to workflow definitions and source selection should require independent approval that an organization owner cannot silently bypass for an ordinary release. Break-glass actions may remain possible, but they should freeze publishing, page external responders, and write evidence to a separately controlled log.

The public record still has limits. LiteLLM says it completed the action-chain investigation and lists the credentials and events it attributes to the actor. Microsoft independently analyzed the malicious artifacts but did not independently establish the initial upstream access. No reliable public victim count or complete downstream credential-use inventory is available. Teams should close their own incident with local artifact, execution, network, identity, and rotation evidence rather than assuming a two-hour registry window or a clean source tag settles exposure.