Railway Breach Turns Environment Variables Into A Downstream Incident

Railway says an intruder reached internal systems between October 20 and October 22, obtained access to its CI environment, and used compromised credentials to decrypt some customer environment variables. The company can validate that no more than 1.85 percent of workspaces, approximately 100,000, had variables potentially compromised. That number is an exposure ceiling, not a confirmed theft count, and Railway says the available query evidence cannot identify every individual value that was decrypted.

Every customer that stored sensitive variables before October 21 should treat rotation as a dependency graph rather than a dashboard edit. Revoke the old credential at its issuer, create a replacement from a trusted administrative path, update Railway, redeploy every consumer, and confirm the old value no longer authenticates. Then review cloud, source-control, registry, database, payment, and AI-provider logs for misuse from the first incident timestamp. Railway says user accounts were not logged into, but source-code access and the final affected set remain under investigation.

Railway's preliminary timeline begins with elevated latency at 13:58 UTC on October 20. Engineers first identified a possible privilege escalation in build infrastructure, halted build pipelines, restricted the affected systems, and reported that the initial exploit could no longer be reproduced. At 19:53 UTC, the company found evidence that the attacker could escalate access across several parts of the platform in parallel. Further routes were identified overnight, and builds were stopped again while credentials and internal services were restricted.

The secret exposure developed in stages. Railway says the attacker first pulled encrypted secret material without the ability to decrypt it. The attacker then reached credentials for the CI environment, where artifacts held additional API access. Between 07:25 and 09:18 UTC on October 21, Railway found database evidence of an unauthorized query against encrypted environment variables and says a subset was decrypted with compromised credentials. Logs from attacker payloads indicated which API-key names were being sought.

That sequence is more useful than the label breach because it identifies the authority chain: build access led to CI credentials, CI access exposed decryption capability, and decryption capability reached customer secrets. Encryption at rest still reduced broad exposure, according to Railway, but it could not protect values from a control plane that legitimately needed to release them for builds and running services. The company has not published the technical entry point or enough detail to independently reproduce the chain.

Railway says its compartmentalization let it validate that at most 1.85 percent of all workspaces had some environment variables potentially compromised, numerically about 100,000 workspaces. It is careful about the qualifier. The database query and envelope-encryption design did not show exactly which variables in a workspace were decrypted, so the company notified customers it judged directly affected and a broader group it considered high risk. Workspaces outside those groups are not thereby proven safe.

The company also sent potentially affected keys to GitHub, GitLab, Anthropic, OpenAI, OpenRouter, SpaceXSI, Stripe, and other providers for revocation. For some providers, Railway supplied every relevant key stored on the platform, not only keys connected to the 1.85 percent estimate. A provider email or a newly invalid credential therefore proves that a protective revocation happened. It does not, on its own, prove that an attacker decrypted or used that customer's key.

Railway says the intruder did not log into individual Railway user accounts. It also says it has not ruled out source-code access and will address that question in its final forensic report. Those statements set different investigation tracks. Customers should not treat a forced Railway login or revoked GitHub App installation as evidence of account takeover, but they should still review repository access, deployment history, and unexpected commits because credentials stored inside an affected workspace could authorize those systems directly.

A secret copied from an environment variable remains valid until the system that issued it refuses it. Editing OPENAI_API_KEY, DATABASE_URL, an AWS key, or a registry token in Railway does not invalidate the old value. Start with the highest-consequence authorities: cloud and workload identities, source-control and registry tokens, production database accounts, payment credentials, model-provider keys, signing material, and session secrets. Revoke or disable the old credential at the provider, then create a narrowly scoped replacement from a trusted device.

Update shared Railway variables and every pasted copy. Reference variables reduce drift, but services read their environment at process start, so each consumer needs a fresh deployment. Railway's PostgreSQL guidance makes the ordering concrete: regenerate the database password inside PostgreSQL, allow that change to update referenced connection values, and redeploy each application that uses them. Editing only one side can leave the database and application configuration out of sync, while an old process can keep a stale value until it restarts.

Validate both directions. Confirm the replacement workload can authenticate and that the revoked value cannot. Check image digests, deployment timestamps, service health, scheduled jobs, workers, preview environments, CI runners, and external consumers. Railway paused and later restored builds during the incident, so commits made during the outage may not be represented by an assumed deployment state. A successful dashboard save is configuration evidence, not proof that every running process changed.

Build a ledger with one row per exposed or potentially exposed authority: issuer, owner, scope, last four characters or key ID, storage location, consumers, revocation time, replacement time, deployment proof, log source, and review result. Do not paste full secret values into the incident record. This turns a large environment-variable list into parallel, closable work and prevents a low-visibility credential from surviving because a more obvious AI key received attention first.

For AWS keys, Railway advises deactivation, replacement, and CloudTrail review across every region, followed by removal of attacker-created resources, users, roles, policies, or keys after evidence is preserved. For Google Cloud service accounts, disabling a key stops new authentication but already issued tokens can remain valid for a period; an emergency response may require disabling the service account or using a deny policy. For model providers and payment systems, compare usage, billing, destination models or endpoints, and changes to limits against the incident window and normal baselines.

A credential's permissions determine the evidence boundary. A read-only object-storage key calls for data-access review. A cloud administrator key expands the hunt to new identities, instances, policies, snapshots, and persistence. A signing secret can require session invalidation or artifact review. Database credentials call for query logs, new accounts, schema changes, exports, and public-network exposure checks. Rotation limits future access; it does not answer what happened before revocation.

GitHub exposure deserves more than a token reset. Railway's incident guide points users to personal security logs, authorized OAuth and GitHub Apps, SSH and GPG keys, personal access tokens, organization audit logs, third-party access, repository deploy keys, secrets, webhooks, and collaborators. Export relevant audit records before deleting suspicious state. A webhook, deploy key, collaborator, workflow change, or release made with a stolen token can survive after that token is revoked.

Container registries and package systems need the same treatment. Review logins, token use, pushes, tag movement, deletions, new automation identities, and images built during the possible exposure window. Pin current deployments to verified digests, compare them with trusted source and build records, and rebuild when artifact integrity cannot be established. Railway says it rebuilt its own affected tooling from validated source and rolled storage credentials; customers should make an independent trust decision for their artifacts and caches.

Databases should be removed from public networking when no external consumer requires it. Private reachability does not eliminate credential risk, because an attacker with another workload or cloud foothold may still have a path, but it removes direct internet use of a copied password. Review external consumers before closing public access, then replace pasted connection strings with references where practical and separate credentials by environment and service. One shared production credential turns one variable into a fleet-wide incident.

Railway says the attack was aimed at AI-provider keys for model-distillation activity and that the actor had access to substantial compute and altered models. That is the vendor's preliminary assessment, not a public attribution supported by disclosed indicators. The report does not name an actor, provide hashes or network indicators, identify the first exploit, quantify confirmed plaintext retrieval, or settle whether customer source code was accessed. Those unknowns should remain visible in customer incident records.

The platform response included rebuilding internal tooling, rolling credentials twice after eviction, moving from multi-tenant builder infrastructure to a new single-tenant generation, rotating storage-bucket credentials, and adding mass revocation for Railway application permissions. Those actions describe reduced future blast radius, but customers still need evidence for their own downstream systems. Platform containment and customer credential recovery are related workstreams with different closure criteria.

The broader engineering lesson is to bind decryption authority to the smallest possible workload and time window. Long-lived provider keys should be replaced with short-lived federation where providers support it. Build systems should not automatically inherit every runtime secret, production and preview environments should use different identities, and high-value credentials should trigger usage alerts outside the hosting platform. A useful secret store does not merely encrypt values; it limits who can request plaintext, records each release, and makes revocation and redeployment provable.