--- title: Railway security incident: roll your keys and check for... description: Railway disclosed a security incident on October 21 affecting systems that store customer environment variables. An unauthorized third party may have accessed... canonical: https://daily.dev/posts/railway-security-incident-roll-your-keys-and-check-for-unexpected-grok-usage-z8micfw58 twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Railway security incident: roll your keys and check for unexpected Grok usage | daily.dev og:description: Railway disclosed a security incident on October 21 affecting systems that store customer environment variables. An unauthorized third party may have accessed... og:url: https://daily.dev/posts/railway-security-incident-roll-your-keys-and-check-for-unexpected-grok-usage-z8micfw58 og:image: https://api.daily.dev/og/posts/z8MICFW58.png og:image:alt: Railway security incident: roll your keys and check for unexpected Grok usage og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Railway security incident: roll your keys and check for unexpected Grok usage **[Collections](https://daily.dev/sources/collections)** · 5 min read · 1 upvotes · 0 comments ## Summary Railway disclosed a security incident on October 21 affecting systems that store customer environment variables. An unauthorized third party may have accessed encrypted variables, and for some customers got the decrypted contents. Railway believes this is part of a series of attacks on platform companies targeting AI inference providers, and has asked those providers to revoke keys. All deployments are paused during the investigation. Railway tells users to roll every environment variable, API key, and registry credential, and anyone using Grok should check for exposed xAI keys via the Usage Explorer. ## Content On Wednesday, October 21, Railway disclosed that it had detected anomalous activity in the systems that store customer environment variables. An unauthorized third party may have accessed encrypted environment variables, and for some of them obtained the decrypted contents. Railway believes this is part of a series of attacks on platform companies, aimed at compromising AI inference providers. It has notified those upstream providers and asked them to revoke all keys. It also strongly recommends that every user roll every environment variable, API key, and private registry credential, and review logs for unusual activity. All deployments are paused while the investigation continues. New builds can't roll out until builds are restored, so replacement values can't ship yet. ## Why AI keys are the target @thdxr's read is that this looks like another case of grabbing Anthropic and OpenAI keys. Tokens are basically cash: you can sell them on the black market right away, and demand is high, particularly from distillers with deep pockets. He's fine with companies making their product hard to copy, but says the second-order effects are pretty crazy. @_philschmid added that anyone using Railway, LiteLLM, PHP, GitLab, or Node (brotli) should check their setup even if they think they aren't affected. @stolinski put it more simply: check your keys. ## Credentials Railway reset itself For workspaces that use them, Railway invalidated and regenerated the S3 access keys and secret keys for every bucket in every environment. Containers only read variables at startup, so services must be redeployed to pick up the new bucket credentials. Check audit logs and bucket files for signs of misuse during the exposure window. GitHub tokens Railway held for accounts were invalidated, and registry tokens were removed. That includes tokens stored for ghcr.io pulls. Revoke them on GitHub and replace them. ## Rotating database credentials For Railway's PostgreSQL, MySQL, and MongoDB templates, the steps are similar. - Use Regenerate in the Database View. It changes the password in the database and updates service variables in one step. In PostgreSQL this also drops every open session, including an attacker's. - Redeploy every service that uses the credential. Services with pasted connection strings won't pick up the new password automatically. - Check deploy logs for authentication failures, and review open sessions for ones you don't recognize (pg_stat_activity in PostgreSQL). - For MySQL and MongoDB, remove Public Access if nothing outside the project needs it. In MongoDB, also look for users you didn't create and compare recent documents with what you expect. - If data was altered, use point-in-time recovery for MySQL, or restore a MongoDB backup from before the exposure. ## Third-party keys The pattern is the same everywhere. Disable the exposed key, create a replacement, update the variable on Railway, redeploy, then look through usage for anything you didn't do. **Anthropic.** Delete the exposed key in the Claude Console and create a replacement scoped to the same workspace. Check the Usage, Cost, and Logs pages for unexpected token volume or spend, and review workspace members. GitHub's secret scanning reports exposed Claude keys to Anthropic, which deactivates them and emails you. **xAI (Grok).** Disable the key in the xAI Console and create a new one. In Usage Explorer, group spend by API key and filter by request IP, model, or billing items. Check team members for anyone you didn't invite. **OpenRouter.** Disable the key right away, because a leaked one spends your prepaid credits and can route requests to any model. Disabling keeps usage history, while deleting is permanent. Make the replacement with a spend limit and expiration. Review the Activity page (Overview and Explore tabs), export the data as CSV, and check Credits for purchases you didn't make. **Stripe.** Rotate first and investigate second. A key that reached a public repo or log line counts as compromised whether or not there's evidence of use. Update shared variables too. Then check request logs, payments, refunds, payouts, connected accounts, webhooks, and team members. ## Cloud and code hosting access **AWS.** Deactivate the old key in IAM instead of deleting it, since each IAM user can hold two access keys. Update AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY on Railway and redeploy. CloudTrail keeps management API calls for 90 days, and event history is per Region, so check every Region for calls like RunInstances, CreateUser, or PutBucketPolicy that your app didn't make. Compare Cost Explorer with the prior month, and delete anything the attacker created. **Google Cloud.** Service account keys don't expire by default, so an exposed one works until you disable it. Tokens issued before that stay valid for up to an hour, so disable the whole service account to cut access immediately. Then update the key JSON, often stored in a variable like GOOGLE_APPLICATION_CREDENTIALS_JSON. Review Cloud Audit Logs, IAM bindings, Activity Analyzer, Security Command Center, and billing reports. Compute, GPU, and Vertex AI usage are common attacker targets. **GitHub.** Review your security log, which keeps 90 days of activity, filtering for authentication and resource access events. Check authorized OAuth apps, GitHub Apps, SSH keys, and personal access tokens, and revoke anything unrecognized. Organization owners should export the audit log (180 days) before revoking anything. Also check deploy keys, secrets, webhooks, and collaborators on affected repos. An attacker with push access can add a webhook that survives a password reset. ## Questions this post answers ### What should I do if Railway was compromised and I use Grok's API through it? Disable your current xAI API key in the xAI Console, create a replacement, update the variable in Railway and deploy the change. Then open the Usage Explorer and group spend by API key, filtering by request IP, model, or billing item to spot requests you didn't make, and check your team members for anyone you didn't invite. _Developers auditing API usage after a platform breach can follow incident coverage like this on daily.dev._ ### What happened in the Railway security incident disclosed on October 21? Railway detected anomalous activity in the systems that store customer environment variables, and an unauthorized third party may have accessed encrypted variables, with some customers' decrypted contents exposed. Railway links it to a wider series of attacks on platform companies aimed at compromising AI inference providers, and paused all deployments during the investigation. _Teams relying on platform providers for secrets storage follow breach disclosures like this on daily.dev._ ### Why are leaked AI API keys like Anthropic or OpenAI tokens considered more dangerous than a leaked database password? A leaked database password is a problem you still have to exploit, but a leaked inference key is already money because tokens can be sold on the black market immediately. Demand is heavy, particularly from distillers with deep pockets looking to extract value from model access, according to commentary following the Railway breach. _Developers weighing how to safeguard inference keys can track these incidents on daily.dev._ ## Community take How the wider developer community reacted, aggregated from 2 discussions and 163 comments across hackernews, lobsters, x (as of 2026-10-22). **TL;DR:** Replies focus on practical remediation advice, urging immediate key rotation and checking provider usage/billing logs since redeployment is blocked until builds resume. Security Disclosure: Wednesday October 21st: The breach is treated as one symptom of a broader wave of AI-accelerated security incidents this week, with deep unease about where software security is heading and pointed criticism of Railway's response and of storing secrets as plain environment variables. **Sentiment:** 5% positive · 25% mixed · 70% skeptical **The case for** - Some find the suggested rotation order (AI keys first) sound given they're the likely target of the breach. - Some acknowledge Railway's hands may be tied during an active investigation and sympathize with its security team. - One customer defended Railway as a reasonable platform for many businesses despite the incident. **The pushback** - Rotating keys isn't enough on its own; people stress checking last-used timestamps and billing logs to confirm misuse. - Users reported their secrets were already being actively exploited by third parties before Railway publicly acknowledged the issue. - Builds being paused leaves a gap where leaked keys can be revoked but not replaced, complicating remediation. - Pausing all deployments while demanding key rotation is seen as contradictory since it effectively forces services offline. - Storing secrets as plain environment variables (rather than references to a secrets store) is criticized as an inherently risky pattern. **By community** - hackernews (heated): Discussion sprawls from sharp criticism of Railway's disclosure and secrets handling into a much larger, anxious debate about an apparent wave of AI-driven breaches and the future of open source and infrastructure security. - lobsters (mixed): No substantive comments were provided, so little can be inferred about sentiment there. - x (mixed): Replies share tactical rotation advice and flag the awkward gap between revoking keys and being unable to redeploy while builds are paused. **Hottest debate:** Security Disclosure: Wednesday October 21st: Whether this incident and the AI framing represent a genuine inflection point requiring fundamental changes to how software and secrets are built and deployed, versus being overhyped marketing-adjacent framing of an ordinary breach. **Open questions** - What should affected users do with keys they can revoke but can't yet replace since deployments are paused? - Was decrypted environment variable data actually exfiltrated via a hacked storage system rather than through runtime environment access? - How many customers were affected and how large was the actual financial/operational impact from compromised keys? - Who is responsible for this and the other concurrent breaches (PHP, eMudhra, GitLab, PingCAP, Shopify) mentioned the same week? **Highlights** > @Joshua_WD AI keys first is exactly right — they're the payout here. I'd rotate at the issuer and check provider billing before anything else, since mystery usage is the fastest proof a key got used. With builds paused, what's your play for keys you can revoke but can't redeploy yet? > — [HETPATELL\_\_ on x](https://x.com/HETPATELL__/status/2113095970883785172) > @Joshua_WD Good order. I'd add: revoke at the provider first, since you can't ship new values on Railway until builds resume anyway. > — [sudheerdotai on x](https://x.com/sudheerdotai/status/2113111054284714068) > @Joshua_WD Useful write-up. Check Railway's own notice for what applies to your account. One step people tend to skip when rotating: rotating isn't finished until you've also looked at whether the old key was used by anyone else. Most providers show last-used times or access logs, so look > — [kybr\_dev on x](https://x.com/kybr_dev/status/2113082736470806580) > Environment variables are a terrible place to store secrets. I'm just going to say it, since having said it non-stop for over 20 years, nobody seems to listen and we keep circling this drain. The 12 Factor manifesto was 100% wrong on this point. Use environment variables to point to where secrets are located, not to store their values. > — [numbsafari on hackernews · 5 comments](https://news.ycombinator.com/item?id=50188974) > Environment variables are generally fine for secrets. Environment files are problematic. Modern systems that inject secrets into the environment launch the program itself. > — [tptacek on hackernews · 1 comments](https://news.ycombinator.com/item?id=50189064) **Source threads** - [hackernews](https://news.ycombinator.com/item?id=50188228) · 308 points · 159 comments - [x](https://x.com/Joshua_WD/status/2113080928704590140) · 1 points · 3 comments - [lobsters](https://lobste.rs/s/wi4nnl/railway_security_disclosure_wednesday) · 9 points · 1 comments ## Similar posts on daily.dev - [Security Disclosure Recommendation: Check for Unexpected OpenAI Usage](https://daily.dev/posts/security-disclosure-recommendation-check-for-unexpected-openai-usage-czy8x3xra) · Railway Blog · 1 upvotes · 0 comments - [AI evaluation startup Braintrust confirms breach, tells every customer to rotate sensitive keys](https://daily.dev/posts/ai-evaluation-startup-braintrust-confirms-breach-tells-every-customer-to-rotate-sensitive-keys-dli5x1hcn) · TechCrunch · 1 upvotes · 0 comments - [Security Disclosure Recommendation: Rotate Redis Database Credentials](https://daily.dev/posts/security-disclosure-recommendation-rotate-redis-database-credentials-cffmbvy7s) · Railway Blog · 1 upvotes · 0 comments - [10 Tips for Securing Your API Keys From AI](https://daily.dev/posts/10-tips-for-securing-your-api-keys-from-ai-j649i1e7u) · Nordic APIs · 0 upvotes · 0 comments --- Tags: [#cyber](https://daily.dev/tags/cyber), [#grok](https://daily.dev/tags/grok) [View this post on daily.dev](https://daily.dev/posts/railway-security-incident-roll-your-keys-and-check-for-unexpected-grok-usage-z8micfw58) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Railway security incident: roll your keys and check for unexpected Grok usage","url":"https://daily.dev/posts/railway-security-incident-roll-your-keys-and-check-for-unexpected-grok-usage-z8micfw58","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/railway-security-incident-roll-your-keys-and-check-for-unexpected-grok-usage-z8micfw58"},"datePublished":"2026-10-22T01:47:30.190Z","dateModified":"2026-10-22T07:01:07.145Z","description":"Railway disclosed a security incident on October 21 affecting systems that store customer environment variables. An unauthorized third party may have accessed...","isAccessibleForFree":true,"articleSection":"Collections","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Collections","logo":"https://media.daily.dev/image/upload/s--fk_6ycEi--/f_auto,q_auto/v1780996001/logos/collections?_a=BAMAMiWQ0","url":"https://daily.dev/sources/collections"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/railway-security-incident-roll-your-keys-and-check-for-unexpected-grok-usage-z8micfw58","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":1},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"cyber,grok","timeRequired":"PT5M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Collections","item":"https://daily.dev/sources/collections"},{"@type":"ListItem","position":3,"name":"Railway security incident: roll your keys and check for unexpected Grok usage"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/railway-security-incident-roll-your-keys-and-check-for-unexpected-grok-usage-z8micfw58#faq","mainEntity":[{"@type":"Question","name":"What should I do if Railway was compromised and I use Grok's API through it?","acceptedAnswer":{"@type":"Answer","text":"Disable your current xAI API key in the xAI Console, create a replacement, update the variable in Railway and deploy the change. Then open the Usage Explorer and group spend by API key, filtering by request IP, model, or billing item to spot requests you didn't make, and check your team members for anyone you didn't invite. Developers auditing API usage after a platform breach can follow incident coverage like this on daily.dev."}},{"@type":"Question","name":"What happened in the Railway security incident disclosed on October 21?","acceptedAnswer":{"@type":"Answer","text":"Railway detected anomalous activity in the systems that store customer environment variables, and an unauthorized third party may have accessed encrypted variables, with some customers' decrypted contents exposed. Railway links it to a wider series of attacks on platform companies aimed at compromising AI inference providers, and paused all deployments during the investigation. Teams relying on platform providers for secrets storage follow breach disclosures like this on daily.dev."}},{"@type":"Question","name":"Why are leaked AI API keys like Anthropic or OpenAI tokens considered more dangerous than a leaked database password?","acceptedAnswer":{"@type":"Answer","text":"A leaked database password is a problem you still have to exploit, but a leaked inference key is already money because tokens can be sold on the black market immediately. Demand is heavy, particularly from distillers with deep pockets looking to extract value from model access, according to commentary following the Railway breach. Developers weighing how to safeguard inference keys can track these incidents on daily.dev."}}]} ```