XZ Backdoor: The full story in one place

In early 2023, a backdoor was discovered in XZ Utils, a popular file compression tool used across Linux systems, which could have allowed hackers to take control of countless computers worldwide. Here's a quick rundown:

Backdoor Discovery: Detected in early 2023, revealing hidden malicious code in XZ Utils versions 5.6.0 to 5.6.1.
How It Works: The backdoor exploited the liblzma library, enabling unauthorized remote command execution during SSH logins.
The Culprit: Known as Jia Tan, this individual gained trust and commit access over two years, gradually implementing the backdoor.
Immediate Response: Quick action included shutting down affected repositories, freezing Jia Tan's GitHub, and releasing patches.
Implications: Highlighted vulnerabilities in open-source project management and the need for more rigorous code review and access controls.

The incident underscores the importance of vigilance and robust security practices in safeguarding the software supply chain from such insidious threats.

What is XZ Utils?

XZ Utils is a free tool that helps make files smaller on Linux and similar systems. It's great for saving space because it can shrink files a lot without losing important details.

Here's what you should know about XZ Utils:

It's really good at making files smaller, often 30-50% more than other tools like gzip or bzip2.
It uses formats called .xz and .lzma for shrinking files. .xz is the one most people use because it can compress files at different levels.
Opening and using .xz files is quick, which is handy.
It comes with simple commands like xz to make files smaller, unxz to bring them back to their original size, and xzcat to look inside compressed files without changing them.
It's important for many big projects and is used by a lot of Linux systems like Fedora and CentOS.

Because it's so efficient and easy to use, XZ Utils has become the go-to choice for making files smaller in the Linux world.

Why is it so widely used?

XZ Utils is everywhere in Linux for a few good reasons:

Top-notch Compression: It can shrink files more than other tools, saving space and making it easier to share files online.
Works with Old Files: It can still open older .lzma files, making it easy to switch from the old format.
Fast: Opening smaller files is quick, which is great when you're in a hurry.
Big Projects Use It: It's part of Linux itself and many other key projects.
The Default Choice: Many Linux versions choose XZ for shrinking files instead of older tools.

In short, XZ Utils is popular because it's really good at making files smaller, works fast, and is used by many important Linux projects. It's the preferred choice for many in the Linux and open source community.

Anatomy of the XZ Backdoor

Backdooring SSH

The hackers found a way to sneak into the system by messing with the liblzma library, which is part of how the SSH service on Linux systems works. This trick allowed them to give commands to the SSH server without changing any files, making it super hard to spot.

Here's what they did: they added some bad code that watches for a specific secret key during the SSH login process. If this key shows up, the code secretly runs commands with the same power as the SSH service.

This means the attackers could control the server from anywhere just by using this secret key. And because they focused on the liblzma library, which SSH uses, they didn't have to change anything directly related to SSH, keeping their backdoor hidden.

A Stealthy Implementation

The hackers were really sneaky in how they hid their backdoor:

They scrambled the code and used tricks to avoid getting caught by tools that check for suspicious code.
They made sure the backdoor didn't change how things normally work, so no one would notice anything odd.
The backdoor was set up to only work in very specific situations, like on certain Linux systems and when certain types of files were being made. This helped keep it hidden during everyday use.

They even made a way to add more sneaky features later on without leaving any clues.

All these tricks made the backdoor really hard to find. It was like looking for a needle in a haystack, but even harder because the needle was disguised as hay.

The Culprit: Jia Tan

Jia Tan is the name used by the person who put the backdoor in the XZ Utils software. But, we don't really know much about who Jia Tan really is.

A Persistent Scheme

For two years, people saying they were Jia Tan kept complaining that XZ Utils was being updated too slowly. They pushed the main developer to trust Jia Tan with special access to make changes to the project.

After getting this access, Jia Tan made a lot of normal changes to seem trustworthy. But it looks like this was all part of a plan to sneak in harmful code without anyone noticing.

Who is Jia Tan?

Jia Tan said they were from California, but when people looked into the changes Jia Tan made, they thought Jia Tan might actually be from Eastern Europe or Russia.

We still don't know much about Jia Tan's real identity or if they're working with a bigger group. The clever way they hid the harmful code makes some think a government might be involved. More digging is needed to figure out the full story.

For now, Jia Tan is a mystery. They tricked a lot of people into trusting them and used that trust to spread harmful software around the world. This shows how open projects that anyone can join can be tricked, and reminds us to be careful about who we trust with important tasks.

Timeline of Events

Early Compromise

In early 2021, someone using the name "Jia Tan" started to prepare for an attack on XZ Utils. They made fake online accounts and got involved in the XZ Utils community by talking and helping out. This helped Jia Tan look trustworthy and eventually get the power to make changes to XZ Utils by early 2023.

Gaining Commit Access

By January 2023, Jia Tan had shown they could help by making small, safe changes. This earned them the trust to make bigger changes directly to XZ Utils. They kept up the act of being helpful while planning to add harmful code.

The Backdoor Insertion

From February to March 2024, Jia Tan sneakily added dangerous code to XZ Utils versions 5.6.0 to 5.6.1. This code was hidden really well so automated checks wouldn't notice it. It was meant to let attackers secretly control parts of the system that use XZ Utils for secure connections.

Discovery and Disclosure

On April 1, 2024, Andres Freund found something odd about how XZ Utils was working with secure connections. A closer look showed there was a hidden harmful code that had been missed for over a year.

After checking everything carefully, Freund shared this discovery on April 4. This set off urgent warnings and a lot of checking and fixing by people who care about keeping systems safe.

The Aftermath

Immediate Actions Taken

Right after the news about the XZ Utils backdoor came out, people moved fast to stop it from causing more trouble:

The Tukaani GitHub page was shut down to stop the spread of the bad code.
Jia Tan's GitHub was frozen while they looked into it.
CISA and other big names sent out warnings and made tools to find if systems were hit.
Key Linux groups quickly shared fixes or ways to avoid the backdoor.

These quick steps helped keep the damage low and stopped more attacks using the bad code Jia Tan put in.

At the same time, the community that makes open software started to check Jia Tan's other work to see if there were more hidden attacks. They looked at projects like libarchive and oss-fuzz but didn't find anything else bad.

This whole thing showed how much trust and power developers can get and how they can use it to sneak bad stuff into important software.

Prevention Challenges

The trouble with XZ Utils showed a few big problems in keeping open software safe from these kinds of sneak attacks:

Trust Needs Checks: Jia Tan worked for years to seem trustworthy before doing anything bad. This means we need to be careful and check everyone's work, even if they seem okay.
Tools Miss Things: The bad code got past checks by messing with how builds are done and hiding what it was really doing. Having a better way to check big changes could have caught this.
Small Projects Can Cause Big Problems: Since XZ Utils is used so much in Linux, the attack could have been really bad. Maybe using more different tools and picking safer options could have helped.

In the end, everyone involved in making, sharing, and using open software has to help watch out for threats like this. We'll need to change how we do things, use better tools, and follow new rules to keep software safe after learning from this attack.

Implications and Lessons

Impact on Open Source

The issue with XZ Utils has made it clear that working together on software that anyone can use and change has its risks. Trusting each other too much can make these projects easy targets for bad actors.

Now, people who make this kind of software are thinking about how to keep things safe without shutting everyone out. They're considering steps like:

Checking all code changes more carefully, even from people who've been around for a while.
Using tools that make sure the software builds the same way every time, so others can check it's right.
Making sure that when someone adds something to the project, it's really from them by checking their digital signature.
Using automatic checks to spot problems in code before it's added to the project.

Projects also need to think twice before letting someone change the code directly. Even though Jia Tan seemed trustworthy after a long time, being more cautious from the start could have avoided this mess.

This situation has been a big reminder that being too trusting can be a problem. It's shown that open source projects need to make some changes to stay safe. Adding a few extra steps might slow things down a bit, but it's worth it to avoid problems like this.

Improving Supply Chain Security

The trouble with XZ Utils has taught us that keeping the software supply chain safe means being on the lookout in many areas:

Code Audits

Check over the parts of open source software that are really important, especially before letting someone make big changes.
Always review updates carefully.
Use tools that automatically check code for problems.

Artifact Verification

Make sure the software packages you download are what they're supposed to be by checking their checksums and signatures.
Use reproducible builds to double-check that the final software matches the original code.

Access Controls

Be very careful about who you let make changes to the project.
Use extra security steps like MFA (Multi-Factor Authentication) for important accounts.
Watch for any unusual activity in accounts.

No one method can catch every threat. Using a mix of these safety steps can help reduce the chance of a big problem happening.

Even though these extra steps might seem like a hassle at first, they're really important for keeping things safe in the long run. Considering how big supply chain attacks can get, it's really important for organizations to invest in keeping the software supply chain secure.

Conclusion

The XZ Utils backdoor found by Jia Tan really shook up the open source world. Although we managed to control the damage quickly, it showed us how vulnerable we are when we trust each other too much in making software together.

Open source projects are all about trust - we believe that everyone adding to the project means well and that the code they add is safe. But this time, someone took advantage of that trust. They worked hard to get the ability to change the code and then hid bad stuff in it. This shows that even parts of software we don't think much about, like those that make files smaller, can be weak spots if bad guys get in.

Here's what we learned about our security weaknesses:

Checking who can change the code isn't strict enough because we want to keep things open. We need to be better at making sure who people are and what they can do.
Looking over the code doesn't always catch sneaky dangers. We might need to check things more closely, even if it slows us down.
Using software made by others can make it easier for attackers to get in. We should think about how we build our software to keep risks low.

Making things safer might mean making some tough choices:

If we make it harder to change the code or require more checks, it could turn away some people who want to help.
Asking for more security steps, like using codes to prove who you are, could make contributing harder.

But we need to make some changes to keep bad things from happening. Here are some ideas:

Use something like a two-step code when you want to add to the code.
Make sure that when someone adds something, we can prove it's really them by checking their signature.
Use tools that automatically look for problems in the code.
Trust people more when they've been helping out and doing good things for a long time.

The open source community did a great job finding, sharing, and fixing this problem together. It shows we're good at working together to make software better for everyone. We might have to give up a little bit of our openness to stay safe, but it's important that we keep the spirit of making software together strong.

What we know about the XZ Utils backdoor that almost infected the world?

XZ Utils backdoor

The XZ Utils backdoor was a harmful piece of code added to versions 5.6.0 and 5.6.1 of the software. This code changed the way the SSH service (a secure way to log into computers) works to let unauthorized people access systems from afar. Luckily, security experts found and fixed the problem quickly before it could cause widespread harm.

How was the XZ backdoor found?

On March 29th, 2024, Andres Freund noticed something odd with the SSH service while doing some routine checks. Digging deeper, he found the sneaky backdoor code in the XZ Utils library, which works with SSH. He let everyone know about this problem on April 1st, leading to quick fixes and careful examination.

What is the XZ exploit?

The XZ exploit is the name for the backdoor code put into the XZ Utils software for compressing and decompressing data. It gave hackers a way to take over computers using the affected XZ Utils library. This serious issue was introduced by someone named "Jia Tan" and was uncovered by Andres Freund on March 29, 2024.

What is XZ in Linux?

XZ Utils is a very popular tool in Linux and other similar systems for making files smaller or returning them to their original size. It's known for the .xz and .lzma formats and is used almost everywhere in Linux, making it a key part of many projects and systems.