Saw this on nix, which was using a compromised version in the unstable channel, I hope not too many systems are affected.

ifunc is a GNU method of interposing function calls with platform-optimized versions of the function. It is used to detect CPU features at runtime and insert, for example, AVX2-optimized versions of memcmp. It is seen in crypto a lot, because CPUs have many crypto-specific instructions.However, I don&#x27;t like it much and I think software should be compiled for the target machine in the first place. My 1 hardened system that is reachable from the public network is based on musl, built mostly with llvm, and with ifunc disabled.

<a href="https:&#x2F;&#x2F;sourceware.org&#x2F;glibc&#x2F;wiki&#x2F;GNU_IFUNC" rel="nofollow">https:&#x2F;&#x2F;sourceware.org&#x2F;glibc&#x2F;wiki&#x2F;GNU_IFUNC</a>

I&#x27;m curious now. What is ifunc? (Had difficulty finding it through a search)

Is there a way to easily&#x2F;reliably disable ifunc globally on a system (e.g. ubuntu&#x2F;debian) without breaking a bunch of things?FYI this looks for pkgs with liblzma:&gt; dpkg -l |grep liblzmaVersions &gt;= 5.6 are compromised

<a href="https:&#x2F;&#x2F;github.com&#x2F;google&#x2F;oss-fuzz&#x2F;pull&#x2F;10667">https:&#x2F;&#x2F;github.com&#x2F;google&#x2F;oss-fuzz&#x2F;pull&#x2F;10667</a>

There is a related issue for LLVM&#x2F;clang by this person:<a href="https:&#x2F;&#x2F;github.com&#x2F;llvm&#x2F;llvm-project&#x2F;issues&#x2F;63957">https:&#x2F;&#x2F;github.com&#x2F;llvm&#x2F;llvm-project&#x2F;issues&#x2F;63957</a>

Interesting, I used <a href="https:&#x2F;&#x2F;ossinsight.io&#x2F;analyze&#x2F;JiaT75" rel="nofollow">https:&#x2F;&#x2F;ossinsight.io&#x2F;analyze&#x2F;JiaT75</a> to identify contributions from the account used by author of the backdoor. It looks like the account made other potentially problematic contributions to other projects.The disabling of ifunc in this PR against Google&#x27;s oss-fuzz project maybe one way they tried to prevent this particular backdoor being flagged by that tool? <a href="https:&#x2F;&#x2F;github.com&#x2F;google&#x2F;oss-fuzz&#x2F;pull&#x2F;10667">https:&#x2F;&#x2F;github.com&#x2F;google&#x2F;oss-fuzz&#x2F;pull&#x2F;10667</a>

Interestingly: <a href="https:&#x2F;&#x2F;github.com&#x2F;tukaani-project&#x2F;xz&#x2F;issues&#x2F;62#issuecomment-1719489932">https:&#x2F;&#x2F;github.com&#x2F;tukaani-project&#x2F;xz&#x2F;issues&#x2F;62#issuecomment...</a>

Safety through obscurity and weirdness! If you disable ifunc, like any sensible person, this backdoor disables itself.

FreeBSD is not affected as the payloads in question were stripped out, however we are looking into improvements to our workflow to further improve the import process.

&gt; openssh does not directly use liblzma. However debian and several other distributions patch openssh to support systemd notification, and libsystemd does depend on lzma.It looks to be limited to Linux systems that are running certain patches. macOS and BSD seem unaffected?

&gt; One portion of the backdoor is solely in the distributed tarballs. For
easier reference, here&#x27;s a link to debian&#x27;s import of the tarball, but it is
also present in the tarballs for 5.6.0 and 5.6.1:Ubuntu 22.04 version:dpkg -l |grep liblzma
ii liblzma5:amd64 5.2.5-2ubuntu1 amd64 XZ-format compression libraryWhew!

I don&#x27;t think you need to worry about the C++ contribution: <a href="https:&#x2F;&#x2F;github.com&#x2F;MicrosoftDocs&#x2F;cpp-docs&#x2F;commit&#x2F;9a96311122acd6c6de1e2624cd0f95655fac6500">https:&#x2F;&#x2F;github.com&#x2F;MicrosoftDocs&#x2F;cpp-docs&#x2F;commit&#x2F;9a96311122a...</a>

No one is being “required by law” to add vulnerabilities, it’s more likely they are foreign agents to begin with.

&gt; appears to have a Chinese nameGiven the complexity of the attack, I&#x27;d assume the name is fake.

I would think a Chinese state hacker would not assume a Chinese name, just in case he was discovered like now.

The contribution to C++ is just a simple markdown change: <a href="https:&#x2F;&#x2F;github.com&#x2F;MicrosoftDocs&#x2F;cpp-docs&#x2F;pull&#x2F;4716">https:&#x2F;&#x2F;github.com&#x2F;MicrosoftDocs&#x2F;cpp-docs&#x2F;pull&#x2F;4716</a> C++ is fine.

One thing to note is that the person that added the commits only started contributing around late 2022 and appears to have a Chinese name. Might be required by law to plant the backdoor.That would be quite scary considering they have contributed to a wide variety of projects including C++ <a href="https:&#x2F;&#x2F;learn.microsoft.com&#x2F;en-us&#x2F;cpp&#x2F;overview&#x2F;whats-new-cpp-docs?view=msvc-170" rel="nofollow">https:&#x2F;&#x2F;learn.microsoft.com&#x2F;en-us&#x2F;cpp&#x2F;overview&#x2F;whats-new-cpp...</a>

LANG only needs to have some value, the concrete value does not seem to matter.

Very strange behavior from the upstream developers. Possible government involvement? I have a feeling LANG is checked to target servers from particular countries

&gt; Not sure what I&#x27;d have done earlier in my careerTo anybody in this sorta situation, you should absolutely share whatever you have. It doesn’t need to be perfect, good, or 100% accurate, but if there’s a risk you could help a lot of people

I hope you&#x27;ve hired a PR person for all the interviews :)

This story is an incredible testament to how open-source software can self-regulate against threats, and more broadly, it reminds us that we all stand on the shoulders of contributors like you. Thank you!

FWIW, it felt intimidating as hell. And I&#x27;m fairly established professionally. Not sure what I&#x27;d have done earlier in my career (although I&#x27;d probably not have found it in the first place).

For what it&#x27;s worth the author is a PostgreSQL committer, he&#x27;s not a security researcher but he&#x27;s a pretty damn good engineer!

Honestly, you only get this kind of humility when you&#x27;re working with absolute wizards on a consistent basis. That&#x27;s how I read that whole analysis. Absolutely fascinating.

<pre><code> I am *not* a security researcher, nor a reverse engineer. There&#x27;s lots of
 stuff I have not analyzed and most of what I observed is purely from
 observation rather than exhaustively analyzing the backdoor code.</code></pre>
I love this sort of technical writing from contributors outside the mainstream debugging world who might be averse to sharing. What an excellently summarized report of his findings that should be seen as a template.

On an MX Linux (non-systemd Debian-derived distro) box I ran ldd on &#x2F;sbin&#x2F;ssh and also ran:[EDIT: this string gives cleaner results:]<pre><code> lsof -w -P -T -p $(pgrep sshd)|grep mem
</code></pre>
and saw liblzma in the results of both, so there is some sort of similar trickery going on.

FWIW, I did a quick check on a Devuan system.
The sshd in Devuan does link to a libsystemd stub - this is to cut down on their maintenance of upstream packages.
However that stub does not link to lzma.

Is that protocol documented&#x2F;stable? For whatever reason, daemons are choosing to link to libsystemd instead of implementing it themselves.It doesn&#x27;t matter that libsystemd links to liblzma for other reasons. It&#x27;s still in the address space of any daemon that is using libsystemd for the notification protocol.

Strange protocol. Why not pass a path to a file that should be `touch`d and&#x2F;or written to, I wonder? Would avoid the complexity of sockets.

&gt; libsystemd links to liblzma for other features not related to notificationsWhich is pretty emblematic of systemd&#x27;s primary architectural fault!

The funny thing is that libsystemd _used_ to be split into several different libraries. I certainly remember libsystemd-journal (which is presumably the part of libsystemd that pulls in liblzma) being separate to libsystemd-daemon (which is the part that implements sd_notify, as used by OpenSSH [after patching by distros]).If that split had never happened, then liblzma wouldn&#x27;t have ended up being linked into sshd...

&gt; The systemd notification protocol could have been as simple as just writing a newline to a pipeIt basically is. libsystemd links to liblzma for other features not related to notifications.(The protocol is that systemd passes the path to a unix socket in the `NOTIFY_SOCKET` env variable, and the daemon writes &quot;READY=1&quot; into it.)

You&#x27;re not wrong. Had Debian not patched it in this way, OP might have never found it, leaving all other distros who do the same vulnerable.Note that OP found this in Debian sid as well, which means it&#x27;s highly unlikely this issue will find its way into any Debian stable systems.

Also thanks to Debian for modifying openssh.

This is what I did for a daemon I&#x27;m maintaining. Type=notify support was requested but I&#x27;m really allergic to adding new libs to a project until they really do some heavy lifting and add enough value. I was pleasantly surprised the protocol was that simple and implemented it myself. I think systemd should just provide a simple standalone reference implementation and encourage people to copy it into their project directly. (But maybe they already do, I did that almost a decade ago IIRC when the feature was relatively new.)

Whoops, you forgot `vsock:`, `@`, `SO_PASSCRED` (I think)... oh and where is that example provided? But yep that&#x27;s all the protocol is for sure (and forever)!

That is all the protocol is. From <a href="https:&#x2F;&#x2F;www.freedesktop.org&#x2F;software&#x2F;systemd&#x2F;man&#x2F;latest&#x2F;sd_notify.html" rel="nofollow">https:&#x2F;&#x2F;www.freedesktop.org&#x2F;software&#x2F;systemd&#x2F;man&#x2F;latest&#x2F;sd_n...</a>:&gt; These functions send a single datagram with the state string as payload to the socket referenced in the $NOTIFY_SOCKET environment variable.The simplest implementation (pseudocode, no error handling, not guaranteed to compile), is something like:<pre><code> const char *addrstr = getenv(&quot;NOTIFY_SOCKET&quot;);
 if (addrstr) {
 int fd = socket(AF_UNIX, SOCK_DGRAM, 0);
 struct sockaddr_un addr = { .sun_family = AF_UNIX };
 strncpy(addr.sun_path, sizeof(addr.sun_path), addrstr);
 connect(fd, (struct sockaddr*) &amp;addr);
 write(fd, &quot;READY=1&quot;);
 close(fd);
 }</code></pre>

I have a design in the works to do just this.The problem? It&#x27;s on the backburner because I don&#x27;t think I could find a business model to make money from it.I don&#x27;t think offering support for a price would work, for example.

&gt; Ah well. I&#x27;m still waiting for someone to basically fork systemd, remove all the fluff (udev, ntp, dns, timers, restart code, specialized logging, on and on and on)Most of the things you named there are modular and can be easily disabled.Furthermore, udev precedes systemd and systemd has in fact its own replacement for it (though the name escapes me).Kind of a classic, people loving harping on systemd without properly understanding it.

Systemd is actually pretty damn good and it&#x27;s GPL licensed free software.I understand that people don&#x27;t like the way it seems to work itself into the rest of Linux user space as a dependency but that&#x27;s actually our own fault for not investing the man power that Red Hat invests. We have better things to do than make our own Linux user space and so they have occupied that niche. It&#x27;s free software though, we always have the freedom to do whatever we want.By the way, all the stuff you mentioned is not really part of the actual init system, namely PID 1. There&#x27;s an actual service manager for example and it&#x27;s entirely separate from init. It manages services really well too, it&#x27;s measurably better than all that &quot;portable&quot; nonsense just by virtue of using cgroups to manage processes which means it can actually supervise poorly written double forking daemons.

This is a bit like complaining that the Linux kernel has 30 million lines of code, while ignoring that 3&#x2F;4 of that is in hardware support (drivers) or filesystems that nobody is actually required to use at any given time.systemd is a collection of tools, one of which is an init system. Nobody accused GNU yes of being bloated just because it&#x27;s in a repository alongside 50 other tools.

&gt; One of the objections that many people do not understand, is that systemd adds complexity. Unnecessary complexity. Boats full, loads full, mountains full of complexity.this is and always has been such a dumb take.if you&#x27;d like to implement an init (and friends) system that doesn&#x27;t have &quot;unnecessary complexity&quot; and still provides all the functionality that people currently want, then go and do so and show us? otherwise it&#x27;s just whinging about things not being like the terrible old days of init being a mass of buggy and racey shell scripts.

As long as Gnome requires bug-compatibility with systemd, nobody will rewrite it.

&gt; One of the objections that many people do not understand, is that systemd adds complexity. Unnecessary complexity. Boats full, loads full, mountains full of complexity.Complexity that would otherwise be distributed to a sea of ad-hoc shell scripts? systemd is a win

One of the objections that many people do not understand, is that systemd adds complexity. Unnecessary complexity. Boats full, loads full, mountains full of complexity.Yes, there are things delivered with that complexity. However, as an example, sysvinit is maybe, oh, 20k lines of code including binaries, heck including all core init scripts.What&#x27;s systemd? 2M lines? It was &gt;1M lines 4+ years ago.For an init system, a thing that is to be the core of stability, security, and most importantly glacial, stable change -- that is absurdly complex. It&#x27;s exceedingly over engineered.And so you get cases like this. And cases like that, and that over there, and that case over there too. All which could not exist, if systemd didn&#x27;t try to overengineer, over complicate everything.Ah well. I&#x27;m still waiting for someone to basically fork systemd, remove all the fluff (udev, ntp, dns, timers, restart code, specialized logging, on and on and on), and just end up with systemd compatible service files.But not yet. So... well, oh well.

1. systemd (at least the PID 1 part) does not talk to the network, so a remotely-accessible backdoor would need to be more complex (and thus more likely to be detected) than a backdoor that can be loaded into a listening daemon like openssh.2. You can run Debian systems without systemd as PID 1, but you&#x27;re still stuck with libsystemd because so many daemons now link with it.

&gt; so now security-critical daemons like openssh have additional dependencies like liblzmaSystemd itself seems security-critical to me. Would removing other dependencies on libsystemd really make a secure system where systemd was compromised through its library?

Deferring the load of the library often just makes things harder to analyze, not necessarily more secure. I imagine many of the comments quoting `ldd` are wrongly forgetting about `dlopen`.(I really wish there were a way to link such that the library isn&#x27;t actually loaded but it still shows in the metadata, so you can get the performance benefits of doing less work but can still analyze the dependency DAG easily)

&gt; And i tell pretty much anyone who wants to listen that they should just implement the proto on their own if thats rhe only reason for a libsystemd dep otherwise.That&#x27;s what I think too. Do the relevant docs point this out too? Ages ago they didn&#x27;t. I think we should try to avoid that people just google &quot;implement systemd notify daemon&quot; and end up on a page that says &quot;link to libsystemd and call sd_notify()&quot;.

&gt; And i tell pretty much anyone who wants to listen that they should just implement the proto on their own if thats rhe only reason for a libsystemd dep otherwiseCould you point out where the man page (<a href="https:&#x2F;&#x2F;www.freedesktop.org&#x2F;software&#x2F;systemd&#x2F;man&#x2F;latest&#x2F;sd_notify.html" rel="nofollow">https:&#x2F;&#x2F;www.freedesktop.org&#x2F;software&#x2F;systemd&#x2F;man&#x2F;latest&#x2F;sd_n...</a>) says this?

And will end up in sshd too (at the very least via libpam&#x2F;pam_selinux).Inaccurate.It&#x27;s not pulled in on any sysvinit Debian system I run. It is on stable, oldstable, and oldoldstable systems via systemd.Not systemd:# ldd $(which sshd)
 linux-vdso.so.1 (0x00007ffcb57f5000)<pre><code> libcrypt.so.1 =&gt; &#x2F;lib&#x2F;x86_64-linux-gnu&#x2F;libcrypt.so.1 (0x00007fbad13c9000)

 libwrap.so.0 =&gt; &#x2F;lib&#x2F;x86_64-linux-gnu&#x2F;libwrap.so.0 (0x00007fbad13bd000)

 libaudit.so.1 =&gt; &#x2F;lib&#x2F;x86_64-linux-gnu&#x2F;libaudit.so.1 (0x00007fbad138c000)

 libpam.so.0 =&gt; &#x2F;lib&#x2F;x86_64-linux-gnu&#x2F;libpam.so.0 (0x00007fbad137a000)

 libsystemd.so.0 =&gt; &#x2F;lib&#x2F;x86_64-linux-gnu&#x2F;libsystemd.so.0 (0x00007fbad12d5000)

 libselinux.so.1 =&gt; &#x2F;lib&#x2F;x86_64-linux-gnu&#x2F;libselinux.so.1 (0x00007fbad12a5000)

 libgssapi_krb5.so.2 =&gt; &#x2F;lib&#x2F;x86_64-linux-gnu&#x2F;libgssapi_krb5.so.2 (0x00007fbad1253000)

 libkrb5.so.3 =&gt; &#x2F;lib&#x2F;x86_64-linux-gnu&#x2F;libkrb5.so.3 (0x00007fbad1179000)

 libcom_err.so.2 =&gt; &#x2F;lib&#x2F;x86_64-linux-gnu&#x2F;libcom_err.so.2 (0x00007fbad1173000)

 libcrypto.so.3 =&gt; &#x2F;lib&#x2F;x86_64-linux-gnu&#x2F;libcrypto.so.3 (0x00007fbad0c00000)

 libz.so.1 =&gt; &#x2F;lib&#x2F;x86_64-linux-gnu&#x2F;libz.so.1 (0x00007fbad1154000)

 libc.so.6 =&gt; &#x2F;lib&#x2F;x86_64-linux-gnu&#x2F;libc.so.6 (0x00007fbad0a1f000)

 libnsl.so.2 =&gt; &#x2F;lib&#x2F;x86_64-linux-gnu&#x2F;libnsl.so.2 (0x00007fbad1137000)

 libcap-ng.so.0 =&gt; &#x2F;lib&#x2F;x86_64-linux-gnu&#x2F;libcap-ng.so.0 (0x00007fbad112f000)

 libcap.so.2 =&gt; &#x2F;lib&#x2F;x86_64-linux-gnu&#x2F;libcap.so.2 (0x00007fbad1123000)

 &#x2F;lib64&#x2F;ld-linux-x86-64.so.2 (0x00007fbad156a000)

 libpcre2-8.so.0 =&gt; &#x2F;lib&#x2F;x86_64-linux-gnu&#x2F;libpcre2-8.so.0 (0x00007fbad1089000)

 libk5crypto.so.3 =&gt; &#x2F;lib&#x2F;x86_64-linux-gnu&#x2F;libk5crypto.so.3 (0x00007fbad09f2000)

 libkrb5support.so.0 =&gt; &#x2F;lib&#x2F;x86_64-linux-gnu&#x2F;libkrb5support.so.0 (0x00007fbad09e4000)

 libkeyutils.so.1 =&gt; &#x2F;lib&#x2F;x86_64-linux-gnu&#x2F;libkeyutils.so.1 (0x00007fbad09dd000)

 libresolv.so.2 =&gt; &#x2F;lib&#x2F;x86_64-linux-gnu&#x2F;libresolv.so.2 (0x00007fbad09cc000)

 libtirpc.so.3 =&gt; &#x2F;lib&#x2F;x86_64-linux-gnu&#x2F;libtirpc.so.3 (0x00007fbad099e000)

</code></pre>
systemd:# ldd $(which sshd)
 linux-vdso.so.1 (0x00007ffc4d3eb000)<pre><code> libcrypt.so.1 =&gt; &#x2F;lib&#x2F;x86_64-linux-gnu&#x2F;libcrypt.so.1 (0x00007feb8aa35000)

 libwrap.so.0 =&gt; &#x2F;lib&#x2F;x86_64-linux-gnu&#x2F;libwrap.so.0 (0x00007feb8aa29000)

 libaudit.so.1 =&gt; &#x2F;lib&#x2F;x86_64-linux-gnu&#x2F;libaudit.so.1 (0x00007feb8a9f8000)

 libpam.so.0 =&gt; &#x2F;lib&#x2F;x86_64-linux-gnu&#x2F;libpam.so.0 (0x00007feb8a9e6000)

 libsystemd.so.0 =&gt; &#x2F;lib&#x2F;x86_64-linux-gnu&#x2F;libsystemd.so.0 (0x00007feb8a916000)

 libselinux.so.1 =&gt; &#x2F;lib&#x2F;x86_64-linux-gnu&#x2F;libselinux.so.1 (0x00007feb8a8e6000)

 libgssapi_krb5.so.2 =&gt; &#x2F;lib&#x2F;x86_64-linux-gnu&#x2F;libgssapi_krb5.so.2 (0x00007feb8a894000)

 libkrb5.so.3 =&gt; &#x2F;lib&#x2F;x86_64-linux-gnu&#x2F;libkrb5.so.3 (0x00007feb8a7ba000)

 libcom_err.so.2 =&gt; &#x2F;lib&#x2F;x86_64-linux-gnu&#x2F;libcom_err.so.2 (0x00007feb8a7b4000)

 libcrypto.so.3 =&gt; &#x2F;lib&#x2F;x86_64-linux-gnu&#x2F;libcrypto.so.3 (0x00007feb8a200000)

 libz.so.1 =&gt; &#x2F;lib&#x2F;x86_64-linux-gnu&#x2F;libz.so.1 (0x00007feb8a795000)

 libc.so.6 =&gt; &#x2F;lib&#x2F;x86_64-linux-gnu&#x2F;libc.so.6 (0x00007feb8a01f000)

 libnsl.so.2 =&gt; &#x2F;lib&#x2F;x86_64-linux-gnu&#x2F;libnsl.so.2 (0x00007feb8a778000)

 libcap-ng.so.0 =&gt; &#x2F;lib&#x2F;x86_64-linux-gnu&#x2F;libcap-ng.so.0 (0x00007feb8a770000)

 libcap.so.2 =&gt; &#x2F;lib&#x2F;x86_64-linux-gnu&#x2F;libcap.so.2 (0x00007feb8a764000)

 libgcrypt.so.20 =&gt; &#x2F;lib&#x2F;x86_64-linux-gnu&#x2F;libgcrypt.so.20 (0x00007feb89ed8000)

 liblzma.so.5 =&gt; &#x2F;lib&#x2F;x86_64-linux-gnu&#x2F;liblzma.so.5 (0x00007feb8a735000)

 libzstd.so.1 =&gt; &#x2F;lib&#x2F;x86_64-linux-gnu&#x2F;libzstd.so.1 (0x00007feb89e1c000)

 liblz4.so.1 =&gt; &#x2F;lib&#x2F;x86_64-linux-gnu&#x2F;liblz4.so.1 (0x00007feb8a70d000)

 &#x2F;lib64&#x2F;ld-linux-x86-64.so.2 (0x00007feb8abb5000)

 libpcre2-8.so.0 =&gt; &#x2F;lib&#x2F;x86_64-linux-gnu&#x2F;libpcre2-8.so.0 (0x00007feb89d82000)

 libk5crypto.so.3 =&gt; &#x2F;lib&#x2F;x86_64-linux-gnu&#x2F;libk5crypto.so.3 (0x00007feb8a6e0000)

 libkrb5support.so.0 =&gt; &#x2F;lib&#x2F;x86_64-linux-gnu&#x2F;libkrb5support.so.0 (0x00007feb8a6d2000)

 libkeyutils.so.1 =&gt; &#x2F;lib&#x2F;x86_64-linux-gnu&#x2F;libkeyutils.so.1 (0x00007feb8a6c9000)

 libresolv.so.2 =&gt; &#x2F;lib&#x2F;x86_64-linux-gnu&#x2F;libresolv.so.2 (0x00007feb8a6b8000)

 libtirpc.so.3 =&gt; &#x2F;lib&#x2F;x86_64-linux-gnu&#x2F;libtirpc.so.3 (0x00007feb8a68a000)

 libgpg-error.so.0 =&gt; &#x2F;lib&#x2F;x86_64-linux-gnu&#x2F;libgpg-error.so.0 (0x00007feb89d5a000)

</code></pre>
EG# ldd $(which sshd) | grep liblz<pre><code> liblzma.so.5 =&gt; &#x2F;lib&#x2F;x86_64-linux-gnu&#x2F;liblzma.so.5 (0x00007fd1e647a000)

 liblz4.so.1 =&gt; &#x2F;lib&#x2F;x86_64-linux-gnu&#x2F;liblz4.so.1 (0x00007fd1e6398000)</code></pre>

The correct thing to do would be to put different unrelated APIs into their own library, instead of everything into libsystemd0. This has always been one of my biggest issues with it. It makes it hard to replace just one API from that library, because on a binary distribution, only one package can provide it. And as a nice side effect, surprises like this one could then be avoided.

IIRC sshd loads libpam only if specifically configured for it. So while it&#x27;s not wrong, it&#x27;s also a more edge case for the backdoor to work.

Uh. systemd documents the protocol at various places and the protocol is trivial: a single text datagram sent to am AF_UNIX socket whose path you get via the NOTIFY_SOCKET. That&#x27;s trivial to implement for any one with some basic unix programming knowledge. And i tell pretty much anyone who wants to listen that they should just implement the proto on their own if thats rhe only reason for a libsystemd dep otherwise. In particular non-C environments really should do their own native impl and not botjer wrapping libsystemd just for this.But let me stress two other things:Libselinux pulls in liblzma too and gets linked into tons more programs than libsystemd. And will end up in sshd too (at the very least via libpam&#x2F;pam_selinux). And most of the really big distros tend do support selinux at least to some level. Hence systemd or not, sshd remains vulnerable by this specific attack.With that in mind libsystemd git dropped the dep on liblzma actually, all compressors are now dlopen deps and thus only pulled in when needed.

What? I don&#x27;t get it? Isn&#x27;t it on Debian if they modified the package to do something like this? Why would you blame systemd for maintainers doing something that upstream has never required or recommended?

Sometimes things evolve in ways that make us feel a little obsolete.I&#x27;ve been learning NixOS for a few years now, and it would have been impossible without systemd. It&#x27;s one heck of a learning curve, but when you get to the other side, you know something of great power and value. Certain kinds of complexity adds &#x27;land&#x27; (eg. systemd) that can become &#x27;real estate&#x27; (eg. NixOS), which in turn hopes to become &#x27;land&#x27; for the next innovation, and so forth.Whether this happens or not (whether it&#x27;s the right kind of complexity) is really hard to assess up-front, and probably impossible without knowing the complex new technology in question very well. (And by then you have the bias of depending, in part, yourself on the success of the new tech, as you&#x27;ve committed significant resources to mastering it, so good luck on convincing skeptical newcomers!)It&#x27;s almost like a sort of event horizon -- once you know a complex new technology well enough to see whether or not it&#x27;s useful, the conflict-of-interest makes your opinion unreliable to outsiders!Nevertheless, the assessment process itself, while difficult to get right, is worth getting better at.It&#x27;s easy for impatience and the sensation of what I&#x27;ve taken to calling &#x27;daunt&#x27; -- that intrinsic recoil that the mind has from absorbing a large amounts of information whose use case is not immediately relevant -- to dissuade one from exploring. But then, one never discovers new &#x27;land&#x27;, and one never builds new real estate![ Aside:
This is why I&#x27;m a little skeptical of the current rebellion against frontend frameworks. Certainly some of them, like tailwind, are clearly adding fetters to an otherwise powerful browser stack. But others, like Svelte, and to some extent, even React, bring significant benefits.The rebellion has this vibe like, well, users _should_ prefer more simply-built interfaces, and if they don&#x27;t, well, they just have bad taste. What would be more humble would be to let the marketplace (e.g. consumers) decide what is preferable, and then build that. 
]

It&#x27;s unfortunate that the anti-systemd party lost the war... years ago. But I don&#x27;t blame systemd, Lennart Pottering or the fanboys (though it would have been so much better if the guy never worked in open source or wasn&#x27;t such a prolific programmer). I blame Debian and its community for succumbing to this assault on Unix philosophy (again, years ago).

The notify protocol isn&#x27;t much more complicated than that. From memory you send a string to a unix socket. I have written both systemd notify and listenfd in a few languages for little experiments and it is hard to imagine how the protocols could be simpler.Looking at most popular projects these days they are a mass of dependencies and I think very few of them can be properly audited and verified by the projects that use them. Rust and Go might be more memory safe than C but look at the number of cargo or go modules in most projects. I have mostly stopped using node&#x2F;npm on my systems.

Not a programmer, but couldn&#x27;t the distribution&#x27;s sshd patches for systemd (and all other distro patches for privileged daemons) use static includes? Wouldn&#x27;t that have only pulled in the simple client-side communication API? Would that have defeated this vector? Would it be doable?

&gt; openssh does not directly use liblzma. However debian and several other distributions patch openssh to support systemd notification, and libsystemd does depend on lzma.The systemd notification protocol could have been as simple as just writing a newline to a pipe, but instead you have to link to the libsystemd C library, so now security-critical daemons like openssh have additional dependencies like liblzma loaded into their address space (even if you don&#x27;t use systemd as PID 1), increasing the risks of supply chain attacks. Thanks, systemd.

Just because macs don&#x27;t use systemd, doesn&#x27;t mean the backdoor won&#x27;t work. The oss-sec post talks about liblzma having backdoors in crc32_resolve() and crc64_resolve() and that it has not been fully reversed. This could perhaps affect more than just sshd on x86-64 linux?

The issue is caused by patches to add integration with systemd, so no, this won&#x27;t affect SSH on a Mac.

Homebrew reverted to 5.4.6 once the maintainers became aware. The current understanding is that macOS is not affected, but that&#x27;s not certain.[1]: <a href="https:&#x2F;&#x2F;github.com&#x2F;Homebrew&#x2F;homebrew-core&#x2F;pull&#x2F;167512">https:&#x2F;&#x2F;github.com&#x2F;Homebrew&#x2F;homebrew-core&#x2F;pull&#x2F;167512</a>

Homebrew is currently shipping 5.6.1 (and was shipping 5.6.0 as well). Hopefully not affected on mac?

You can be certain it has happened, many times. Now think of all the software we mindlessly consume via docker, language package managers, and the like.Remember, there is no such thing as computer security. Make your decisions accordingly :)

Yeah, now imagine they succeeded and it didn&#x27;t cause any performance issues...Can we even be sure no such successful attempt has already been made?

This is my main take-away from this.  We must stop using upstream configure and other &quot;binary&quot; scripts.  Delete them all and run &quot;autoreconf -fi&quot; to recreate them.  (Debian already does something like this I think.)

Maybe the US Government needs to put its line in the sand and mandate the end of autotools. :D

Maybe it&#x27;s time to dramatically simplify autoconf?How long do we need to (pretend to) keep compatibility with pre-ANSI C compilers, broken shells on exotic retro-unixes, and running scripts that check how many bits are in a byte?

Autoconf is m4 macros and Bourne shell. Most mainstream programming languages have a packaging system that lets you invoke a shell script. This attack is a reminder to keep your shell scripts clean. Don&#x27;t treat them as an afterthought.

I&#x27;m wondering is there i.e. no way to add an automated flagging system that A&#x2F;B &#x2F; `diff` checks the tarball contents against the repo&#x27;s files and warns if there&#x27;s a mismatch? This would be on i.e. GitHub&#x27;s end so that there&#x27;d be this sort of automated integrity test and subsequent warning? Just a thought, since tainted tarballs like these might be altogether be (and become) a threat vector, regardless of the repo.

Not just autoconf. Build systems in general are a bad abstraction, which leads to lots and lots of code to try to make them do what you want. It&#x27;s a sad reality of the mismatch between a prodecural task (compile files X, Y, and Z into binary A) and what we want (compile some random subset of files X, Y, and Z, doing an arbitrary number of other tasks first, into binary B).For fun, you can read the responses to my musing that maybe build systems aren&#x27;t needed: <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=35474996">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=35474996</a> (People can&#x27;t imagine programming without a build system - it&#x27;s sad)

Thanks to autoconf, we&#x27;re now used to build scripts looking like gibberish.  A perfect place to hide a backdoor.

It looks like an earlier commit with a binary blob &quot;test data&quot; contained the bulk of the backdoor, then the configure script enabled it, and then later commits patched up valgrind errors caused by the backdoor. See the commit links in the &quot;Compromised Repository&quot; section.Also, seems like the same user who made these changes are still submitting changes to various repositories as of a few days ago. Maybe these projects need to temporarily stop accepting commits until further review is done?

&gt; &quot;Given the activity over several weeks, the committer is either directly
involved or there was some quite severe compromise of their
system. Unfortunately the latter looks like the less likely explanation, given
they communicated on various lists about the &quot;fixes&quot; mentioned above.&quot;Crazy indeed.

How many people read autoconf scripts, though? I think those filters are symptom of the larger problem that many popular C&#x2F;C++ codebases have these gigantic build files which even experts try to avoid dealing with. I know why we have them but it does seem like something which might be worth reconsidering now that the tool chain is considerably more stable than it was in the 80s and 90s.

I don&#x27;t see how showing the binary diffs would help. 99.99999% of people would just scroll right past them anyways.

testdata should not be on the same machine as the build is done. testdata (and tests generally) aren&#x27;t as well audited, and therefore shouldn&#x27;t be allowed to leak into the finished product.Sure - you want to test stuff, but that can be done with a special &quot;test build&quot; in it&#x27;s own VM.

00011900: 0000 4883 f804 7416 b85f 5f5f 5f33 010f  ..H...t..____3..   │                                                                                                                      
00011910: b651 0483 f25a 09c2 0f84 5903 0000 488d  .Q...Z....Y...H.   │                                                                                                                      
00011920: 7c24 40e8 5875 0000 488b 4c24 4848 3b4c  |$@.Xu..H.L$HH;L   │                                                                                                                      
00011930: 2440 7516 4885 c074 114d 85ff 0f84 3202  $@u.H..t.M....2.   │                                                                                                                      
00011940: 0000 498b 0ee9 2c02 0000 b9fe ffff ff45  ..I...,........E   │                                                                                                                      
00011950: 31f6 4885 db74 0289 0b48 8bbc 2470 1300  1.H..t...H..$p..   │                                                                                                                      
00011960: 0048 85ff 0f85 c200 0000 0f57 c00f 2945  .H.........W..)E   │                                                                                                                      
00011970: 0048 89ac 2470 1300 0048 8bbc 2410 0300  .H..$p...H..$...   │                                                                                                                      
00011980: 0048 8d84 2428 0300 0048 39c7 7405 e8ad  .H..$(...H9.t...   │                                                                                                                      
00011990: e6ff ff48 8bbc 24d8 0200 0048 8d84 24f0  ...H..$....H..$.   │                                                                                                                      
000119a0: 0200 0048 39c7 7405 e893 e6ff ff48 8bbc  ...H9.t......H..   │                                                                                                                      
000119b0: 2480 0200 0048 8d84 2498 0200 0048 39c7  $....H..$....H9.   │                                                                                                                      
000119c0: 7405 e879 e6ff ff48 8bbc 2468 0100 004c  t..y...H..$h...L   │ 
Please tell me what this code does, Sheldon

in this case the backdoor was hidden in a nesting doll of compressed data manipulated with head&#x2F;tail and tr, even replacing byte ranges inbetween. Would&#x27;ve been impossible to find if you were just looking at the test fixtures.

A big part of the problem is all the tooling around git (like the default github UI) which hides diffs for binary files like these pseudo-&quot;test&quot; files. Makes them an ideal place to hide exploit data since comparatively few people would bother opening a hex editor manually.

Unfortunately eval in a shell script has an effect on the semantics but is not necessary to do some kind of parsing of the contents of a variable, unlike Python or Perl or JavaScript. A<pre><code> $goo
</code></pre>
line (without quotes) will already do word splitting, though it won&#x27;t do another layer of variable expansion and unquoting, for which you&#x27;ll need<pre><code> eval &quot;$goo&quot;
</code></pre>
(This time with quotes).

eval in autoconf macros is nothing unusual.In (pre-backdoor) xz 5.4.5:<pre><code> $ grep -wl eval m4&#x2F;*
 m4&#x2F;gettext.m4
 m4&#x2F;lib-link.m4
 m4&#x2F;lib-prefix.m4
 m4&#x2F;libtool.m4</code></pre>

&gt; Usually there is a better way to do it in almost all cases where people feel the need to reach for &quot;eval&quot;unfortunately thats just standard in configure scripts, for example from python:```
grep eval Python-3.12.2&#x2F;configure | wc -l 
165
```and its 32,958 lines of code, plenty of binary fixtures as well in the tarball to hide stuff.who knows, but I have feeling us finding the backdoor in this case was more of a happy accident.

The use of &quot;eval&quot; stands out, or at least it should stand out – but there are two more instances of it in the same script, which presumably are not used maliciously.A while back there was a discussion[0] of an arbitrary code execution vulnerability in exiftool which was also the result of &quot;eval&quot;.Avoiding casual use of this overpowered footgun might make it easier to spot malicious backdoors. Usually there is a better way to do it in almost all cases where people feel the need to reach for &quot;eval&quot;, unless the feature you&#x27;re implementing really is &quot;take a piece of arbitrary code from the user and execute it&quot;.[0] <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=39154825">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=39154825</a>

That&#x27;s completely crazy, the backdoor is introduced through a very cryptic addition to the configure script. Just looking at the diff, it doesn&#x27;t look malicious at all, it looks like build script gibberish.

Summary: &quot;The upstream xz repository and the xz tarballs have been backdoored.&quot;It is known to be in version 5.6.0 and 5.6.1, and the obfuscated code is found in the test directory.

This potentially could be a full automated rootkit type breach right? Great - is any system with 5.6.1 possibly vulnerable?Also super weird a contributor thought they could slip this in and not have it be noticed at some point. It may point to burning that person (aka, they go to jail) for whatever they achieved with this. (And whoever they are…)

Right, but in this case it&#x27;s not even compiled it, which is arguably better than compiled in but assumed dormant :) (at least until someone actually does a full analysis of the payload).

It may not have really mattered much for NixOS:&gt; b) argv[0] needs to be &#x2F;usr&#x2F;sbin&#x2F;sshdFor once, the lack of FHS interoperability is a benefit, if only on accident.

Note that NixOS has a unique advantage in that `dlopen` is easier to analyze, but you do have to check for it. A lot of people are looking only at `ldd` and missing that they can be vulnerable at runtime.

It&#x27;s actually not an advantage. The reason why the exploit wasn&#x27;t included is because the attacker specifically decided to only inject x86_64 Debian and RHEL to reduce the chances of this getting detected.

That&#x27;s just security by obscurity, not something I&#x27;d consider a good security measure.

That&#x27;s one of the advantages of NixOS - viruses and mass hacks have lesser chance to function due to how different this OS is. Until it gets more popular, of course.

Not affected by the latest CVE, but the author had unrestricted access to xz for 2 years, so I would say it is affected until the other contributions are proven safe (never gonna happen) or it reverts to pre-adversarial actor version.

NixOS&#x2F;Pkgs 23.11 unaffected, unstable contains backdoored implementations (5.6.0, 5.6.1) but their OpenSSH sshd does not seem to link against systemd&#x2F;liblzma, and the backdoor doesn&#x27;t get configured in (only happens on .deb&#x2F;.rpm systems).

Note that Fedora 40 isn&#x27;t even released yet, it&#x27;s in beta, Fedora 41 &#x2F; rawhide is basically a development branch used only by a small number of people.

RHEL no, Fedora 41 and Rawhide yes.<a href="https:&#x2F;&#x2F;www.redhat.com&#x2F;en&#x2F;blog&#x2F;urgent-security-alert-fedora-41-and-rawhide-users" rel="nofollow">https:&#x2F;&#x2F;www.redhat.com&#x2F;en&#x2F;blog&#x2F;urgent-security-alert-fedora-...</a><a href="https:&#x2F;&#x2F;lists.debian.org&#x2F;debian-security-announce&#x2F;2024&#x2F;msg00057.html" rel="nofollow">https:&#x2F;&#x2F;lists.debian.org&#x2F;debian-security-announce&#x2F;2024&#x2F;msg00...</a>

i knew there was an advantage to being 8-10 years out of date at all times...and when they do port finally backport this bug in 2026, they will probably implement the systemd integration with openssl (pbthththt...) via 600 patch files in some nonstandard divergent manner that thwarts the payload anyhow. see? i knew they were super duper secure.

RHEL won&#x27;t get this bug for 2 years =)

Red Hat helps to do the job of making sure OSS has CVEs so there&#x27;s common vernacular for the problem.

&gt; Red Hat assigned this issue CVE-2024-3094.Does that mean this affects RHEL and Fedora?

This needs a Rust joke. You know, the problem with the whole certification charade is it slows down jobs and prevents __actual_problems getting evaluated. But is it safe?

&gt; cannot resist feeling the entire point of that industry is to make companies pay for a security certificate so you can shift the blame onto someone else when things go wrong.That&#x27;s the entire point. You did everything you could by getting someone else look at it and saying it&#x27;s fine.

&gt; the entire point of that industry is to make companies pay for a security certificate so you can shift the blame onto someone else when things go wrong.That is actually a major point of a lot of corporate security measures (shifting risk)

It&#x27;s been reverted now: <a href="https:&#x2F;&#x2F;github.com&#x2F;Homebrew&#x2F;homebrew-core&#x2F;blob&#x2F;9a0603b474804ef76cef17cf1bc90397996088bb&#x2F;Formula&#x2F;x&#x2F;xz.rb">https:&#x2F;&#x2F;github.com&#x2F;Homebrew&#x2F;homebrew-core&#x2F;blob&#x2F;9a0603b474804...</a>

Similarly if you&#x27;re using MacPorts, make sure to sync and upgrade xz if you have it installed.5.6.1 was available for a few days and just rolled back ~20 minutes ago: <a href="https:&#x2F;&#x2F;github.com&#x2F;macports&#x2F;macports-ports&#x2F;commit&#x2F;a1388aee09c9e921e3a9d47cf9d37e5d3f3c10ad">https:&#x2F;&#x2F;github.com&#x2F;macports&#x2F;macports-ports&#x2F;commit&#x2F;a1388aee09...</a>

Thank you for this tip. `brew upgrade xz` worked.I was going to uninstall but it&#x27;s used by so many things…<pre><code> brew uninstall xz
 Error: Refusing to uninstall &#x2F;opt&#x2F;homebrew&#x2F;Cellar&#x2F;xz&#x2F;5.6.1
 because it is required by aom, composer, curl, ffmpeg, gcc, gd, ghostscript, glib, google-cloud-sdk, grc, harfbuzz, httpie, img2pdf, jbig2enc, jpeg-xl, leptonica, libarchive, libavif, libheif, libraw, libtiff, libzip, little-cms2, numpy, ocrmypdf, openblas, openjpeg, openvino, php, pillow, pipx, pngquant, poppler, python@3.11, python@3.12, rsync, tesseract, tesseract-lang, unpaper, webp, wp-cli, yt-dlp and zstd, which are currently installed.</code></pre>

If you installed xz on macOS using brew, then you have<pre><code> xz (XZ Utils) 5.6.1
 liblzma 5.6.1
</code></pre>
which are within the release target for the vuln. As elsewhere in these comments, people say macOS effect is uncertain. If concerned you can revert to 5.4.6 with<pre><code> brew upgrade xz</code></pre>

&gt; And the company gets to wash their hands of liability (most of the time).Certification theater.It&#x27;s completely performative.

What you describe is a normal process in order to minimise damage from attacks. The damage of hacking is ultimately property damage. The procedures you&#x27;ve described allow you to minimise it.And that&#x27;s a good thing.

Thats basically the whole point actually... A company pays for insurance for the business. The insurance company says sure we will insure you, but you need to go through audits A B and C, and you need certifications X and Y to be insured by us. Those audits are often industry dependent, mostly for topics like HIPAA, PCI, SOC, etc.Insurance company hears about supply chain attacks. Declares that insured must have supply chain validation. Company goes and gets a shiny cert.Now when things go wrong, the company can point to the cert and go &quot;it wasnt us, see we have the cert you told us to get and its up to date&quot;. And the company gets to wash their hands of liability (most of the time).

That&#x27;s the entire point of certification, and any certification at all. Certification does not guarantee performance. Actually, I would always cast a suspect glance to anyone who is FOCUSED on getting certification after certification without any side project.

So many security companies publishing daily generic blog posts about &quot;serious supply chain compromises&quot; in various distros on packages with 0 downloads, and yet it takes a developer debugging performance issues to find an actual compromise.I worked in the software supply chain field and cannot resist feeling the entire point of that industry is to make companies pay for a security certificate so you can shift the blame onto someone else when things go wrong.

5.6.1-1 was built from what I understand to be one of the affected tarballs. This was patched in 5.6.1-2: <a href="https:&#x2F;&#x2F;gitlab.archlinux.org&#x2F;archlinux&#x2F;packaging&#x2F;packages&#x2F;xz&#x2F;-&#x2F;commit&#x2F;881385757abdc39d3cfea1c3e34ec09f637424ad" rel="nofollow">https:&#x2F;&#x2F;gitlab.archlinux.org&#x2F;archlinux&#x2F;packaging&#x2F;packages&#x2F;xz...</a>I agree on the sshd linking part.

I upgraded Arch Linux on my server a few hours ago. Arch Linux does not fetch one of the compromised tarballs but builds from source and sshd does not link against liblzma on Arch.<pre><code> [root@archlinux ~]# pacman -Qi xz | head -n2 
 Name : xz 
 Version : 5.6.1-2 
 [root@archlinux ~]# pacman -Qi openssh | head -n2
 Name : openssh
 Version : 9.7p1-1
 [root@archlinux ~]# ldd $(which sshd) | grep liblzma
 [root@archlinux ~]#
</code></pre>
It seems that Arch Linux is not affected.

My Arch setup is the same, they must not patch openssh.

On arch, `ldd $(which sshd)` doesn&#x27;t list lzma or xz, so I think it&#x27;s unaffected? Obviously still not great to be shipping malicious code that just happens to not trigger.

Are you sure about that? The diff moves away from using the compromised tarballs to the not-compromised (by this) git source. The comment message says it&#x27;s about reproducibility, but especially combined with the timing it looks to me like that was just to avoid breaking an embargo.

5.6.1-2 is not an attempted fix, it&#x27;s just some tweaks to Arch&#x27;s own build script to improve reproducibility. Arch&#x27;s build script ultimately delegates to the compromised build script unfortunately, but it also appears the payload itself is specifically targeting deb&#x2F;RPM based distros, so a narrow miss for Arch here.(EDIT: as others have pointed out, part of the exploit is in the artifact from libxz, which Arch is now avoiding by switching to building from a git checkout)

The writeup indicates that the backdoor only gets applied when building for rpm or deb, so Arch probably would have been okay either way? Same with Nix, Homebrew, etc.

The project has made an official post on the subject<a href="https:&#x2F;&#x2F;archlinux.org&#x2F;news&#x2F;the-xz-package-has-been-backdoored&#x2F;" rel="nofollow">https:&#x2F;&#x2F;archlinux.org&#x2F;news&#x2F;the-xz-package-has-been-backdoore...</a>

Looks like Arch Linux shipped both compromised versions - and 5.6.1-2 is out to hopefully resolve it.

Do you have a writeup or any details as to what it does? The logical thing based on this post is that it hooks the SSH key verification mechanism to silently allow some attacker-controlled keys but I wonder if there&#x27;s more to it?

I guess it seems like the operational parts are a bit poorly done. Valgrind issues, adding a new version with symbols removed, the aforementioned performance issues. Like i would assume the type of person who would do this sort of thing, over a 2 year period no less, would test extensively and be sure all their i&#x27;s are dotted. Its all kind of surprising given how audacious the attack is.

I&#x27;ve analysed the backdoor myself and it&#x27;s very sophisticated, not poorly made at all.  The performance problem is surprising in this context, but I think next time they won&#x27;t make that mistake.

Case in point: <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=39843930">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=39843930</a>

Which is why a really good backdoor is a one line logic bug somewhere which is fiendishly difficult to trigger.

Unrelated: as a dog&#x2F;pointer lover i really like the term &quot;to bird-dog the problem&quot;. Never heard of it (iam from germany though)

So many malicious actors have been caught because they accidentally created a mild annoyance for someone that went on to bird-dog the problem.

You must mean, &quot;Makes you wonder what more competent actors are doing&quot;

The terrifying part is that this was primarily found because the backdoor was poorly made and causing performance problems.Makes you wonder what more competent actors can do.

They might have burnt the reputation built for this particular pseudonym but what is stopping them from doing it again? They were clearly in it for the long run.

Please don&#x27;t?1. You don&#x27;t actually know what has been done by whom or why. You don&#x27;t know if the author intended all of this, or if their account was compromised. You don&#x27;t know if someone is pretending to be someone else. You don&#x27;t know if this person was being blackmailed, forced against their will, etc. You don&#x27;t really know much of anything, except a backdoor was introduced by somebody.2. Assuming the author did do something maliciously, relying on personal reputation is bad security practice. The majority of successful security attacks come from insiders. You have to trust insiders, because someone has to get work done, and you don&#x27;t know who&#x27;s an insider attacker until they are found out. It&#x27;s therefore a best security practice to limit access, provide audit logs, sign artifacts, etc, so you can trace back where an incursion happened, identify poisoned artifacts, remove them, etc. Just saying &quot;let&#x27;s ostracize Phil and hope this never happens again&quot; doesn&#x27;t work.3. A lot of today&#x27;s famous and important security researchers were, at one time or another, absolute dirtbags who did bad things. Human beings are fallible. But human beings can also grow and change. Nobody wants to listen to reason or compassion when their blood is up, so nobody wants to hear this right now. But that&#x27;s why it needs to be said now. If someone is found guilty beyond a reasonable doubt (that&#x27;s really the important part...), then name and shame, sure, shame can work wonders. But at some point people need to be given another chance.

Name and shame this author. They should never be allowed anywhere near any open projects ever again.

Every Linux box inside AWS, Azure, and GCP and other cloud providers that retains the default admin sudo-able user (e.g., “ec2”) and is running ssh on port 22.I bet they intended for their back door to eventually be merged into the base Amazon Linux image.

Probably less of an individual and more of an exploit to sell.

Distro build hosts and distro package maintainers might not be a bad guess. Depends on whether getting this shipped was the final goal. It might have been just the beginning, part of some bootstrapping.

Also I saw this hans jansen user pushing for merging the 5.6.1 update in debian:
<a href="https:&#x2F;&#x2F;bugs.debian.org&#x2F;cgi-bin&#x2F;bugreport.cgi?bug=1067708" rel="nofollow">https:&#x2F;&#x2F;bugs.debian.org&#x2F;cgi-bin&#x2F;bugreport.cgi?bug=1067708</a>

Also I see this PR:
<a href="https:&#x2F;&#x2F;github.com&#x2F;tukaani-project&#x2F;xz&#x2F;pull&#x2F;64">https:&#x2F;&#x2F;github.com&#x2F;tukaani-project&#x2F;xz&#x2F;pull&#x2F;64</a>

&gt; because the ifunc code was breaking with all sorts of build options and obviously caused many problems with various sanitizersfor example, <a href="https:&#x2F;&#x2F;github.com&#x2F;google&#x2F;oss-fuzz&#x2F;pull&#x2F;10667">https:&#x2F;&#x2F;github.com&#x2F;google&#x2F;oss-fuzz&#x2F;pull&#x2F;10667</a>

1 week ago &quot;Hans Jansen&quot; user &quot;hjansen&quot; was created in debian and opened 8 PRs including the upgrade to 5.6.1 to xz-utilsFrom <a href="https:&#x2F;&#x2F;salsa.debian.org&#x2F;users&#x2F;hjansen&#x2F;activity" rel="nofollow">https:&#x2F;&#x2F;salsa.debian.org&#x2F;users&#x2F;hjansen&#x2F;activity</a>Author: Hans Jansen &lt;hansjansen162@outlook.com&gt;- [Debian Games &#x2F; empire](<a href="https:&#x2F;&#x2F;salsa.debian.org&#x2F;games-team&#x2F;empire" rel="nofollow">https:&#x2F;&#x2F;salsa.debian.org&#x2F;games-team&#x2F;empire</a>): opened merge request &quot;!2 New upstream version 1.17&quot; - March 17, 2024- [Debian Games &#x2F; empire](<a href="https:&#x2F;&#x2F;salsa.debian.org&#x2F;games-team&#x2F;empire" rel="nofollow">https:&#x2F;&#x2F;salsa.debian.org&#x2F;games-team&#x2F;empire</a>): opened merge request &quot;!1 Update to upstream 1.17&quot; - March 17, 2024- [Debian Games &#x2F; libretro &#x2F; libretro-core-info](<a href="https:&#x2F;&#x2F;salsa.debian.org&#x2F;games-team&#x2F;libretro&#x2F;libretro-core-info" rel="nofollow">https:&#x2F;&#x2F;salsa.debian.org&#x2F;games-team&#x2F;libretro&#x2F;libretro-core-i...</a>): opened merge request &quot;!2 New upstream version 1.17.0&quot; - March 17, 2024- [Debian Games &#x2F; libretro &#x2F; libretro-core-info](<a href="https:&#x2F;&#x2F;salsa.debian.org&#x2F;games-team&#x2F;libretro&#x2F;libretro-core-info" rel="nofollow">https:&#x2F;&#x2F;salsa.debian.org&#x2F;games-team&#x2F;libretro&#x2F;libretro-core-i...</a>): opened merge request &quot;!1 Update to upstream 1.17.0&quot; - March 17, 2024- [Debian Games &#x2F; endless-sky](<a href="https:&#x2F;&#x2F;salsa.debian.org&#x2F;games-team&#x2F;endless-sky" rel="nofollow">https:&#x2F;&#x2F;salsa.debian.org&#x2F;games-team&#x2F;endless-sky</a>): opened merge request &quot;!6 Update upstream branch to 0.10.6&quot; - March 17, 2024- [Debian Games &#x2F; endless-sky](<a href="https:&#x2F;&#x2F;salsa.debian.org&#x2F;games-team&#x2F;endless-sky" rel="nofollow">https:&#x2F;&#x2F;salsa.debian.org&#x2F;games-team&#x2F;endless-sky</a>): opened merge request &quot;!5 Update to upstream 0.10.6&quot; - March 17, 2024- [Debian &#x2F; Xz Utils](<a href="https:&#x2F;&#x2F;salsa.debian.org&#x2F;debian&#x2F;xz-utils" rel="nofollow">https:&#x2F;&#x2F;salsa.debian.org&#x2F;debian&#x2F;xz-utils</a>): opened merge request &quot;!1 Update to upstream 5.6.1&quot; - March 17, 2024

Wow, what a big pile of infrastructure for a non-optimization.An internal call via ifunc is not magic — it’s just a call via the GOT or PLT, which boils down to function pointers. An internal call through a hidden visibility function pointer (the right way to do this) is also a function pointer.The even better solution is a plain old if statement, which implements the very very fancy “devirtualization” optimization, and the result will be effectively predicted on most CPUs and is not subject to the whole pile of issue that retpolines are needed to work around.

&gt;Hans Jansen and Jia TanAre they really two people conspiring?Unless proven otherwise, it is safe to assume one is just a pseudonym alias of the other.

Make it two years.Jia Tan getting maintainer access looks like it is almost certainly to be part of the operation. Lasse Colling mentioned multiple times how Jia has helped off-list and to me it seems like Jia befriended Lasse as well (see how Lasse talks about them in 2023).Also the pattern of astroturfing dates back to 2022. See for example this thread where Jia, who has helped at this point for a few weeks, posts a patch, and a &lt;name&gt;&lt;number&gt;@protonmail (jigarkumar17) user pops up and then bumps the thread three times(!) lamenting the slowness of the project and pushing for Jia to get commit access: <a href="https:&#x2F;&#x2F;www.mail-archive.com&#x2F;xz-devel@tukaani.org&#x2F;msg00553.html" rel="nofollow">https:&#x2F;&#x2F;www.mail-archive.com&#x2F;xz-devel@tukaani.org&#x2F;msg00553.h...</a>Naturally, like in the other instances of this happening, this user only appears once on the internet.

Does anybody know anything about Jia Tan? Is it likely just a made up persona? Or is this a well-known person.

It seems like Hans Jansen has also an account on proton.me (hansjansen162@proton.me) with the Outlook address configured as recovery-email.

I think this has been in the making for almost a year. The whole ifunc infrastructure was added in June 2023 by Hans Jansen and Jia Tan. The initial patch is &quot;authored by&quot; Lasse Collin in the git metadata, but the code actually came from Hans Jansen: <a href="https:&#x2F;&#x2F;github.com&#x2F;tukaani-project&#x2F;xz&#x2F;commit&#x2F;ee44863ae88e377a5df10db007ba9bfadde3d314">https:&#x2F;&#x2F;github.com&#x2F;tukaani-project&#x2F;xz&#x2F;commit&#x2F;ee44863ae88e377...</a>&gt; Thanks to Hans Jansen for the original patch.<a href="https:&#x2F;&#x2F;github.com&#x2F;tukaani-project&#x2F;xz&#x2F;pull&#x2F;53">https:&#x2F;&#x2F;github.com&#x2F;tukaani-project&#x2F;xz&#x2F;pull&#x2F;53</a>There were a ton of patches by these two subsequently because the ifunc code was breaking with all sorts of build options and obviously caused many problems with various sanitizers. Subsequently the configure script was modified multiple times to detect the use of sanitizers and abort the build unless either the sanitizer was disabled or the use of ifuncs was disabled. That would&#x27;ve masked the payload in many testing and debugging environments.The hansjans162 Github account was created in 2023 and the only thing it did was add this code to liblzma. The same name later applied to do a NMU at Debian for the vulnerable version. Another &quot;&lt;name&gt;&lt;number&gt;&quot; account (which only appears here, once) then pops up and asks for the vulnerable version to be imported: <a href="https:&#x2F;&#x2F;www.mail-archive.com&#x2F;search?l=debian-bugs-dist@lists.debian.org&amp;q=from:%22krygorin4545%22" rel="nofollow">https:&#x2F;&#x2F;www.mail-archive.com&#x2F;search?l=debian-bugs-dist@lists...</a>

Is it normal that when I try to uninstall xz it is trying to install lzma?

PSA: I just noticed homebrew installed the compromised version on my Mac as a dependency of some other package. You may want to check this to see what version you get:<pre><code> xz --version
</code></pre>
Homebrew has already taken action, a `brew upgrade` will downgrade back to the last known good version.

Yesterday sure was fun wasn&#x27;t it :p  Thanks for all your help&#x2F;working with me on getting this cleaned up in Fedora.

Is that version truly vetted? &quot;Jia Tan&quot; has been the official maintainer since 5.4.3, could have pushed code under any other pseudonym, and controls the signing keys. I would have felt better about reverting farther back, xz hasn&#x27;t had any breaking changes for a long time.

There are suggestions to roll back further

TIL that +really is a canonical string. [0][0]: <a href="https:&#x2F;&#x2F;www.debian.org&#x2F;doc&#x2F;debian-policy&#x2F;ch-controlfields.html#epochs-should-be-used-sparingly" rel="nofollow">https:&#x2F;&#x2F;www.debian.org&#x2F;doc&#x2F;debian-policy&#x2F;ch-controlfields.ht...</a>

Debian have reverted xz-utils (in unstable) to 5.4.5 – actual version string is “5.6.1+really5.4.5-1”. So presumably that version&#x27;s safe; we shall see…

That&#x27;s pure speculation and there are plenty of hints to the contrary.

You can find more examples of that kind of puffer if you go to a website&#x27;s cookie consent pop-up and find the clause after &quot;we use cookies to...&quot;.

I made a library where version 2 is really really much faster than version 1. I&#x27;d want everyone to just move to version 2.

because of it&#x27;s &quot;great new features&quot;&quot;great&quot; for whom? I&#x27;ve seen enough of the industry to immediately feel suspicious when someone uses that sort of phrasing in an attempt to persuade me. It&#x27;s no different from claiming a &quot;better experience&quot; or similar.

Not true, the original author wasn&#x27;t suspended: <a href="https:&#x2F;&#x2F;github.com&#x2F;Larhzu">https:&#x2F;&#x2F;github.com&#x2F;Larhzu</a><a href="https:&#x2F;&#x2F;github.com&#x2F;JiaT75">https:&#x2F;&#x2F;github.com&#x2F;JiaT75</a> was suspended for a moment, but isn&#x27;t anymore?

These shouldn&#x27;t be suspended, and neither should their repositories. People might want to dig through the source code. It&#x27;s okay if they add a warning on the repository, but suspending _everything_ is a stupid thing to do.

Github accounts of both xz maintainers have been suspended.

I agree on principle, but sometimes programmatic generating test data is not so easy.E.g.: I have a specific JPEG committed into a repository because it triggers a specific issue when reading its metadata. It&#x27;s not just _random_ data, but specific bogus data.But yeah, if the test blob is purely random, then you can just commit a seed and generate in during tests.

Interesting that one of the commits commented on update of the test file that it was for better reproducibility for having been generated by a fixed random seed (although how goes unmentioned). For the future, random test data better be generated as part of the build, rather than being committed as opaque blobs...

There probably is. Way more than anyone knows. I bet every major project on github is riddled with state actors.

Imagine if sshd was distributed by PyPI or cargo or npm instead of by a distro.

I’m surprised there isn’t way more of this stuff. The supply chain is so huge and therefore represents so much surface area.

Just for posterity since I can no longer edit: Libera staff has been firm and unrelenting in their position not to disclose anything whatsoever about the account. I obtained the last point on my own. Libera has made it clear they will not budge on this topic, which I applaud and respect. They were not involved whatsoever in ascertaining a VPN was used, and since that fact makes anything else about the connection information moot, there&#x27;s nothing else to say about it.

The alpine patch includes gettext-dev which is likely also exploited as the same authors have been pushing gettext to projects where their changes have been questioned

FWIW, that&#x27;s mingw-w64-xz (cross-compiled xz utils) in AUR, not ming-w64 (which would normally refer to the compiler toolchain itself).

&gt; EDIT: Github has disabled all Tukaani repositories, including downloads from the releases page.Why? Isn&#x27;t it better to freeze them and let as many people as possible analyze the code?

Asking this here too: why isn&#x27;t there an automated A&#x2F;B or diff match for the tarball contents to match the repo, auto-flag with a warning if that happens? Am I missing something here?

I&#x27;ve posted an earlier WHOWAS of jiatan here: <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=39868773">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=39868773</a>

It appears to be an RCE, not a public key bypass: <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=39877312">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=39877312</a>

account is back online 
<a href="https:&#x2F;&#x2F;github.com&#x2F;JiaT75">https:&#x2F;&#x2F;github.com&#x2F;JiaT75</a>

GitHub has suspended @JiaT75&#x27;s account.EDIT: Lasse Collin&#x27;s account @Larhzu has also been suspended.EDIT: Github has disabled all Tukaani repositories, including downloads from the releases page.--EDIT: Just did a bit of poking. xz-embedded was touched by Jia as well and it appears to be used in the linux kernel. I did quick look and it doesn&#x27;t appear Jia touched anything of interest in there. I also checked the previous mirror at the tukaani project website, and nothing was out of place other than lagging a few commits behind:<a href="https:&#x2F;&#x2F;gist.github.com&#x2F;Qix-&#x2F;f1a1b9a933e8847f56103bc14783ab7b" rel="nofollow">https:&#x2F;&#x2F;gist.github.com&#x2F;Qix-&#x2F;f1a1b9a933e8847f56103bc14783ab7...</a>--Here&#x27;s a mailing list message from them ca. 2022.<a href="https:&#x2F;&#x2F;listor.tp-sv.se&#x2F;pipermail&#x2F;tp-sv_listor.tp-sv.se&#x2F;2022-November&#x2F;000802.html" rel="nofollow">https:&#x2F;&#x2F;listor.tp-sv.se&#x2F;pipermail&#x2F;tp-sv_listor.tp-sv.se&#x2F;2022...</a>--MinGW w64 on AUR was last published by Jia on Feb 29:
<a href="https:&#x2F;&#x2F;aur.archlinux.org&#x2F;cgit&#x2F;aur.git&#x2F;log&#x2F;?h=mingw-w64-xz" rel="nofollow">https:&#x2F;&#x2F;aur.archlinux.org&#x2F;cgit&#x2F;aur.git&#x2F;log&#x2F;?h=mingw-w64-xz</a> (found by searching their public key: 22D465F2B4C173803B20C6DE59FCF207FEA7F445)--pacman-static on AUR still lists their public key as a contributor, xz was last updated to 5.4.5 on 17-11-2023: <a href="https:&#x2F;&#x2F;aur.archlinux.org&#x2F;cgit&#x2F;aur.git&#x2F;?h=pacman-static" rel="nofollow">https:&#x2F;&#x2F;aur.archlinux.org&#x2F;cgit&#x2F;aur.git&#x2F;?h=pacman-static</a>EDIT: I&#x27;ve emailed the maintainer to have the key removed.--Alpine was patched as of 6 hours ago.<a href="https:&#x2F;&#x2F;git.alpinelinux.org&#x2F;aports&#x2F;commit&#x2F;?id=982d2c6bcbbb579e85bb27c40be84072ca0b1fd9" rel="nofollow">https:&#x2F;&#x2F;git.alpinelinux.org&#x2F;aports&#x2F;commit&#x2F;?id=982d2c6bcbbb57...</a>--OpenSUSE is still listing Jia&#x27;s public key: <a href="https:&#x2F;&#x2F;sources.suse.com&#x2F;SUSE:SLE-15-SP6:GA&#x2F;xz&#x2F;576e550c49a3667a44b6375fb28916e5&#x2F;xz.keyring" rel="nofollow">https:&#x2F;&#x2F;sources.suse.com&#x2F;SUSE:SLE-15-SP6:GA&#x2F;xz&#x2F;576e550c49a36...</a> (cross-ref with <a href="https:&#x2F;&#x2F;web.archive.org&#x2F;web&#x2F;20240329235153&#x2F;https:&#x2F;&#x2F;tukaani.org&#x2F;misc&#x2F;jia_tan_pubkey.txt" rel="nofollow">https:&#x2F;&#x2F;web.archive.org&#x2F;web&#x2F;20240329235153&#x2F;https:&#x2F;&#x2F;tukaani.o...</a>)EDIT: Spoke with some folks in the package channel on libera, seems to be a non-issue. It is not used as attestation nor an ACL.--Arch appears to still list Jia as an approved publisher, if I&#x27;m understanding this page correctly.<a href="https:&#x2F;&#x2F;gitlab.archlinux.org&#x2F;archlinux&#x2F;packaging&#x2F;packages&#x2F;xz&#x2F;-&#x2F;blob&#x2F;main&#x2F;keys&#x2F;pgp&#x2F;22D465F2B4C173803B20C6DE59FCF207FEA7F445.asc" rel="nofollow">https:&#x2F;&#x2F;gitlab.archlinux.org&#x2F;archlinux&#x2F;packaging&#x2F;packages&#x2F;xz...</a>EDIT: Just sent an email to the last committer to bring it to their attention.EDIT: It&#x27;s been removed.--jiatan&#x27;s Libera info indicates they registered on Dec 12 13:43:12 2022 with no timezone information.<pre><code> -NickServ- Information on jiatan (account jiatan):
 -NickServ- Registered : Dec 12 13:43:12 2022 +0000 (1y 15w 3d ago)
 -NickServ- Last seen : (less than two weeks ago)
 -NickServ- User seen : (less than two weeks ago)
 -NickServ- Flags : HideMail, Private
 -NickServ- jiatan has enabled nick protection
 -NickServ- *** End of Info ***
</code></pre>
&#x2F;whowas expired not too long ago, unfortunately. If anyone has it I&#x27;d love to know.They are not registered on freenode.EDIT: Libera has stated they have not received any requests for information from any agencies as of yet (30th Saturday March 2024 00:39:31 UTC).EDIT: Jia Tan was using a VPN to connect; that&#x27;s all I&#x27;ll be sharing here.

Isn&#x27;t it rather that the attacker can log in to the compromised server by exploiting the RSA code path?

After reading the original post by Andres Freund, <a href="https:&#x2F;&#x2F;www.openwall.com&#x2F;lists&#x2F;oss-security&#x2F;2024&#x2F;03&#x2F;29&#x2F;4" rel="nofollow">https:&#x2F;&#x2F;www.openwall.com&#x2F;lists&#x2F;oss-security&#x2F;2024&#x2F;03&#x2F;29&#x2F;4</a>, his analysis indicates that the RSA_public_decrypt function is being redirected to the malware code. Since RSA_public_decrypt is only used in the context of RSA public key - private key authentication, can we reasonably conclude that the backdoor does not affect username-password authentication?

If it was a compromise it also included the signing keys as the release tarball was modified vs the source available on GitHub.

Do you know if it was actually the commit author, of if their commit access was compromised?

Nice. I worked on a Linux disto when I was a wee lad and all we did was compute a new md5 and ship it.

Good luck with that.
We don&#x27;t even know what country is he from.
Probably from China but even if so. 
Good luck finding him among 1.5 Billions.

Can legal action be taken against the author if it&#x27;s found he maliciously added the backdoor?

his account is active again on github <a href="https:&#x2F;&#x2F;github.com&#x2F;JiaT75">https:&#x2F;&#x2F;github.com&#x2F;JiaT75</a>

It is not good to take into consideration something with any unreadable text instead of the open text of the programme. It should be excluded.

Very annoying - the apparent author of the backdoor was in communication with me over several weeks trying to get xz 5.6.x added to Fedora 40 &amp; 41 because of it&#x27;s &quot;great new features&quot;. We even worked with him to fix the valgrind issue (which it turns out now was caused by the backdoor he had added). We had to race last night to fix the problem after an inadvertent break of the embargo.He has been part of the xz project for 2 years, adding all sorts of binary test files, and to be honest with this level of sophistication I would be suspicious of even older versions of xz until proven otherwise.

&gt; running around salsa.debian.org pushing for more updates in other projects as wellThis is quite common in most (all?) distributions. People are going through lists of outdated packages, updating them, testing them, and pushing them.

And now we see why I don&#x27;t trust anons, aliases, or anime characters to make contributions.My GitHub says exactly who I am!

&gt;that&#x27;s a lot of anonymous accountsJust FYI, krygorin4545@proton.me (the latest message before the upload) was created Tue Mar 26 18:30:02 UTC 2024, about an hour earlier than the message was posted.Proton generates PGP key upon creating the account, with the real datetime of the key (but the key does not include the timezone).

Wow, that&#x27;s a lot of anonymous accounts adding comments there urging for a fast merge!And this &quot;Hans Jansen&quot; guy is apparently running around salsa.debian.org pushing for more updates in other projects as well: <a href="https:&#x2F;&#x2F;salsa.debian.org&#x2F;users&#x2F;hjansen&#x2F;activity" rel="nofollow">https:&#x2F;&#x2F;salsa.debian.org&#x2F;users&#x2F;hjansen&#x2F;activity</a>

See comments about &quot;Hans Janson&quot; upthread, he appeared to collaborate on the exploit in other ways as well.

Hans Gruber would have Been a much more stylish choice…

That name jumped out at me, Hans Jansen is the name Dominic Monaghan used when posing as a German interviewer with Elijah Woods. Not that it can&#x27;t be a real person<a href="https:&#x2F;&#x2F;youtu.be&#x2F;IfhMILe8C84" rel="nofollow">https:&#x2F;&#x2F;youtu.be&#x2F;IfhMILe8C84</a>

I get the feeling that a number of the comments are all the same person &#x2F; group.

For anyone else feeling some deja vu about ifunc &#x2F; Valgrind errors, this Red Hat issue [1] was previously linked from HN 12 days ago [2].[1]: <a href="https:&#x2F;&#x2F;bugzilla.redhat.com&#x2F;show_bug.cgi?id=2267598" rel="nofollow">https:&#x2F;&#x2F;bugzilla.redhat.com&#x2F;show_bug.cgi?id=2267598</a>[2]: <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=39733185">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=39733185</a>

The discussion to upload it to Debian is interesting on its own 
<a href="https:&#x2F;&#x2F;bugs.debian.org&#x2F;cgi-bin&#x2F;bugreport.cgi?bug=1067708" rel="nofollow">https:&#x2F;&#x2F;bugs.debian.org&#x2F;cgi-bin&#x2F;bugreport.cgi?bug=1067708</a>

It&#x27;s not ironic, this change is really sinister IMO. They want you to waste more time after you&#x27;ve submitted the security report and maximize the amount of back and forth. Basically the hope is that they&#x27;d be able to pester you with requests for more info&#x2F;details in order to &quot;resolve the issue&quot; which would give them more time to exploit their targets.

Quite ironic: The most recent commit in the git repo is &quot;Simplify SECURITY.md&quot;, committed by the same Github account which added the backdoor.<a href="https:&#x2F;&#x2F;github.com&#x2F;tukaani-project&#x2F;xz&#x2F;commit&#x2F;af071ef7702debef4f1d324616a0137a5001c14c">https:&#x2F;&#x2F;github.com&#x2F;tukaani-project&#x2F;xz&#x2F;commit&#x2F;af071ef7702debe...</a>

Backdoor in upstream xz/liblzma leading to SSH server compromise