For those who run on and wonder why there is no connection:

Matrix announced an emergency maintenance… on Twitter:

Sadly @matrix didn't receive the love it deserves and informs the Fediverse.

Anyway, that's why we have a community. We compensate short coming of each other and together make sure the world becomes a better place!

Okay… I just realized that @matrix went down as well. Maybe that's the reason the announcement didn't make it through.

Show thread

@matrix Turns out that there was a successful compromise of the Matrix infrastructure happening.

Details from Matrix on Twitter:

You may ask how that could happen, but more important: It didn't stay unnoticed and that's a good sign.

Show thread

Matrix is coming back up! One of the first things happening was writing a new blog post about the incident which you can find here:

TL;DR: Some outdated software was discovered and cracked by an attack which then had access to various data points.

Important: Change your password ASAP (including NickServ when you used the IRC bridges)

Hint: The homeserver is not back up yet.

Show thread

The homeservers are back up 🎉

It seems like they are missing some pictures right now, I guess those will come back later.

Make sure you change your password (and NickServ passwords) and happy chatting!

See you around 👋

Show thread

Too early to be happy, seems like the attacker found their way in and is still around on Matrix's infrastructure.

The attack has proven themselves to have shell access on their synapse instance, which is definitely bad. It means that all user accounts are compromised and have to be reset.

There will go a lot of efforts into figuring out the details and fixing the vulnerability.

Meanwhile, send some love to the people behind matrix!

Show thread just announced they are back once more:

Let's hope things stay up as they are. There are definitely some new challenges to tackle, which came up in their issue tracker:

Let's see if they got really rid of the attacker 🤞

Show thread

After Matrix has restored its major services, they noticed that the GPG keys used for signing packages where compromised.

The key IDs are:

AD0592FE47F0DF61 (synapse)
E019645248E8F4A1 (Riot/Web)

Please make sure to no longer use those keys.

Show thread

There are new keys for the official matrix repositories with the key ids:
CF45A512DE2DA058 (synapse)
D7B0B66941D01538 (riot)

Those come along with a new package that are build on fresh infrastructure. No details if they now sign packages offline, yet.

Show thread

Since Matrix reset all logins recently, you may lost some of your E2EE keys. Those were erased when being forcefully logged out.

Those who used the Key Backup mechanism by can recover quite easily, those who didn't bother to set them up, might have a problem.

In we discussed that today and someone provided a detailed guide on how to recover using BTRFS:!boLskYiwabbCQNNhl

Show thread

@sheogorath great time to remind everyone there's other public #Matrix homeservers like the ones at and other Riot web clients like 😄

Time to end's de facto centralization anyways.

#matrixDown #riot



Yeah and comes to show how you don't want centralisation in federated systems. Or store private communication in unencrypted forms. The amount of data matrix has collected over time is scary. Federation did little to nothing to help that in this case. was the default for everything and it was confusing to use another instance, resulting in that almost every room at at least one matrix user.

@sheogorath @matrix however it happened, I'm glad that it didn't have anything to do with the security of the software itself (synapse), but rather the underlying infrastructure.
My guess would be that there was a server configuration issue, but we'll have to wait for the official explanation :)
Lets just hope they'll get everything back on the rails soon.

According to their blog update, it was a CVE in Jenkins, their CI server.

So yeah, good news that the compromise was unrelated to Matrix itself. 👍
@sheogorath @matrix

@sheogorath @matrix Thanks for telling us.

Sadly I'm cut off from Freenode now as I'm behind Tor and therefore can neither register a Freenode account nor run my own Matrix server (that anyone without Tor could access)

@sheogorath they should really think about suspending all services until they get the threat actor out of their systems and they know they’re contained and recovered.

The irony of the strap line “An open network for secure, decentralized communication.” Also isn’t lost on me. 🤔

I wish them well; I know how tough this can be.

@sheogorath the guy making github issues is the attacker 😂

@sheogorath by "they noticed", you mean " the attacker told them"

How do you get rid of these keys and get the new ones?

@Divert Since I guess you use some Debian base system:

apt-key del AD0592FE47F0DF61

or apt-key del E019645248E8F4A1

Yes, thanks. that is what I did. I am wondering now how to get the correct ones..

@Divert As far as I know there aren't new ones yet. The keys along with the repositories where removed and will be rebuild during the upcoming week.

Untrusting GPG keys for packages 

@sheogorath @matrix And then they got pwned again after that announcement.
Sign in to participate in the conversation
Sheogorath's Microblog

The social network of the future: No ads, no corporate surveillance, ethical design, and decentralization! Own your data with Mastodon!