Follow

For those who run on Matrix.org and wonder why there is no connection:

Matrix announced an emergency maintenance… on Twitter:

twitter.com/matrixdotorg/statu

Sadly @matrix didn't receive the love it deserves and informs the Fediverse.

Anyway, that's why we have a community. We compensate short coming of each other and together make sure the world becomes a better place!

Okay… I just realized that @matrix went down as well. Maybe that's the reason the announcement didn't make it through.

@matrix Turns out that there was a successful compromise of the Matrix infrastructure happening.

Details from Matrix on Twitter: twitter.com/matrixdotorg/statu

You may ask how that could happen, but more important: It didn't stay unnoticed and that's a good sign.

Matrix is coming back up! One of the first things happening was writing a new blog post about the incident which you can find here:

matrix.org/blog/2019/04/11/sec

TL;DR: Some outdated software was discovered and cracked by an attack which then had access to various data points.

Important: Change your password ASAP (including NickServ when you used the IRC bridges)

Hint: The homeserver is not back up yet.

The homeservers are back up 🎉

It seems like they are missing some pictures right now, I guess those will come back later.

Make sure you change your password (and NickServ passwords) and happy chatting!

See you around 👋

Too early to be happy, seems like the attacker found their way in and is still around on Matrix's infrastructure.

The attack has proven themselves to have shell access on their synapse instance, which is definitely bad. It means that all user accounts are compromised and have to be reset.

twitter.com/matrixdotorg/statu

There will go a lot of efforts into figuring out the details and fixing the vulnerability.

Meanwhile, send some love to the people behind matrix!

Matrix.org just announced they are back once more:

twitter.com/matrixdotorg/statu

Let's hope things stay up as they are. There are definitely some new challenges to tackle, which came up in their issue tracker:

github.com/matrix-org/matrix.o

Let's see if they got really rid of the attacker 🤞

After Matrix has restored its major services, they noticed that the GPG keys used for signing packages where compromised.

The key IDs are:

AD0592FE47F0DF61 (synapse)
E019645248E8F4A1 (Riot/Web)

Please make sure to no longer use those keys.

There are new keys for the official matrix repositories with the key ids:
CF45A512DE2DA058 (synapse)
D7B0B66941D01538 (riot)

Those come along with a new package that are build on fresh infrastructure. No details if they now sign packages offline, yet.

twitter.com/matrixdotorg/statu

twitter.com/RiotChat/status/11

Since Matrix reset all logins recently, you may lost some of your E2EE keys. Those were erased when being forcefully logged out.

Those who used the Key Backup mechanism by Matrix.org can recover quite easily, those who didn't bother to set them up, might have a problem.

In :matrix.org we discussed that today and someone provided a detailed guide on how to recover using BTRFS:

matrix.to/#/!boLskYiwabbCQNNhl

@sheogorath great time to remind everyone there's other public #Matrix homeservers like the ones at hello-matrix.net/public_server and other Riot web clients like riot.privacytools.io 😄

Time to end matrix.org's de facto centralization anyways.

#matrixDown #riot

@sheogorath

@matrix

Yeah and comes to show how you don't want centralisation in federated systems. Or store private communication in unencrypted forms. The amount of data matrix has collected over time is scary. Federation did little to nothing to help that in this case. Matrix.org was the default for everything and it was confusing to use another instance, resulting in that almost every room at at least one matrix user.

@sheogorath @matrix however it happened, I'm glad that it didn't have anything to do with the security of the software itself (synapse), but rather the underlying infrastructure.
My guess would be that there was a server configuration issue, but we'll have to wait for the official explanation :)
Lets just hope they'll get everything back on the rails soon.

@vancha
According to their blog update, it was a CVE in Jenkins, their CI server.

So yeah, good news that the compromise was unrelated to Matrix itself. 👍
@sheogorath @matrix

@sheogorath @matrix Thanks for telling us.

Sadly I'm cut off from Freenode now as I'm behind Tor and therefore can neither register a Freenode account nor run my own Matrix server (that anyone without Tor could access)

@sheogorath they should really think about suspending all services until they get the threat actor out of their systems and they know they’re contained and recovered.

The irony of the strap line “An open network for secure, decentralized communication.” Also isn’t lost on me. 🤔

I wish them well; I know how tough this can be.

@sheogorath the guy making github issues is the attacker 😂

@sheogorath by "they noticed", you mean " the attacker told them"

@sheogorath
How do you get rid of these keys and get the new ones?

@Divert Since I guess you use some Debian base system:

apt-key del AD0592FE47F0DF61

or apt-key del E019645248E8F4A1

@sheogorath
Yes, thanks. that is what I did. I am wondering now how to get the correct ones..

@Divert As far as I know there aren't new ones yet. The keys along with the repositories where removed and will be rebuild during the upcoming week.

twitter.com/RiotChat/status/11

Untrusting GPG keys for packages 

@sheogorath @matrix And then they got pwned again after that announcement.
Sign in to participate in the conversation
Sheogorath's Microblog

This instance is the microblog to my blog. You'll probably find more recent content here while finding more elaborated content on the blog. Impressum / Datenschutz / Privacy