Social media took a turn in 2022 as the Twitter Co. takeover motivated many to bail; my personal journey up to barely a week ago is here:
blogs.sap.com/2022/11/25/my-mastodon-pack-and-go-journey/ When I noted that user and traffic counts were increasing rapidly, I also viewed the volunteer hosts and moderators as key to having Mastodon succeed. This tale is about network "attacks" over this past weekend, and exemplary response along with community evolution (like being a gangly teen-ager going out into the "real world.).
With the 4.x upgrade, I saw cache delays, mysterious error messages, etc. The instance I'm inhabiting has since stopped new account creation, up-scaled system resources, and limited public response efforts to limited daily hours.
So it might be difficult to distinguish slow response time causes to ongoing tuning (on purpose), or actions by nefarious entities to disrupt operations through exploits. One instance manager noted they went from a handful of users to tens of thousands in just a few weeks.
ONE
Scrolling through different timelines I noticed an alert about troll accounts causing or having caused interruptions to service though a form of escalating connection or transaction attempts. Without knowing how this was crafted or when, the community information sharing helped to clarify the situation.
https://chaos.social/@ruud@mastodon.world/109449235621264631
To all Mastodon-admins:
seems like there's an attack on all instances by troll accounts.
[ Dec 03, 2022, 05:19 · Edited Dec 03, 05:58 ]
Responses to this thread had good information with minimal speculation, and at least one workaround to the perceived denial of service attacks was by filtering (quicker and easier than code changes). I may need to revisit block lists another time as this social link aspect is secondary to rapid software change management but not unconnected.
An immediate result of new overload conditions is sometimes too hasty changes, or incorrect fault analysis. I noticed the result of one "block this site/sites" with a pleading for acting carefully.
/ Mail us at [] with spamming or otherwise malicious instances and we'll take care of them. /
As the Mastodon instance and user base grows, the relative closeness of one operator to another gets farther away. My observation is there are a lot of people dedicated to making this work.
TWO
Code fix ideas showed up within hours. The suspected root cause of the vulnerability was surmised based on the symptoms. Collaborators found patterns in the source sites.
https://chaos.social/@analog_cafe@mas.to/109451039766034001
[Dec 03, 2022, 12:58]
https://github.com/mastodon/mastodon/issues/21977
Any idea to stop activitypub-troll.cf or likewise attacks?
THREE
Ongoing (as I post this) "transports". The code blocks with weaknesses in bounds checking have already been committed to both the 4.x and 3.x trees (if I read the GIT repository correctly).
https://github.com/mastodon/mastodon/pull/22025
Fix unbounded recursion in account discovery
Meanwhile, preventing or mitigating future issues comes back up again.
https://chaos.social/@jerry@infosec.exchange/109452233436890369
I can see a need for a coordination/alerting capability for security threats, vulnerability notices, and so on.
[Dec 03, 2022, 18:02]
This is where open source software shines, and where it can be tempestuous. By sharing fixes in the open to a wide audience, many reviews and suggestions can happen quickly. On the other hand, our nemeses see the holes being plugged and can scheme to move elsewhere or try another approach. We always need to succeed, it's sense, while they can continue to fail but succeed once and "win."
What next?
For me, the first change is I feel I am posting less than I would, in order not to contribute to a tidal wave during a time operators are doing their best to analyze and tune performance. I'll note the most recent publication from the
chaos.social team:
https://leah.is/posts/scaling-the-mastodon/
Having scaling issues arise coincidentally just after one has documented how to scale for the already past reality must be
Sisyphean.
In related discussions, I found a migration tool that displays more context of your prior followers/following that have advertised their migration status. Even better, there is a toggle to hide those already being followed, to avoid wasted effort ("wait, am I following them or not? I thought I was). Minor defect is still showing some whom I've asked asked but not been approved yet.
Past notes:
Another view:
