Challenges of Creating a Decentralized, Open Source Twitter
Evan Prodromou portraiture by K. Ryabitsev.
This is not the first time social media giant Twitter has been besieged by would-be open source competitors. As concerns grow about an undue influence that Elon Musk, who is set to buy the social media giant, might have over the world’s unofficial town square, many are now contemplating jumping to a true open source, peer-to-peer social media network such as Mastadon, or perhaps even starting a new one from scratch.
But standing up a federated, open source equivalent free from corporate influence may be more difficult than one might image.
Open source software developer and advocate Evan Prodromou has been down this path before. A few years after Twitter launched, Prodromou fielded an open source, decidedly non-commercial Twitter-like microblogging service, called StatusNet, which, at least for an audience of technically-inclined open source types, gave Twitter a run for its money. More than 8,000 folks (myself included) signed on within 24 hours of its launch on July 2, 2008, and accumulated more than a million notices by that November.
Over time, StatusNet grew into into a commercial service called Identi.ca, and its code base was eventually rewritten, by way of Node.js, into an activity streams engine that can be used to power internal or public-facing social media services.
We spoke with Prodromou, by email, to learn more about the challenges and potential benefits of creating an open source, decentralized competitor to Twitter. What were the technical, and social challenges to running a social media service, especially an open source federated one? And what can we achieve by establishing web standards in this space?
TNS: What inspired you to create StatusNet?
Prodromou: It feels funny from the perspective of 2022, when tech platform monopolies are the norm, but back in the 2000s having competitive web sites that did the same kind of thing was the rule, not the exception. There were literally dozens of Twitter-like “microblogging” sites on the web in 2008 and 2009. I created StatusNet (now GNU Social) to make an easy, open source engine for running your own microblogging site. You can think of it as the WordPress for microblogging.
Was the creation a response in part to the then-recent launch of Twitter? What were your concerns about a commercial service like Twitter?
Definitely Twitter was the genre-defining site at the time, and a lot of the people using StatusNet approached it from that perspective. But Twitter was kind of unique in that it had a really close connection to SMS messaging. A lot of people used SMS as their main way to interact with the service.
I think the main concerns with Twitter at the time, that I had and that many other people had, was that it was not very stable platform. It crashed a lot — the famous “fail whale” — and there were a lot of outages. I think the idea of distributing the workload over the whole internet, rather than one monolithic service, was more of an architectural goal than a problem with the business. After all, there wasn’t even a business model at first. The big problem was that Twitter was a single point of failure.
How difficult was it to create?
Creating a small social network software package isn’t that hard. I think that a developer who could build a TODO app or a blog platform could manage it in a few weeks. The big problem is scaling up to hundreds, thousands, and tens of thousands of users. The data gets big very fast, and solutions that work with slow, small sites just can’t keep up.
A good way to deal with this problem is to have lots of small, slow sites that connect across the internet with open protocols. It’s a lot easier than running one big site!
What was the importance of writing to the OStatus and OpenMicroBlogging specs?
My team at StatusNet had built our interoperability on the open social stack that was in vogue at the time. Protocols like PubSubHubbub (now WebSub), Salmon, Activity Streams, Webfinger and OpenID.
I and my co-authors on the OStatus spec, James Walker, Zach Copley, and Brion Vibber, wanted to make it clear for people who were interoperating with our software which parts of the stack to use. So, OStatus is more of like a package manifest than a protocol; it just says what protocols to use for what. We used that to replace OpenMicroBlogging, which had been a really simple text-only protocol we used for the early versions of StatusNet.
Once you have a public area, there is a central battlefield for people to fight over. Spammers, partisans, griefers and creeps are going to use those features to try to get attention
We were lucky that there were other apps supporting the same protocols in OStatus, so we could interoperate with Google Buzz and Diaspora. That’s the best way to have a robust network; multiple implementers that participate through the standards.
StatusNet built up quite a healthy user base initially. How did this success come about?
People were excited about real-time messaging in 2008. I’d had the service in private beta for a few months, and when Twitter had a major outage, friends there asked me to open up registration to the public. So through word of mouth, while Twitter was down, we got thousands of people coming to the service.
How difficult was StatusNet to maintain, technically and socially?
That depends on what you mean! You can run an instance of GNU Social today quite comfortably on a Raspberry Pi or a small free server from Amazon. We had a massive cloud platform, running tens of thousands of public and private sites, as well as the big public flagship identi.ca. That took a lot of work from our operations team.
Socially, we had a great group of people doing customer and technical support, and we had an amazing community. A lot of our spam moderation, group management and peer support came out of that identi.ca community, which is so great to think about.
What was the biggest challenge in running a social network, technically or socially?
Well, technically it was managing mixed networks of public and private accounts. All your access management gets funky, as well as figuring out how different streams look to different people.
Socially, the hardest part is having a public square either [as] a main public feed, or hashtag feeds, or a popular or recommended posts feed, or public search. I mean, they’re all really hairy problems to deal with technically, too.
But once you have a public area, there is a central battlefield for people to fight over. Spammers, partisans, griefers and creeps are going to use those features to try to get attention they won’t otherwise get through the follow mechanism.
Why did you evolve StatusNet into Identi.ca and pump.io?
By 2013, I’d run out of money. We’d pivoted to focusing on enterprise social networking, like Yammer, and we had a lot of solid customers, but we weren’t making enough revenue to pay for the free service at identi.ca. So, after laying off the team, I bought the investors out and kept the service running. I needed to get the resource usage way, way down from its peak, so I used NodeJS and MongoDB to get about a 20x resource saving over the original LAMP stack. It was a big win!
What is your advice for someone creating/scaling a Twitter-like social media service that they may not initially see?
I’d stay start with Mastodon, and build from there. You’ll benefit from the network effect and great engineering and years of experience with social problems like moderation. They’ve solved the hard problems you don’t even know about yet.
Have you tried Mastadon? What is your impression?
It’s a wonderful platform. I think Gargron [Eugen Rochko] and the other contributors are gifted developers. I’m so happy they took the work we did at the W3C creating the ActivityPub standard and breathed life into it. There are some things I wish they’d done, like implementing the ActivityPub standard client API, but overall I think it’s been a resounding success.
Any advice for Elon Musk?
Not really. I guess my main message would be for the people at Twitter who are hearing about this sale today. I want to say that you’re doing great work at an important platform, and I really appreciate what you’re making and what you’ve made. Best of luck as you move into this next phase.
I’m hopeful that Twitter’s commitment to support the Blue Sky project [Twitter’s own decentralized social network protocol] and protocol will continue. I think with a Twitter that’s open the world is much less fragile than a walled garden.
