Federation in general seems to be pretty buggy, I’m running my own instance and I can see a ton of failed jobs happening in the logs. Seems like some of the issues are short comings of the ActivityPub protocol
@jeff@federated.fun i don’t know if there is much we can do about that though. I have read through the activity pub spec as I was considering writing a fediverse web app…I won’t disagree, it is glorified json. Unfortunately it’s the “standard” that has been loosely agreed upon. I might have the willpower to write a passion project FOSS fediverse web app but I know I couldn’t remotely begin architecting a new federation protocol as well.
The one that’s bouncing around in the back of my mind is a ActivityPub <-> NNTP gateway. Have a server that federates to things on ActivityPub and then exposes then via NNTP so that Gnus or trn would be able to speak to the server, read messages and post new messages.
That doesn’t seem impossible… I’m not sure what business logic would be needed to make them easily interoperable. Honestly the biggest complaints I’ve read about the fediverse isn’t the UIs available to view content. The issue is the bugginess of federation and the lack of content recommendation algorithms on the platforms.
I’ve been mulling over the idea of a fediverse content crawler to allow instances to mass federate content to their instance…but then like I said you also need a good recommendation algo as well.
We have a ton of dev work going into making new UIs for Lemmy but personally I think what I said above should get some love too.
For Reddit, you’ve got The Algorithm which keeps suggesting things. There’s /r/all and /r/popular which has a bunch of stuff… and sometimes /r/popular has something interesting that isn’t part of the groups you subscribe that you’d want to subscribe to. There’s also people suggesting other groups to crosspost something too - which adds to the discovery for the person with content and everyone reading with a “oh, that sub also has things… another day, another cat sub.”
Twitter had its algorithm - but one of the important things was the hashtags. For example, there is #energytwitter - https://twitter.com/search?q=%23energytwitter which has lots of material about the electric grid. If you’re on twitter, subscribe to that hash tag and that interesting stuff goes past your feed.
However, with the twitter exodus, people formed instances around a hashtag… mastodon.social was “full” and not accepting new accounts, and “taking over” another instance with all that community’s content was impolite… so there’s https://mastodon.energy/explore which is the people who left twitter and went to mastodon to be active there. There’s non energy stuff, but the local hashtags and activity is good.
Now, if you’ve got an account on techhub.social but want the energy twitter experience… you can’t really get it. You’d have to subscribe to everyone on mastodon.energy or browse it in read only mode… and to get a general “question to the experts”, you’d need someone there to boost it rather than just tagging it with #energytwitter. Or you get two accounts… which is one of the things that federation tries to not have. Unfortunately, identity doesn’t have a good federation solution yet so that one identity could log into two different mastodon instances.
The other one is the multiple instances of the same thing. !java@lemmy.ml vs !java@programming.dev - subscribe to both? Just one? Got content - post to both? or just one?
Anyways, this is where we are now. ActivityPub isn’t awful. Mastodon instances seem alright. Lemmy seems a bit “by itself” with poor federation to others. I can subscribe to events on https://events.nixnet.services in Mastodon without any seeming problem… or things on Wordpress.
I believe Lemmy needs to work on its federation. It’s ok as a reddit clone, but without a really good reason to be federated other than to be federated. Being implemented as a reddit clone, it also brought with it the social problems of Reddit without too much consideration about the design of social software and trying to correct those problems. A Group Is Its Own Worst Enemy is a good read - https://gwern.net/doc/technology/2005-shirky-agroupisitsownworstenemy.pdf - all the things that Reddit did wrong (and right) are attempting to be repeated here.
Wow I appreciate the lengthy response! So where do we go from here? Do we need to add new features that make content more discoverable? Do we need some great sorting algorithm here? Do we need a 3rd party tool that scrapes the fediverse and tells us what we might like?
I’m trying to answer a bunch of questions like this. I’ve been messing with ActivityPub in my spare time to see what my implementation of it would look like. I think I have some ideas that could solve some of the issues we are seeing pop up. Honestly I’d love to have some people help me brainstorm features and architect out a system.
At the end of the day… I’m on the fediverse because I don’t like Facebook, twitter, instagram, and (now) Reddit. It’s not just the people and the content on those platforms I didn’t enjoy, it’s also how they wanted you to interact with the content. I’m not looking to build a platform clone, I’m looking to build something that can fully utilize ActivityPub and can provide a feature rich experience to the user.
Maybe I have too lofty of goals…idk, let me know!
I believe this is a shortcoming of the fediverse and applications being built as (moderately siloed) web applications.
There’s content out there, but reading it requires being on that web application. The interoperability between the servers outside of their domain (e.g. Lemmy - Mastodon) gets clunky.
This goes back to the “how I’d like to see the fediverse”. A content generic front end that federates with other systems.
Look at email for an example - you don’t need to use gmail’s web app to use email and once there was a rich selection of email clients. Now its Apple Mail or Exchange/Outlook for the most part.
The only problem with that application is that while it is likely quite good for consuming content, it doesn’t have anything for pushing content to something on the fediverse. Furthermore, because much of the fediverse activity happens either in the subscribed content sources or in local it returns to the problem of “discovery of someone is hard”.
What I think I want is a myspace / geo cities old school thing where someone could just push activity pub content to it, have content pushed to it (permissioning the pushing of content is a very hard problem) that other people could also consume (another hard permissions problem) and just have it be there.
Push a post “Hey, I did this thing” and other people could comment on it (pushes the comment back to your node for others to be able to consume also from your node - yes, I said permissioning was hard). And then also have aggregators of content. An aggregator of question type content from everywhere that people could browse and push comments back to the aggregator that would then push the response back to the original node.
… but that’s all with a 1970s ideal of the net with a bit of early web and NNTP mixed together. The problems that they didn’t solve are difficult ones that aren’t entirely technically solvable. Spam is still a problem on NNTP and is also an issue on email. The quality of questions on Yahoo Answers where anyone could post were laughably poor (Stack Exchange does it better… at the cost of its a centralized system that manages what you see and people complain about their questions being closed or deleted or banned from one of the sites).
And this touches on the core problem of the fediverse that I’m not sure anyone has sat down and tackled. It’s the “how do you make social software” (if you haven’t read the pdf about the group by Shirky - I strongly suggest that). Lemmy is a a clone of Reddit, that will have Reddit’s social problems some day (if they aren’t here already). Mastodon has Twitter’s problems - though it fights them with a much more active moderation team that shunts off the undesirables to their own instances that no one else talks to (see also Lemmygrad and exploding-heads for Lemmy equivalent sites).
All this boils down to a “I really don’t have a solution.” I mostly “live” in the cosy web of https://maggieappleton.com/cozy-web - moderated / restricted slack and discord sites. I’d like to see more digital gardens, and I have hope that ActivityPub will enable that, though that hasn’t been the case yet. There aren’t enough tools to make Lemmy or Mastodon part of the Cozy Web (small Mastodon instances that are largely single topic focused may show up here) and I worry that its just time before adtech and trolls are able to make inroads into Lemmy. Lemmy instances with the most local communities that are only loosely affiliated with the reason for the instance are likely (crystal ball gazing here) most susceptible to it as they also have the highest costs of hosting and moderation commitment time.
@lohrun @snoopa lemmy’s federation is buggy af and is causing excessive congestion from failed jobs
Federation in general seems to be pretty buggy, I’m running my own instance and I can see a ton of failed jobs happening in the logs. Seems like some of the issues are short comings of the ActivityPub protocol
@lohrun they are shortcomings in lemmy’s flavor of apub. activity pub is more akin to json than it is an actual social networking protocol.
@jeff@federated.fun i don’t know if there is much we can do about that though. I have read through the activity pub spec as I was considering writing a fediverse web app…I won’t disagree, it is glorified json. Unfortunately it’s the “standard” that has been loosely agreed upon. I might have the willpower to write a passion project FOSS fediverse web app but I know I couldn’t remotely begin architecting a new federation protocol as well.
The one that’s bouncing around in the back of my mind is a ActivityPub <-> NNTP gateway. Have a server that federates to things on ActivityPub and then exposes then via NNTP so that Gnus or trn would be able to speak to the server, read messages and post new messages.
That doesn’t seem impossible… I’m not sure what business logic would be needed to make them easily interoperable. Honestly the biggest complaints I’ve read about the fediverse isn’t the UIs available to view content. The issue is the bugginess of federation and the lack of content recommendation algorithms on the platforms.
I’ve been mulling over the idea of a fediverse content crawler to allow instances to mass federate content to their instance…but then like I said you also need a good recommendation algo as well.
We have a ton of dev work going into making new UIs for Lemmy but personally I think what I said above should get some love too.
Discovery is one of the great problems of the fediverse.
Old school, nntp - you got a list of all the groups and grep’ed for whatever was interesting. https://www.eternal-september.org/groups.php?hierarchy=alt - anything interesting? Subscribe. Discovery wasn’t too hard.
For Reddit, you’ve got The Algorithm which keeps suggesting things. There’s /r/all and /r/popular which has a bunch of stuff… and sometimes /r/popular has something interesting that isn’t part of the groups you subscribe that you’d want to subscribe to. There’s also people suggesting other groups to crosspost something too - which adds to the discovery for the person with content and everyone reading with a “oh, that sub also has things… another day, another cat sub.”
Twitter had its algorithm - but one of the important things was the hashtags. For example, there is #energytwitter - https://twitter.com/search?q=%23energytwitter which has lots of material about the electric grid. If you’re on twitter, subscribe to that hash tag and that interesting stuff goes past your feed.
However, with the twitter exodus, people formed instances around a hashtag… mastodon.social was “full” and not accepting new accounts, and “taking over” another instance with all that community’s content was impolite… so there’s https://mastodon.energy/explore which is the people who left twitter and went to mastodon to be active there. There’s non energy stuff, but the local hashtags and activity is good.
Now, if you’ve got an account on techhub.social but want the energy twitter experience… you can’t really get it. You’d have to subscribe to everyone on mastodon.energy or browse it in read only mode… and to get a general “question to the experts”, you’d need someone there to boost it rather than just tagging it with #energytwitter. Or you get two accounts… which is one of the things that federation tries to not have. Unfortunately, identity doesn’t have a good federation solution yet so that one identity could log into two different mastodon instances.
And so - the great federation discovery problem. And I’m not referring to !startrek@startrek.website
The other one is the multiple instances of the same thing. !java@lemmy.ml vs !java@programming.dev - subscribe to both? Just one? Got content - post to both? or just one?
Anyways, this is where we are now. ActivityPub isn’t awful. Mastodon instances seem alright. Lemmy seems a bit “by itself” with poor federation to others. I can subscribe to events on https://events.nixnet.services in Mastodon without any seeming problem… or things on Wordpress.
I believe Lemmy needs to work on its federation. It’s ok as a reddit clone, but without a really good reason to be federated other than to be federated. Being implemented as a reddit clone, it also brought with it the social problems of Reddit without too much consideration about the design of social software and trying to correct those problems. A Group Is Its Own Worst Enemy is a good read - https://gwern.net/doc/technology/2005-shirky-agroupisitsownworstenemy.pdf - all the things that Reddit did wrong (and right) are attempting to be repeated here.
Wow I appreciate the lengthy response! So where do we go from here? Do we need to add new features that make content more discoverable? Do we need some great sorting algorithm here? Do we need a 3rd party tool that scrapes the fediverse and tells us what we might like? I’m trying to answer a bunch of questions like this. I’ve been messing with ActivityPub in my spare time to see what my implementation of it would look like. I think I have some ideas that could solve some of the issues we are seeing pop up. Honestly I’d love to have some people help me brainstorm features and architect out a system. At the end of the day… I’m on the fediverse because I don’t like Facebook, twitter, instagram, and (now) Reddit. It’s not just the people and the content on those platforms I didn’t enjoy, it’s also how they wanted you to interact with the content. I’m not looking to build a platform clone, I’m looking to build something that can fully utilize ActivityPub and can provide a feature rich experience to the user. Maybe I have too lofty of goals…idk, let me know!
I believe this is a shortcoming of the fediverse and applications being built as (moderately siloed) web applications.
There’s content out there, but reading it requires being on that web application. The interoperability between the servers outside of their domain (e.g. Lemmy - Mastodon) gets clunky.
This goes back to the “how I’d like to see the fediverse”. A content generic front end that federates with other systems.
Look at email for an example - you don’t need to use gmail’s web app to use email and once there was a rich selection of email clients. Now its Apple Mail or Exchange/Outlook for the most part.
There was a link the other day to https://github.com/BentonEdmondson/servitor // https://programming.dev/post/577464 which is closer to that ideal of a fediverse reader. It consumes data via ActivityPub rather than having the person use the front end of the website.
The only problem with that application is that while it is likely quite good for consuming content, it doesn’t have anything for pushing content to something on the fediverse. Furthermore, because much of the fediverse activity happens either in the subscribed content sources or in local it returns to the problem of “discovery of someone is hard”.
What I think I want is a myspace / geo cities old school thing where someone could just push activity pub content to it, have content pushed to it (permissioning the pushing of content is a very hard problem) that other people could also consume (another hard permissions problem) and just have it be there.
Push a post “Hey, I did this thing” and other people could comment on it (pushes the comment back to your node for others to be able to consume also from your node - yes, I said permissioning was hard). And then also have aggregators of content. An aggregator of question type content from everywhere that people could browse and push comments back to the aggregator that would then push the response back to the original node.
… but that’s all with a 1970s ideal of the net with a bit of early web and NNTP mixed together. The problems that they didn’t solve are difficult ones that aren’t entirely technically solvable. Spam is still a problem on NNTP and is also an issue on email. The quality of questions on Yahoo Answers where anyone could post were laughably poor (Stack Exchange does it better… at the cost of its a centralized system that manages what you see and people complain about their questions being closed or deleted or banned from one of the sites).
And this touches on the core problem of the fediverse that I’m not sure anyone has sat down and tackled. It’s the “how do you make social software” (if you haven’t read the pdf about the group by Shirky - I strongly suggest that). Lemmy is a a clone of Reddit, that will have Reddit’s social problems some day (if they aren’t here already). Mastodon has Twitter’s problems - though it fights them with a much more active moderation team that shunts off the undesirables to their own instances that no one else talks to (see also Lemmygrad and exploding-heads for Lemmy equivalent sites).
All this boils down to a “I really don’t have a solution.” I mostly “live” in the cosy web of https://maggieappleton.com/cozy-web - moderated / restricted slack and discord sites. I’d like to see more digital gardens, and I have hope that ActivityPub will enable that, though that hasn’t been the case yet. There aren’t enough tools to make Lemmy or Mastodon part of the Cozy Web (small Mastodon instances that are largely single topic focused may show up here) and I worry that its just time before adtech and trolls are able to make inroads into Lemmy. Lemmy instances with the most local communities that are only loosely affiliated with the reason for the instance are likely (crystal ball gazing here) most susceptible to it as they also have the highest costs of hosting and moderation commitment time.