How do you account for the duplicate Riker in TNG? Who’s the real one and where did the extra matter come from then to assemble William vs Tom?
(It’s been a long time since I’ve seen that episode so I don’t remember if they covered that but on-screen)
A similar question could be raised for the Rascals episode…
I’m also running Ubuntu as my main machine at home. (I have a Mac and do Android development for my day job).
But at home, I do a lot of website and backend dev.
docker buildxdocker compose up -dProblem is finding the difference between repost bots and bots that are helpful like automod and link redirectors.
Maybe. 2nd idea I’ve got is that if no one is replying after say 24hrs and something like 75-80% of your posts are as such and you have at least 100 such posts, you get added to the list?
Main concern I see about something like this is false positives and how someone real could end up getting blocked.
I definitely want to think on this some more but it might have some legs.
…I wonder if there’s a programmatic way to detect these bots? Some sort of analysis on their posting behavior?
If they’re playing nice they’ll have the bot flag checked in their profile, and then maybe build a list of any bot that creates posts? As most of the “good” bots just reply to comments? Anyway just thinking out loud. But I’m thinking I could easily add a public API to my search engine that just returns a list of “posting bots”…
The target folder may be quite large. You can look at the dependencies for my project but my end binary is only a few MB.
I don’t see anything wrong initially. Can you also test with something like postman? It’ll allow you to control the headers etc… that you send.
You can also compare to what I have here and see if that helps: https://github.com/marsara9/lemmy-search/blob/b6c88355aba49abca52862473650526821ee165a/server/src/api/lemmy/fetcher.rs#L123
Let’s say I just sent a request from my non-existent server with my user id…
Who or what is going to send this request if not some server that implements ActivityPub? This could be a Lemmy or Mastodon or Kbin instance… Or anything else that implements ActivityPub.
…and just every time I wanted to check whether I got replies I would query the other server (which a Lemmy server would do to get notifications about replies or upvotes)
ActivityPub works via pushes. So there’s nothing to query. There HAS to be some server for it to send and store that data.
So you can’t just send data from a domain. There has to be a service running behind that domain name to do something.
Without a server, it’d be like asking “why do I need tires on my car?”. Well it’s not going to go anywhere without them.
Now this could be a private instance with only you as the single user. And it could federated with the rest of the fediverse. But you still have to run some software to do that.
Now in theory someone I guess could come up with a slim version of Lemmy that only has a single user, and you can’t post or comment directly to that instance but again something has to be running on a server behind that domain.
There are APIs that you can use to post and comment, etc… But there has to be an instance at that domain.
That’s how most of the mobile apps work btw, they just send a network request direct to lemmy.world etc… Saying that iopq wants to create a post with this title and this content…
But there’s no need for your own personal domain name in this scenario, you just need an account on the server that you’re trying to post or comment to.
Not sure if that helps.
Thanks for the callout. For anyone using that site, I have a list of bugs I’m hoping to get fixed in the next couple days that should fix a lot of the major issues (broken links, etc…).
I’m not an expert, but I thought the issue was generally that big instances like lemmy.world were getting overloaded on the server side, not that they were enforcing a manufactured rate limit on individual IPs.
From what I can see it’s both. lemmy.world and others are getting overloaded, but there is an inherit built-in rate-limit in the code itself. You can see what those limits are via the api/v3/site. Now in theory if you’re actually getting rate-limited you should be seeing HTTP 429 responses from the server. If the server is just overloaded, you’ll get a 5xx response, the request will just timeout or at best you’ll still get a response but after a significant delay (what most people are seeing).
Also, someone else mentioned that on the fediverse even simple things like an upvote are slower and require more work here than in centralized platforms because they must be sent to all the instances that are indexing that user/community. As I understand, that’s inherent to the fediverse, a bug not a feature, designed for redundancy and resilience.
I don’t want to comment on this too much as I’m not an expert here, but here’s how federation / ActivityPub works from what I understand looking at the code:
Whenever you take any action (or activity) your browser will first send that message to your instance. If your instance then owns the community that message is then propagated out to EVERY linked instance listed here: /instances / api/v3/federated_instances. If your instance doesn’t own the community, that message is forwarded off to the instance that does and they sent it out to EVERYONE on their federated instances list. As you can see this creates A LOT of network traffic.
This posing an interesting problem… the number of ActivityPub messages goes up as the number of instances increase. But at the same time as more and more users join a single instance that require that that instance send more and more traffic to individual user’s browsers as they view and respond to posts. So the problem here is trying to find a good balance. And to top it off, the default behavior of most users is going to be to join the largest instances, making that instance incur more and more traffic to view content.
Again uniformed, but Lemmy seems like it should scale fine. Bigger instances will monetize, driving prospective users to smaller instances, and then rate limiting and server lag won’t be so bad anymore.
Will it though? How would an individual instance monetize? They would have to use donations. If an instance tries to add Ads, users will leave to an instance that doesn’t, making it so that they don’t get any income. They could charge a subscription fee, but again users would just leave and the admins get nothing.
The ideal configuration of the fediverse as I see it, is if we had two types of servers 1) content servers that only hosted communities but didn’t have any real number of users, and 2) user servers that have no communities but most of the users. This way the number of API requests between instances is rather limited. When you end up with a server that has both most of the content and the userbase, the workload of that server appears to grow exponentially instead of linearly as the number of new instances rises.
Lemmy is entirely open source, so you can see what their architecture looks like, etc… here: https://github.com/LemmyNet/lemmy.
Rate limits, as I understand them from the code, should only apply on a per-IP basis. So you should only be seeing rate limit errors if:
/api/v3/site)For now the only search is using the exact text, for the most part.
For example I could find this post by searching for “Is way search post”. But if I rearrange any of the words it won’t work.
I had to do something similar for the search engine I’m working on. What I’m currently testing is indexing the api/v3/post/list endpoint and matching on the actor_id to the internal id. The only major problem with this is that I now need to crawl the entire fediverse in order to support all of the different sites.
HTML + JavaScript frontend. Rust backend with a postgres database.
It’ll be open sourced once I can get the MVP ready.
Would the canonical tag make any sense for Lemmy? The problem is, if you search for something your preferred site / URL is your instance. So the canonical would be different for every user?
I’m using the public API to grab every post / comment and then I essentially replace the content with only the unique words. Then when you go to search it just looks for any post or comment, in my database, that has the words you typed in. Finally I sort based on the number of upvotes.
Right now it only craws a specific instance that you point it to. But as long as that instance is federated it /should/ get everything. But eventually I plan on using that instance’s list of federated instances to scan everything and lighten the load on any one particular instance.
Edit: I thought about tapping into the existing database but the existing database is more geared towards serving content but not necessarily searching. The database that I’m building you can search but I drop so much of the original data that using it for content is worthless.
I’m doing tests in the next couple days. But I’m trying to build a search engine specifically for Lemmy.
I’m hoping I can open it to the public in a week or so.
Fair enough. Sometimes you can’t help but go down these rabbit holes though.