Any post and community could be accessed through a theoretically limitless amount of instances, which also means a theoretically limitless amount of URLs.

Will this hinder Lemmy from ever coming into the mainstream? If I type any topic in Google, I will get a reddit thread that deals with that. Can something like that ever happen for Lemmy?

  • authed@lemmy.ml
    link
    fedilink
    English
    arrow-up
    8
    ·
    1 year ago

    It’s probably the search engine that is unfriendly to Lemmy and others.

  • Send_me_nude_girls@feddit.de
    link
    fedilink
    English
    arrow-up
    4
    ·
    1 year ago

    This is in fact my biggest worry of Lemmy’s future. People need to be able to search for stuff and I currently don’t see how.

    • marsara9@lemmy.world
      link
      fedilink
      English
      arrow-up
      7
      ·
      1 year ago

      I’m doing tests in the next couple days. But I’m trying to build a search engine specifically for Lemmy.

      • It should in theory work similar-ish to Google / Bing.
      • You can filter by instance, community or author.
      • it only indexes Lemmy posts and it won’t keep duplicates.
      • It’ll also open any link you find in your instance.
      • You’ll be able to self host it and point it to any instance you want as well.

      I’m hoping I can open it to the public in a week or so.

      • ShittyKopper [they/them]@lemmy.w.on-t.work
        link
        fedilink
        English
        arrow-up
        3
        ·
        edit-2
        1 year ago

        Please make sure that you’re only indexing Lemmy communities and Kbin magazines (i.e. not microblogs)

        In the wider fediverse, there is an actual expectation of privacy beyond “well it’s technically possible to scrape everything so we may as well give up”. Several people (with reasons of innocent naivete & explicit and blatant malice alike) have tried making fediverse search engines, but all of them are either dead or blocked.

        Lemmy/Kbin is in a unique position where global search does make some sense to have, due to it being a public forum focused on topics (and not people), but there is a very real chance that assholes could use an “unbounded” fediverse search engine to find vulnerable people (quite a few of them specifically fleeing to the fediverse to avoid that kind of problem) and harass them.

        • Muddybulldog@mylemmy.win
          link
          fedilink
          English
          arrow-up
          1
          ·
          1 year ago

          The concept of privacy within today’s Fediverse is asinine and everyone should be pointing that out at every opportunity. Doing otherwise, making believe that some sort of code of conduct or public shame cycle is somehow going to keep people safe, is ridiculous and even more dangerous than a public search engine. By not talking about, very loudly, just how trivial it is to gather this data and how impossible it is to remove it we’re sticking our heads in the sand and there will be people who suffer as a result.

      • peereboominc@lemm.ee
        link
        fedilink
        English
        arrow-up
        0
        ·
        1 year ago

        Cool! How does it technically work? Does it fetch all titles (and maybe the body and comments) via the api from each instance or do you set up your own private instance and tap into the instance database?

        • marsara9@lemmy.world
          link
          fedilink
          English
          arrow-up
          0
          ·
          edit-2
          1 year ago

          I’m using the public API to grab every post / comment and then I essentially replace the content with only the unique words. Then when you go to search it just looks for any post or comment, in my database, that has the words you typed in. Finally I sort based on the number of upvotes.

          Right now it only craws a specific instance that you point it to. But as long as that instance is federated it /should/ get everything. But eventually I plan on using that instance’s list of federated instances to scan everything and lighten the load on any one particular instance.

          Edit: I thought about tapping into the existing database but the existing database is more geared towards serving content but not necessarily searching. The database that I’m building you can search but I drop so much of the original data that using it for content is worthless.

            • marsara9@lemmy.world
              link
              fedilink
              arrow-up
              1
              ·
              1 year ago

              HTML + JavaScript frontend. Rust backend with a postgres database.

              It’ll be open sourced once I can get the MVP ready.

    • person@fenbushi.site
      link
      fedilink
      English
      arrow-up
      1
      ·
      1 year ago

      I think you’re right. Looking at the html source for this page I don’t see a canonical tag, though. Maybe they haven’t added it yet? Or I missed it.

      • marsara9@lemmy.world
        link
        fedilink
        English
        arrow-up
        0
        ·
        1 year ago

        Would the canonical tag make any sense for Lemmy? The problem is, if you search for something your preferred site / URL is your instance. So the canonical would be different for every user?

        • person@fenbushi.site
          link
          fedilink
          English
          arrow-up
          1
          ·
          1 year ago

          Yeah, I think that’s one of the user experience issues we’re facing. Setting the canonical as the original server makes the most sense, but that would mean if you find something interesting via a search engine you have to figure out how to get it to show up on your home instance.

          Like for me, since I run my own instance for myself and one other person so far, I have to find interesting communities manually. It’s really annoying. Though, looking at Lemmy v0.18 release notes, a lot of new devs have made contributions and I’m sure more will help in the future. One improvement from yesterday’s release is visiting a remote community on your home server will pull the community rather than returning a 404. I think changes like that are big first steps towards improving this specific aspect of the user experience.

  • fubo@lemmy.world
    link
    fedilink
    English
    arrow-up
    1
    arrow-down
    1
    ·
    1 year ago

    Currently it appears that a non-logged-in user (try an incognito window!) will only see posts on a particular server’s local communities. So a search engine bot crawling multiple Lemmy servers will only see duplicates if they’ve been explicitly crossposted.