Choices have slowly been running out when it comes to effective search engines. It seems inevitable an open source search engine project independent from big tech will be needed.

Some of my own tricks are:

  • Use the blacklist plugin to block sites from search.
  • Search for forum sites and communities instead of specific queries. (Wikipedia has a list of forums that might be useful)
  • For technical questions favor Q&A websites like stack exchange.
  • YouTube videos often offer better information than results from search engines. (Use search engines instead of YT search)
  • Look for blogs and journals that specialize in the topic you’re searching for.
  • Use boolean search when possible.
  • Self-host and customize your own metadata search engine. Create a graph network linking websites based on subject/topic. You may not be able to query specific questions but you can discover sites that you otherwise can’t in traditonal search. This is a great way to discover hidden gems! (Example: https://internet-map.net/)
  • (Difficult) Self-host and scrape sites across the web in order to create your own query-able database. This would be the most effective way to search the internet and would be completely independent from potential enshittification and censorship. The cost however is quite high both in term of hardware and time. Kiwix offers a way to download websites for offline use. (Ex: Wikipedia, Stack exchange). This is a good starting point to build your own custom search engine.

I would love to hear the tips and tricks you use! I hope this post helps others in more efficiently finding information on the internet!

  • bluemoon@piefed.socialEnglish
    1·
    1 day ago
    • StartPage, Mojeek, SearXNG, YaCy
    • hyperlink surfing “extranets”, as you would WikiMedia WikiPedia InternetArchive FediVerse posts etc.
    • webscrapers like Monolith etc. for offline PIR and just as you say convenience of having it all there

    i look forward to reading what you come up with, because i am still kinda at the theoretical stage with keeping such a knowledgebase.

    edit: i keep thinking a plaintext document of information is way simpler to deal with than webpages. at what point is information posted online preserved in it’s “original” form? just dumping this FediThread into a plaintext file or a folder of plaintext files with names being ‘hierarchy•postID•username’ or something so it is presented self-organized.

    OP is ¤, 1st rank comments are ¤a ¤b ¤c and 2nd rank comments attached to comment ¤a are ¤a-a ¤a-b ¤a-c and 3rd rank comments attached to ¤a-c are ¤a-c-a ¤a-c-b ¤a-c-c so on. this then lists itself in a self-organized way, given all ASCII & unicode characters are provided in order. not just a-Z… because that would limit size of posts to take on.

    ofcourse more difficult and complicated solutions like selfhosting webservers and managing ports and databases exist… not that i grasp the necessity for so many services.