• 👍Maximum Derek👍@discuss.tchncs.de
    link
    fedilink
    English
    arrow-up
    9
    ·
    7 months ago

    So do DDG and a lot of other search engines. In addition to the time and cost of running a spider and maintaining a database (for little to no technological benefit these days), a lot of server admins will block crawlers that aren’t googlebot or msnbot/bingbot.