Bots are currently scraping the internet for LLM training data at unprecedented rates[1][2][3], driving up costs and destabilizing public-facing websites. I want to talk about how this has been particularly difficult for wikis, and has gotten much worse in the last few months.
  • thatsnothowyoudoit@lemmy.ca
    1·
    5 months ago

    We use NGINX’s 444 response A LOT.

    In coordination with careful rate-limiting, it’s been a dramatic improvement.

    The worst of the bots don’t advertise their User Agent (or worse, attempt to present they’re a normal user making 100s of requests a second) but there’s lots of low hanging fruit.

  • Tiresia@slrpnk.net
    0·
    5 months ago

    On the plus side, this isn’t a problem with AI, this is a problem with AI companies having more investment money than they know what to do with. The moment the hype fades and they don’t want to hemmorage money scraping every wiki on the internet thousands of times per day, this traffic will go back to a far more sane amount.