I have a web site, sfx.thelazy.net that also have a forum. A very basic forum, more as an afterthought really. Also, the registration is really basic. And I have mostly ignored it for the last years.
Now, if you’ve been on the internet for more than five minutes, you can probably guess what happened.
On Friday someone contacted me saying the site was spammed down, and boy he wasn’t kidding. The forum part of the site is completely overrun by spam, and getting a new post approximately every 30 minutes. Multiply 30 minutes with a few years, and you start getting the scope of this infestation.
Now, when I still mucked about with the site I had some simple scripts I ran that listed the latest posts one by one and let me delete the user with one keypress if it was spam. Deleting the user was a nice force multiplier, as that would delete ALL posts by that user too. So by checking one post I could remove multiple posts. It also kept the database clean. Of course, that meant I had to be absolutely sure it was spam before deleting.
That tool was a nice start, but clearly inadequate to remove the hundreds of thousands of spam posts currently on there. I needed something a bit more automated.
My first idea was to look for links, maybe have a whitelist of good domains, but that turned into a fool’s errand. Not only did legit posts have a surprising amount of random links (image sharing sites, various file hosting sites, articles, other forums and so on), but many of the spam posts didn’t even have a link in it.
In some cases, many cases in fact, a thread was started with a bot asking about something. Like for example what was a good cryptocurrency webpage or good casinos in Spain or how to find academic essays online or… And then another bot would helpfully answer. Some posts would also just have a phone number, an email, or just name drop a brand.
So after a while I dropped that approach and started looking for what was available on the web. Spam is an age old problem after all, and there must be many many solutions and services out there. Well, the services I found cost money, and the local solutions I tried had far too many false positives and negatives. Especially the posts asking for help that had log pastes or external links.
No, I needed something that could understand things more, something that could evaluate a post uniquely in the site’s domain. And preferably without having to train a model specifically to my site. Something that could understand a list of rules and evaluate a post against that. Something that understood plain English..
I’ve been playing around with LLM’s since ChatGPT was released, and toying with local LLM’s since llama first got quantized. I know they’re fickle and hard to control, but also surprisingly good at parsing plain text and answering questions about it. I’ve set up some RAG pipelines in some earlier projects, and when they worked it was amazing. When it worked..
I have a server with a Nvidia P40 in the basement that I usually use for my AI experiments. I started with KoboldCPP as the LLM backend, which has an excellent API I can use, and have used on various projects before. I loaded up the Mistral 7b instruct model as a starting point, and started working on connecting my existing manual scripts to that backend.