Nginx: limit flooding and aggressive sniffing

| 4 Comments

5.00 avg. rating (91% score) - 1 vote

Last week, have been faced on a big sniffing issue on my wiki. The guy wanted to download all my wiki content. In reality I do not really care as it is open, free for read and contribution is welcome. However, the current VM where the blog and wiki are running didn’t support aggressive sniffing that this guy made. That’s why CPUs were at 100% of usage because of PHP requests, PHP-FPM was overloaded (reached my configuration limits).

The only way to temporary solve the issue was to block the IP address (from OVH) with iptables. This guy recurred a little bit slower from another IP (from Orange) a few hours later. I pretty sure this was the same guy as the user agent in the logs was the same in addition of a non usual useragent name:

I finally find a long term solution to solve this problem using an Nginx module called req_limit. You can specify how many requests per seconds per IPs are allowed, if you return a 503 error or simply delay aggressive requests, burst for a virtualhost etc…

I’ve added to my current Nginx documentation (in French) the way to configure this req_limit module. I hope this will help some of you too.

Note for the sniffer guy: you can try now it should be OK 🙂

Author: Deimos

I'm a passionate DevOps. I love transmit my skills and I love working on high availability infrastructures/technologies.

4 Comments

  1. I think you’re using your wiki as a read-only wiki for external public users. That mean you don’t need any kind of authentication, right?
    If that’s the case, why don’t you put some cache in front of it? For example a varnish server (on the same machine) could do the work (You’d need to purge the cookies sent by Mediawiki, otherwise no page would be cached, varnish has this functionnality and does this really easily). Or maybe you can use a service like Cloudflare (free).
    Both solutions would tranform your wiki in a static pages version, which would allow you to have much more connections/s!

  2. You’re right, it’s a read only wiki for public users.

    I already have a Varnish in front of it, however, today I do not have enough RAM to send it all in RAM. More than that, a warm-up Varnish is not a built-in task and I’m not sure this is the best solution…it seams to be more a workaround to avoid that issue than real solution.

    I also took a look at Cloudflare, however I need to pay to get SSL available which is problematic for me as I want SSL for authentication :-(. I totally agree with you that having static pages is the best solution to be as fast as possible.

    I will shortly change the server to a bigger one. I’ll think about warm-up caching for Varnish but not for solving that issue 😉

    • Or you can be even more “dirty”, and do some script that would do some webscraping on all your pages as static local files 🙂 Maybe that’s what the guy was doing? He wanted to help you 🙂

      OKOK you’re right, that’s really dirty

Laisser une réponse