How do we install Fail2Ban?

[martian101]

Hi, everyone.

I wanted to keep my site secure as much as possible and I recently read about Fail2Ban, which helps in blocking IP addresses with excessive 404s, as far as what I've read (correct me if I'm wrong).

So, my question is, anyone here who has a guide/manual/video in installing and using Fail2Ban correctly on FutureQuest?

Although I've read some things on the internet, but I really can't understand everything (newbie here).

Any help would be much appreciated.

Thanks!

[Terra]

FAil2Ban is not going to work for you as to block 404's would require that you block the offenders at the firewall from even reaching your site... As you can surmise, this is a privileged operation that site owners do not have access to perform... I have pondered writing a custom site owner firewall manager, however doing it securely becomes really tricky and the risk of a mistake somewhere could accidentally block a whole server... At this point, the reward vs risk is just not enough to justify writing an unprivileged conduit into the kernel netfilter subsystems...

But alas, I just had an idea that might work... If you were to specify your criteria as to what an 'excessive 404' is, I don't think it would be too hard for us to automate doing the scan and adding the firewall rule for you... I'd call it something like FutureQuest Denier:4XX custom subsystem...

My initial thoughts are that you would create the following file:
xdom/FQdenier.yaml that contains:

FQdenier:
  email: (email address you want notices sent to, and it must be one of your domain's email addresses)
  status:
  - code: (HTTP status code 4XX like 404 or 403)
    maxretry: 3 (Number of matches (i.e. value of the counter) which triggers ban action on the IP.)
    findtime: 600 (The counter is set to zero if no match is found within "findtime" seconds.)
    bantime: 600 (Duration (in seconds) for IP to be banned for. Negative number for "permanent" ban.)
  - code: ...
     ...

Make note that spacing is important with YAML files

You can add more than one '- code:' list item if you have multiple 4XX status codes you want evaluated...

Sample YAML file contents:

FQdenier:
  email: "user@domain.com"
  status:
  - code: 404
    maxretry: 3
    findtime: 600
    bantime: 600

I may also look for a FQdenier.json file, if you are more comfortable with that:

{
  "FQdenier": {
    "email": "user@domain.com",
    "status": [
      {
        "code": 404,
        "maxretry": 3,
        "findtime": 600,
        "bantime": 600
      }
    ]
  }
}

At this point, I'm just roughing out the overall design, and would be building this new FutureQuest subsystem for scale... It would be a stepping stone to getting it added to the CNC proper... However in the interim I would accept site owner written FQdenier.yaml or .json files... Since this is being written from scratch, it could be a few weeks before the first Alpha is available for testing... However since I can probably borrow pieces from the custom system I wrote, it may turn out to be a Lego adventure by using borrowed (production ready) pieces already written...

For periodicity, I'd most likely either:

• Run the scans every 5 minutes
• Setup an inotify watcher on your log and handle in realtime

To keep it K.I.S.S., I'd most likely start with the first option, and leave hooks to wire in an inotify system later which is a bit more complicated...

Let me know what you think about the above... If others would like this as well, please chime in, as a lot of our development is site owner driven...

[chrisheng]

@Terra, you may be overthinking this. You're a technically-capable person, and you assume that everyone knows what he/she is talking about. But martian101 says he's a newbie.

Also, if you just allow anyone to add rules to your firewall, won't those rules affect everybody on FQ too? How if those rules are spurious and block valid traffic for other sites?

@martian101: unless you really know what you are doing, you'll probably just end up blocking Google, which will be a death-knell for your site, since their crawlers tend to pick up all URLs (broken or not) and try to spider them, causing 404s.

[Edit] @martian101: Re-reading your message again, I realize what you're trying to do. It's not so much to block 404s, but to block IP addresses that are probing your site for possible vulnerable scripts (like old WordPress versions, etc), and generating 404s because those scripts are not installed. If so, be careful of blindly blocking such addresses. How do you know that they are not just computers that have been hacked and enslaved to be part of a botnet or something? Blocking such IPs may result in your blocking legitimate ISPs.

[Terra]

@Terra, you may be overthinking this. You're a technically-capable person, and you assume that everyone knows what he/she is talking about. But martian101 says he's a newbie.

Correct, which is why I'm trying to find a solution that is integrated into the services we provide with the final goal of making it newbie friendly via CNC point-n-click control... The core has to be designed before a GUI interface can be layered on top...

Also, if you just allow anyone to add rules to your firewall, won't those rules affect everybody on FQ too? How if those rules are spurious and block valid traffic for other sites?

Because the IPs to be firewalled would be scoped to the domain's dedicated IP address therefore it would not affect any other domain... The rules would also be classified in a 3x2 layer table design, where each XDOM's rulesets would be isolated from one another... This will help with making the administration of it much easier and also prevent any cross pollination...

In short: src src_port dst dst_port, and we would control the ingress 'dst' where the client specified Denier criteria would stipulate the src IP...

On the topic of Google, later it could be expanded to accept a client provided whitelisted IP or CIDR range of IPs that the FutureQuest Denier:4XX would filter it through first... On second thought, it wouldn't even need to filter it, using the '3x2' design, I'd have a sideband short-circuiting RETURN... e.g. CORE ==> LAYERED ==> (isolation) XDOM (grouped) ==> XDOM:DENIER:WHITELIST (short-circuiting) ==> XDOM:DENIER:BLACKLIST

FQdenier:
  ...
  whitelist:
  - ip: 3.4.5.6
  - ip: 1.2.3.0/24
  ...

Given the above design, it is forward evolution design where I could easily stack things into those structures in the future for more fine grained and non-privileged firewall control...

Chris, these are just rough sketches of a hierarchical+tiered design (that can be made low overhead with a proper hashed design), but wanted to show you that I had thought of your concern and designed it in at the ground floor... Of course implementation is a whole other can of worms... :)

[chrisheng]

Originally Posted by Terra

Chris, these are just rough sketches of a hierarchical+tiered design (that can be made low overhead with a proper hashed design), but wanted to show you that I had thought of your concern and designed it in at the ground floor... Of course implementation is a whole other can of worms...

Terra, thanks for your reply. It was indeed reassuring.

(And by the way, I didn't mean that "he's a newbie, therefore he needs a GUI". I meant, "he's a newbie, so I'm not sure that he's really asking for something so extensive". That is, he may not have expressed it accurately (since he's new), and therefore you may be wasting your time working on a solution at the firewall level when all he wants is something simple that he himself can solve.)

[Terra]

That is, he may not have expressed it accurately (since he's new), and therefore you may be wasting your time working on a solution at the firewall level when all he wants is something simple that he himself can solve.)

Yes, I had considered that as well... However, I tend to read things literally and at face value... Knowing how Fail2Ban works, I surmised and concluded that he was seeking a way of blocking nuisance 404s, and the only way to do that is to go deeper and block them at the firewall from even reaching the Apache engines... As an aside, there are times when I get annoyed at stupid zombies hammering away after being rejected with a 403, or just too stupid to realize that 404 is not magically going to exist no matter how much the zombie(s) asks for it...

Therefore I did see merit in formulating a solution that could be brought to the site owners (en masse), that now is only available at the administrative level... I guess what you saw earlier was me brain dumping (inline notes) as the whole system was coming together in my head while typing out my initial response...

So in the end, my goal was to do away with the need for Fail2Ban, which wasn't going to work for them (firewalling), and offer a new custom service that would be of value and satisfies their original request...

[martian101]

@Terra: Thanks for taking the time! I really appreciate it so much!

Regarding this:

If you were to specify your criteria as to what an 'excessive 404' is, I don't think it would be too hard for us to automate doing the scan and adding the firewall rule for you

As far as I've read, they say that I should be alarmed when I see suspicious 404s happening on my site, especially those that occur in a short span of time, because they might be an intruder's attempts to find weaknesses on my site via the URL.

I've also read that I should pay attention to unnatural URLs, especially non-existent ones. So, I want to protect my site from these type of events.

I really want to block intruders from accessing my site (front and back) before they get successful in cracking my site's weakness.

What I am trying to implement is something that would "temporarily" or "permanently" block certain IPs that are making these suspicious 404s to my sites.

To keep it K.I.S.S., I'd most likely start with the first option

For the two codes, I would go with this then.

At this point, I'm just roughing out the overall design, and would be building this new FutureQuest subsystem for scale... It would be a stepping stone to getting it added to the CNC proper...

This is going to be really a big help for us once it officially rolls out. We'll be supporting you.

[martian101]

@chrisheng: Thanks for chiming in!

I realize what you're trying to do. It's not so much to block 404s, but to block IP addresses that are probing your site for possible vulnerable scripts (like old WordPress versions, etc), and generating 404s because those scripts are not installed.

Exactly!

If so, be careful of blindly blocking such addresses. How do you know that they are not just computers that have been hacked and enslaved to be part of a botnet or something? Blocking such IPs may result in your blocking legitimate ISPs.

Honestly, I really don't know how to determine between a legitimate and a malicious IP. But if I can at least block them temporarily, that would be fine for me.

[martian101]

@Terra, @chrisheng:

(And by the way, I didn't mean that "he's a newbie, therefore he needs a GUI". I meant, "he's a newbie, so I'm not sure that he's really asking for something so extensive". That is, he may not have expressed it accurately (since he's new), and therefore you may be wasting your time working on a solution at the firewall level when all he wants is something simple that he himself can solve.)

Sorry, it was not clear. Really sorry about that. Hehe.

What I mean about being a newbie is that I am learning everything about building and protecting a website all from scratch. I have not IT background and I'm the only one who's doing all the work for my two sites.

Researching and reading everything on Google, WordPress, and other resources that teach me about building a website has really made me

But I'm enjoying the learning process so far.

And oh, by the way, I was able to find FutureQuest via www.thesitewizard.com. I've learned a lot also from that site. Bookmarked that one.

So far, my stay in FutureQuest has been good. I'm totally impressed with the support.

So, yeah, thanks again for the help! Looking forward to the official FutureQuest Denier!

[martian101]

Sample YAML file contents:
Code:
FQdenier:
email: "user@domain.com"
status:
- code: 404
maxretry: 3
findtime: 600
bantime: 600

I forgot to ask. This is the script, right?

If yes, I need to download YAML file, right (as per Google)?