Fail2ban

From HerzbubeWiki
Jump to: navigation, search

This page has information on the security monitoring tool fail2ban. The tool's basic mode of operation is to scan log files for failed login attempts and, if excessive attempts are detected, to ban the offending IP address from accessing any network services for a period of time. The ban is pronounced by installing a firewall rule that simply ignores all network traffic from the offending IP address.

The central task of fail2ban is to prevent brute-force attacks by slowing attackers down sufficiently so that they give up because the attack is simply too time-consuming to succeed within a useful timespan. Because fail2ban can be configured with arbitrary hand-crafted rules, it is flexible enough to also react to other things than just login attempts, so other toxic or obnoxious behaviours can also be stopped.


Why use fail2ban?

In the past I have been operating with hand-crafted iptables rules that protect SSH from brute-force attacks. This worked well, but when pelargir began its reincarnation as a Dedicated Server I wanted to try out something new, a network security solution that would be more flexible and thus - hopefully - provide more "powerful" protection. After reviewing a few solutions (I don't recall which ones they were) the clear favourite was fail2ban, because

  • With its rule system it provides a huge amount of flexibility
  • It is actively maintained


At the beginning, my greatest concern was that fail2ban would have a larger impact on my system than I would like, because it requires more resources than my hand-crafted iptables rules to run a dedicated daemon that continuously scans all those log files. This is mitigated by two things:

  • The fact that fail2ban is kind of "self-regulating": If an attacker is starting to fill the log files, fail2ban will soon create a firewall rule that will cut off the attacker, so log file activity will return to normal shortly afterwards.
  • The fear of too much log file polling is unfounded because fail2ban is not polling at all - instead it is notified by the Linux kernel when any of the monitored log files are changing. fail2ban uses python-pyinotify for this, which in turn interfaces with the inotify Linux kernel feature.


Once the resource question is out of the way, failban clearly has many advantages:

  • I don't have to maintain a system start/stop script
  • I don't have to deal with the complexities of iptables rules
  • fail2ban is more flexible than my hand-crafted iptables rules because it can do more than just protect SSH
  • fail2ban is also more flexible because it can do more than just create firewall rules. For instance, it can report evildoers to badips.com.


Some disadvantages that I found:

  • fail2ban does not support IPv6 (yet). This is not a problem, though, because pelargir is not set up for IPv6 anyway.
  • iptable rules are reacting faster to a new threat than any log analysis tool such as fail2ban can hope to do. While this is certainly a valid point, I believe it doesn't weigh strongly because an attacker can't perform too many login attempts in the few seconds (max!) that it takes fail2ban to react. In my specific case this is also mitigated by the fact that I'm exclusively using very strong, randomly generated passwords for all accounts, so a few hundreds of additional password checks won't matter.


References


Debian packages

fail2ban
curl
wget
jq

Notes:

  • curl is required only if you want to report banned IP addresses to badips.com.
  • wget is required only if you want to retrieve the API key from badips.com.
  • jq is required only to pretty-print the JSON output that you get from wget requests to badips.com.


Client/server architecture

fail2ban consists of a server and a client. The server itself knows nothing about the configuration files. Thus, at start-up, the server is in a "default" state where it knows no rules and does nothing.


The client is the command line utility

fail2ban-client


The client can be used to read the configuration, or to query or send commands to the server. After the server has started up, the client is used to configure the server with rules, typically those from the configuration in

/etc/fail2ban


The client communicates with the server over a Unix domain socket. More on how to use the client follows later.


Configuration

Introduction

The fail2ban configuration is located in

/etc/fail2ban

The configuration has two parts

  • A main configuration file that defines global options
  • A jails configuration file that defines the so-called "jails"


Main configuration file

The main configuration file is

/etc/fail2ban/fail2ban.conf

The main configuration file defines basic things like

  • The log file location (fail2ban has its own log file)
  • The log level
  • The location of the socket and the PID file


Jails configuration file

In addition to the main configuration file there is another file that declares the so-called "jails":

/etc/fail2ban/jail.conf


A "jail" is a combination of

  • Which log file to watch
  • What to watch for. This is called a "filter". Filter definitions are located as separate files in the filter.d subfolder.
  • What actions to take if something has been detected. Action definitions are located as separate files in the action.d subfolder.


A new jail can be declared by creating a new jail configuration file in

/etc/fail2ban/jail.d

Jail configurations in the jail.d subfolder are automatically added to the jails that are defined in jail.conf.


Example: The "ssh" jail

As an example, let's take a look at the "ssh" jail. This is the only jail that is enabled when fail2ban is newly installed, all other jails in jail.conf are disabled.

Here's the snippet for the "ssh" jail from jail.conf:

[ssh]
enabled  = true
port     = ssh
filter   = sshd
logpath  = /var/log/auth.log
maxretry = 6

Discussion

  • This jail watches the log file /var/log/auth.log
  • The filter named "sshd" is used to examine the content of the log file. The filter definition is located in a separate file in the filter.d subfolder, i.e. in
/etc/fail2ban/filter.d/sshd.conf
  • The jail takes action if the same offender has been detected for the 6th time within 600 seconds (10 minutes). Not visible in this jail is the time frame of 600 seconds, this is a default value that appears elsewhere in jail.conf. If necessary, the jail could override the default time frame and specify its own value with the "findtime" option. For instance:
findtime = 300
  • Also not visible in the jail declaration is the action that should be taken. Again, a default value that appears elsewhere in jail.conf is used. If necessary, the jail could specify its own action with the "action" option. For now, let's look at the default action defined in jail.conf:
action = %(action_)s
  • The action "action_" is a placeholder. Its value is defined elsewhere in jail.conf like this:
action_ = %(banaction)s[name=%(__name__)s, port="%(port)s", protocol="%(protocol)s", chain="%(chain)s"]
  • Again, the action "banaction" is a placeholder whose value is defined elsewhere in jail.conf as
banaction = iptables-multiport
  • Finally, the action "iptables-multiport" is no longer a placeholder but a "real" action. The details of that are located in the action.d subfolder, i.e. in
/etc/fail2ban/action.d/iptables-multiport.conf


As we have seen above, some sort of placeholder handling can take place. For instance

action = %(action_)s

The construct %(...)s is a construct from Python that is variously called "string formatting operator" or "string interpolation operator". This works similarly as sprintf(). Without going into the (complicated) details, the construct %(action_)s simply acts as a placeholder for the content of the variable that is "action_". Some references:


Customizing the default configuration

If you want to change something about the default configuration of fail2ban you could simply edit the configuration file in question and make your changes. This becomes a problem when you want to install a new version of fail2ban, because then you will have to manually merge your customizations with the default configuration from the new package. For this reason fail2ban offers a better way to make customizations.


For every configuration file you can create another file with the same name, but replacing ".conf" with ".local". For instance

/etc/fail2ban/jail.local

This local file does not replace the original file, it contains settings that override those from the original file.


Important: This mechanism works not only for jail.conf / jails.local, but also for filter configurations in the filter.d subfolder, and for action configurations in the action.d subfolder.


Filters

Filter definitions are located in

/etc/fail2ban/filter.d

This section has no more information because until now I did not have to deal with filter definitions - the default filter definitions have been sufficient for my needs.


Actions

Action definitions are located in

/etc/fail2ban/action.d


An action definition is an .ini file that consists of two parts:

  • An "[Init]" section that defines some initial values
  • A "[Definition]" section that defines the various actions that can be executed


The following actions can appear in the "[Definition]" section

  • actionstart = Action to perform when the jail starts
  • actionstop = Action to perform when the jail ends
  • actionban = Action to perform when a ban occurs
  • actionunban = Action to perform when the ban is lifted
  • actioncheck = Action to perform before any other action; this is supposed to be used to "check if the enironment is still OK" (quote from man jail.conf)


Actions can be parameterized with so-called "tags":

  • A "tag" is simply a placeholder, or a variable
  • A "tag" is delimited by angular brackets. In the following example, <foo> denotes a tag named "foo":
actionban = /path/to/some/tool --parameter1 <foo>
  • Before an action is executed, all of its tags are replaced with their current value.
  • All settings made in the action's "[Init]" section work as tags for that action. So to provide a value for tag "foo" from the above example, an action could specify this in its "[Init]" section:
[Init]
foo = bar
  • In addition to action-specific tags, fail2ban also defines a couple of useful generic/global tags (e.g. <ip>)
  • When you "call" an action from a jail in jail.conf you can override the values that the action sets in its "[Init]" section. For an action called "doIt" that contains the "foo" setting from the examples above, this would look like this:
action = doIt[foo=yoyodyne]


Example: The "badips" action

As an example, let's take a look at the "badips" action. Here is the jail configuration that "calls" the action:

root@pelargir:~# cat /etc/fail2ban/jail.local
[ssh]
action = %(action_)s
         badips[category=ssh]

And here is the actual action definition:

root@pelargir:~# cat /etc/fail2ban/action.d/badips.conf 
[Definition]

actionban = curl --fail  --user-agent "fail2ban v0.8.12" http://www.badips.com/add/<category>/<ip>

[Init]

# Option: category
# Notes.: Values are from the list here: http://www.badips.com/get/categories
category = 

Discussion

  • The "badips" action only defines one of the 5 possible actions, because only that one is necessary to report the IP address to badips.com
  • The action "actionban" uses the tag "<category>"
  • The "<category>" tag is replaced with the value from the "category" setting in the "[Init]" section
  • The "category" setting is overridden with the value "ssh" when the "badips" action is called from the "ssh" jail
  • This example also demonstrates how a part of a jail definition in jail.conf can be overridden by values in jail.local


Log file

fail2ban has its own log file where it logs all bans and unbans:

/var/log/fail2ban.log

This can be useful if you want to have a jail that monitors for repeat offenders.


The following command counts all ban actions in all the log files that are currently present on the system (both compressed and uncompressed), and prints them grouped by IP address and sorted descending by "IP address with most bans":

zcat --force /var/log/fail2ban.log* | grep "fail2ban.actions.*\[ssh\] Ban " | sed -e 's/^.* Ban //' | sort | uniq -c | sort -nr

Example output:

root@pelargir:~# zcat --force /var/log/fail2ban.log* | grep "fail2ban.actions.*\[ssh\] Ban " | sed -e 's/^.* Ban //' | sort | uniq -c | sort -nr
    288 116.31.116.48
     68 116.31.116.49
     67 116.31.116.44
     49 91.224.160.10
     38 116.31.116.45
[...]


Using the client

This command starts the server and configures it:

fail2ban-client start

This command starts clears out the server's current configuration and re-configures it with the values that the client reads from the current configuration files:

fail2ban-client reload

This command shows the status of a specific jail in the server. If no jail is specified the command lists the jails that are configured in the server.

fail2ban-client status <jail-name>

Example output:

root@pelargir:~# fail2ban-client status ssh
Status for the jail: ssh
|- filter
|  |- File list:	/var/log/auth.log 
|  |- Currently failed:	0
|  `- Total failed:	4612
`- action
   |- Currently banned:	0
   |  `- IP list:	
   `- Total banned:	617

This command shows how to send a command to the server. It sets the server's log level.

fail2ban-client set loglevel 1

This command reads the configuration files and dumps them in a format that corresponds to the commands that will be sent to the server if the "start" or "reload" commands are used.

fail2ban-client -d

Example output:

root@pelargir:/etc/fail2ban# fail2ban-client -d
WARNING 'actionstart' not defined in 'Definition'. Using default one: ''
WARNING 'actionstop' not defined in 'Definition'. Using default one: ''
WARNING 'actioncheck' not defined in 'Definition'. Using default one: ''
WARNING 'actionunban' not defined in 'Definition'. Using default one: ''
['set', 'loglevel', 3]
['set', 'logtarget', '/var/log/fail2ban.log']
['add', 'ssh', 'auto']
['set', 'ssh', 'usedns', 'warn']
['set', 'ssh', 'addlogpath', '/var/log/auth.log']
['set', 'ssh', 'maxretry', 6]
['set', 'ssh', 'addignoreip', '127.0.0.1/8']
['set', 'ssh', 'ignorecommand', '']
['set', 'ssh', 'findtime', 600]
['set', 'ssh', 'bantime', 600]
['set', 'ssh', 'addfailregex', '^\\s*(<[^.]+\\.[^.]+>)?\\s*(?:\\S+ )?(?:kernel: \\[ *\\d+\\.\\d+\\] )?(?:@vserver_\\S+ )?(?:(?:\\[\\d+\\])?:\\s+[\\[\\(]?sshd(?:\\(\\S+\\))?[\\]\\)]?:?|[\\[\\(]?sshd(?:\\(\\S+\\))?[\\]\\)]?:?(?:\\[\\d+\\])?:?)?\\s(?:\\[ID \\d+ \\S+\\])?\\s*(?:error: PAM: )?[aA]uthentication (?:failure|error) for .* from <HOST>( via \\S+)?\\s*$']

[...]

Regexes can be tested against a string, or against the content of a file:

fail2ban-regex "line" "failregex"
fail2ban-regex /var/log/auth.log "failregex"

If the regex is in a file:

fail2ban-regex /var/log/auth.log /etc/fail2ban/filter.d/sshd.conf

Manually unban an IP

fail2ban-client set <jail_name> unbanip <ip_address>


Configuration customizations on pelargir

Currently I only have redefined the "ssh" jail to submit banned IP addresses to badips.com. To achieve this I have overriden the ban action for the jail like this:

root@pelargir:~# cat /etc/fail2ban/jail.local
[ssh]
action = %(action_)s
         badips[category=ssh]


badips.com

From the badips.com website

badips.com is a community based IP blacklist service. You can report malicious IPs and you can download blacklists or query our API to find out if a IP is listed.

As shown in the section that lists my fail2ban configuration customizations, it is very easy to add reporting to badips.com to fail2ban. Once you have reported your first IP address, badips.com assigns an API key to you, probably based on your own IP address.


The following command retrieves your API key. Note: The jq tool pretty-prints the JSON output from badips.com. If you haven't got jq on your system you can simply leave it out of the command:

root@pelargir:/etc/fail2ban# wget https://www.badips.com/get/key -qO - | jq '.'
{
  "err": "",
  "suc": "Your Key was already present! To overwrite, see http://www.badips.com/apidoc.",
  "key": "your-api-key"
}


This command prints the URL that you can open in your browser to get your personalized stats and graphs:

wget https://www.badips.com/get/key -qO - | jq '@uri "https://www.badips.com/stats?key=\(.key)"'