Fail2ban

This page has information on the security monitoring tool fail2ban. The tool's basic mode of operation is to scan log files for failed login attempts and, if excessive attempts are detected, to ban the offending IP address from accessing any network services for a period of time. The ban is pronounced by installing a firewall rule that simply ignores all network traffic from the offending IP address.

The central task of fail2ban is to prevent brute-force attacks by slowing attackers down sufficiently so that they give up because the attack is simply too time-consuming to succeed within a useful timespan. Because fail2ban can be configured with arbitrary hand-crafted rules, it is flexible enough to also react to other things than just login attempts, so other toxic or obnoxious behaviours can also be stopped.

Why use fail2ban?

In the past I have been operating with hand-crafted iptables rules that protect SSH from brute-force attacks. This worked well, but when pelargir began its reincarnation as a Dedicated Server I wanted to try out something new, a network security solution that would be more flexible and thus - hopefully - provide more "powerful" protection. After reviewing a few solutions (I don't recall which ones they were) the clear favourite was fail2ban, because

• With its rule system it provides a huge amount of flexibility
• It is actively maintained

At the beginning, my greatest concern was that fail2ban would have a larger impact on my system than I would like, because it requires more resources than my hand-crafted iptables rules to run a dedicated daemon that continuously scans all those log files. This is mitigated by two things:

• The fact that fail2ban is kind of "self-regulating": If an attacker is starting to fill the log files, fail2ban will soon create a firewall rule that will cut off the attacker, so log file activity will return to normal shortly afterwards.
• The fear of too much log file polling is unfounded because fail2ban is not polling at all - instead it is notified by the Linux kernel when any of the monitored log files are changing. fail2ban uses python-pyinotify for this, which in turn interfaces with the inotify Linux kernel feature.

Once the resource question is out of the way, failban clearly has many advantages:

• I don't have to maintain a system start/stop script
• I don't have to deal with the complexities of iptables rules
• fail2ban is more flexible than my hand-crafted iptables rules because it can do more than just protect SSH
• fail2ban is also more flexible because it can do more than just create firewall rules. For instance, it can report evildoers to badips.com.

Some disadvantages that I found:

• fail2ban does not support IPv6 (yet). This is not a problem, though, because pelargir is not set up for IPv6 anyway.
• iptable rules are reacting faster to a new threat than any log analysis tool such as fail2ban can hope to do. While this is certainly a valid point, I believe it doesn't weigh strongly because an attacker can't perform too many login attempts in the few seconds (max!) that it takes fail2ban to react. In my specific case this is also mitigated by the fact that I'm exclusively using very strong, randomly generated passwords for all accounts, so a few hundreds of additional password checks won't matter.

References

• fail2ban website
• Wikipedia article

Debian packages

fail2ban
curl
wget
jq

Notes:

• curl is required only if you want to report banned IP addresses to badips.com.
• wget is required only if you want to retrieve the API key from badips.com.
• jq is required only to pretty-print the JSON output that you get from wget requests to badips.com.

Client/server architecture

fail2ban consists of a server and a client. The server itself knows nothing about the configuration files. Thus, at start-up, the server is in a "default" state where it knows no rules and does nothing.

The client is the command line utility

fail2ban-client

The client can be used to read the configuration, or to query or send commands to the server. After the server has started up, the client is used to configure the server with rules, typically those from the configuration in

/etc/fail2ban

The client communicates with the server over a Unix domain socket. More on how to use the client follows later.

Configuration

Introduction

The fail2ban configuration is located in

/etc/fail2ban

The configuration has two parts

• A main configuration file that defines global options
• A jails configuration file that defines the so-called "jails"

Main configuration file

The main configuration file is

/etc/fail2ban/fail2ban.conf

The main configuration file defines basic things like

• The log file location (fail2ban has its own log file)
• The log level
• The location of the socket and the PID file

Jails configuration file

In addition to the main configuration file there is another file that declares the so-called "jails":

/etc/fail2ban/jail.conf

A "jail" is a combination of

• Which log file to watch
• What to watch for. This is called a "filter". Filter definitions are located as separate files in the filter.d subfolder.
• What actions to take if something has been detected. Action definitions are located as separate files in the action.d subfolder.

A new jail can be declared by creating a new jail configuration file in

/etc/fail2ban/jail.d

Jail configurations in the jail.d subfolder are automatically added to the jails that are defined in jail.conf. Also settings in jail.d override settings in jail.conf. Example:

• With the following line in the [DEFAULT] section, jail.conf defines that jails are disabled by default: enabled = false
• In jail.d/defaults-debian.conf the "sshd" jail is re-enabled by setting enabled = true just for that single jail.

Example: The "sshd" jail

As an example, let's take a look at the "sshd" jail. This is the only jail that is enabled when fail2ban is newly installed, all other jails in jail.conf are disabled.

Here's the snippet for the "sshd" jail from jail.conf:

[sshd]
enabled  = true
port     = ssh
logpath = %(sshd_log)s
backend = %(sshd_backend)s

Discussion:

• Not visible in the jail declaration is that the jail is enabled (enabled = true). Although the [DEFAULT] section in jail.conf specifies that jails are disabled by default, the "sshd" jail is re-enabled in jail.d/defaults-debian.conf.

• The jail watches the log file /var/log/auth.log. The file name is determined like this:
  • "sshd_log" is a placeholder. Its value is defined in paths-common.conf, which in turn is included by jail.conf.
  • The placeholder "sshd_log" is defined like this: sshd_log = %(syslog_authpriv)s
  • "syslog_authpriv" is yet another placeholder, whhich is defined like this: syslog_authpriv = /var/log/auth.log

• The backend setting specifies the backend used to get modifications to the log file. The backend is determined like this:
  • "sshd_backend" is a placeholder. Its value is defined in paths-common.conf, which in turn is included by jail.conf.
  • The placeholder "sshd_backend" is defined like this: sshd_backend = %(default_backend)s
  • "default_backend" is yet another placeholder, which is defined defined like this: default_backend = %(default/backend)s
  • "default/backend" is a third placeholder which refers to the backend option in the [DEFAULT] section.
  • That option is defined in jail.conf like this: backend = auto
  • The value auto means that fail2ban will try to use the following backends, in order: pyinotify, gamin, polling

• The filter named "sshd" is used to examine the content of the log file. The filter definition is located in a separate file in the filter.d subfolder, in this case in /etc/fail2ban/filter.d/sshd.conf. The filter name is determined like this:
  • The filter name is defined by the filter option.
  • The jail configuration does not explicitly set filter, it inherits the filter definition from the [DEFAULT] section: filter = %(__name__)s[mode=%(mode)s]
  • "__name__" is a keyword that refers to the jail name, in this case "sshd".
  • "mode" is a placeholder. Its value is defined in the [DEFAULT] section: mode = normal

• The jail takes action if the same offender has been detected for the 5th time within 600 seconds (10 minutes). These values are determined like this:
  • The number of failures before a host get banned is defined by the maxretry option.
  • The jail configuration does not explicitly set maxretry, it inherits the value from the [DEFAULT] section: maxretry = 5
  • The timeframe within which maxretry failures have to occur is defined by the findtime option.
  • The jail configuration does not explicitly set findtime, it inherits the value from the [DEFAULT] section: findtime = 10m

• The action "iptables-multiport" is used to take action. The action definition is located in a separate file in the action.d subfolder, in this case in /etc/fail2ban/action.d/iptables-multiport.conf. The action name is determined like this:
  • The action name is defined by the action option.
  • The jail configuration does not explicitly set action, it inherits the action definition from the [DEFAULT] section: action = %(action_)s
  • "action_" is a placeholder. Its value is defined like this: action_ = %(banaction)s[name=%(__name__)s, bantime="%(bantime)s", port="%(port)s", protocol="%(protocol)s", chain="%(chain)s"]
  • "banaction" is yet another placeholder, which is defined defined like this: banaction = iptables-multiport

• The offender is banned for 600 seconds (10 minutes). This value is determined like this:
  • The duration how long a host gets banned is defined by the bantime option.
  • The jail configuration does not explicitly set bantime, it inherits the value from the [DEFAULT] section: bantime = 10m

Placeholders

As we have seen above, some sort of placeholder handling can take place. For instance

action = %(action_)s

The construct %(...)s is a construct from Python that is variously called "string formatting operator" or "string interpolation operator". This works similarly as sprintf(). Without going into the (complicated) details, the construct %(action_)s simply acts as a placeholder for the content of the variable that is "action_".

fail2ban extends the Python string interpolation syntax in three ways:

• A placeholder can refer to a value in another config file section using the syntax %(section/parameter)s, where "section" and "parameter" are the desired section and parameter names. As a special case, the syntax %(default/parameter)s refers to a value from the [DEFAULT] section - the man page does not clearly state whether "default" is a special keyword here. Example: default_backend = %(default/backend)s.
• A placeholder can refer to the "last known value" of an option using the syntax %(known/parameter)s. The keyword "known" must be used for this syntax. As a special case, the syntax %(default/parameter)s refers to a value from the [DEFAULT] section. Example: default_backend = %(default/backend)s.

A third way how placeholders can work is by using the "__name__" syntax:

• Example: filter = %(__name__)s[mode=%(mode)s]
• Apparently "__name__" is a keyword that refers to the jail name.
• I have not found a formal documentation of this syntax anywhere in the jail.conf man page or the Python PEP that specifies the literal string interpolation feature.

Some references:

• http://unix.stackexchange.com/questions/156271/what-are-these-s-strings-in-fail2bans-jail-conf-file-and-how-do-they-wor
• https://docs.python.org/2/library/stdtypes.html
• man jail.conf

Customizing the default configuration

If you want to change something about the default configuration of fail2ban you could simply edit the configuration file in question and make your changes. This becomes a problem when you want to install a new version of fail2ban, because then you will have to manually merge your customizations with the default configuration from the new package. For this reason fail2ban offers a better way to make customizations.

For every configuration file you can create another file with the same name, but replacing ".conf" with ".local". For instance

/etc/fail2ban/jail.local

This local file does not replace the original file, it contains settings that override those from the original file.

Important: This mechanism works not only for jail.conf / jail.local, but also for filter configurations in the filter.d subfolder, and for action configurations in the action.d subfolder.

Filters

A jail can have only one filter.

Filter definitions are located in

/etc/fail2ban/filter.d

This section has no more information because until now I did not have to deal with filter definitions - the default filter definitions have been sufficient for my needs.

Actions

A jail can have multiple actions.

Action definitions are located in

/etc/fail2ban/action.d

An action definition is an .ini file that consists of two parts:

• An "[Init]" section that defines some initial values
• A "[Definition]" section that defines the various actions that can be executed

The following actions can appear in the "[Definition]" section

• actionstart = Action to perform when the jail starts
• actionstop = Action to perform when the jail ends
• actionban = Action to perform when a ban occurs
• actionunban = Action to perform when the ban is lifted
• actioncheck = Action to perform before any other action; this is supposed to be used to "check if the enironment is still OK" (quote from man jail.conf)

Actions can be parameterized with so-called "tags":

• A "tag" is simply a placeholder, or a variable
• A "tag" is delimited by angular brackets. In the following example, <foo> denotes a tag named "foo":

actionban = /path/to/some/tool --parameter1 <foo>

• Before an action is executed, all of its tags are replaced with their current value.
• All settings made in the action's "[Init]" section work as tags for that action. So to provide a value for tag "foo" from the above example, an action could specify this in its "[Init]" section:

[Init]
foo = bar

• In addition to action-specific tags, fail2ban also defines a couple of useful generic/global tags (e.g. <ip>)
• When you "call" an action from a jail in jail.conf you can override the values that the action sets in its "[Init]" section. For an action called "doIt" that contains the "foo" setting from the examples above, this would look like this:

action = doIt[foo=yoyodyne]

Example: The "badips" action

As an example, let's take a look at the "badips" action. Here is the jail configuration that "calls" the action. Note that the action option in this example defines two actions.

root@pelargir:~# cat /etc/fail2ban/jail.local
[sshd]
action = %(action_)s
         badips[category=ssh]

And here is the actual action definition:

root@pelargir:~# cat /etc/fail2ban/action.d/badips.conf 
# Fail2ban reporting to badips.com
#
# Note: This reports an IP only and does not actually ban traffic. Use
# another action in the same jail if you want bans to occur.
#
# Set the category to the appropriate value before use.
#
# To get see register and optional key to get personalised graphs see:
# http://www.badips.com/blog/personalized-statistics-track-the-attackers-of-all-your-servers-with-one-key

[Definition]

actionban = curl --fail  --user-agent "<agent>" http://www.badips.com/add/<category>/<ip>

[Init]

# Option: category
# Notes.: Values are from the list here: http://www.badips.com/get/categories
category = 

Discussion

• The "badips" action only defines one of the 5 possible actions, because only that one is necessary to report the IP address to badips.com
• The action "actionban" uses the tag "<category>"
• The "<category>" tag is replaced with the value from the "category" setting in the "[Init]" section
• The "category" setting is overridden with the value "ssh" when the "badips" action is called from the "sshd" jail
• This example also demonstrates how a part of a jail definition in jail.conf can be overridden by values in jail.local

Log file

fail2ban has its own log file where it logs all bans and unbans:

/var/log/fail2ban.log

This can be useful if you want to have a jail that monitors for repeat offenders.

The following command counts all ban actions in all the log files that are currently present on the system (both compressed and uncompressed), and prints them grouped by IP address and sorted descending by "IP address with most bans":

zcat --force /var/log/fail2ban.log* | grep "fail2ban.actions.*\[sshd\] Ban " | sed -e 's/^.* Ban //' | sort | uniq -c | sort -nr

Example output:

root@pelargir:~# zcat --force /var/log/fail2ban.log* | grep "fail2ban.actions.*\[sshd\] Ban " | sed -e 's/^.* Ban //' | sort | uniq -c | sort -nr
    288 116.31.116.48
     68 116.31.116.49
     67 116.31.116.44
     49 91.224.160.10
     38 116.31.116.45
[...]

Using the client

This command starts the server and configures it:

fail2ban-client start

This command clears out the server's current configuration and re-configures it with the values that the client reads from the current configuration files:

fail2ban-client reload

This command shows the status of a specific jail in the server. If no jail is specified the command lists the jails that are configured in the server.

fail2ban-client status <jail-name>

Example output:

root@pelargir:~# fail2ban-client status sshd
Status for the jail: sshd
|- Filter
|  |- Currently failed:	6
|  |- Total failed:	11550
|  `- File list:	/var/log/auth.log
`- Actions
   |- Currently banned:	3
   |- Total banned:	1571
   `- Banned IP list:	222.186.175.150 112.85.42.232 218.92.0.171

This command shows how to send a command to the server. It sets the server's log level.

fail2ban-client set loglevel 1

This command reads the configuration files and dumps them in a format that corresponds to the commands that will be sent to the server if the "start" or "reload" commands are used.

fail2ban-client -d

Example output:

root@pelargir:/etc/fail2ban# fail2ban-client -d
['set', 'syslogsocket', 'auto']
['set', 'loglevel', 'INFO']
['set', 'logtarget', '/var/log/fail2ban.log']
['set', 'dbfile', '/var/lib/fail2ban/fail2ban.sqlite3']
['set', 'dbpurgeage', '1d']
['add', 'sshd', 'auto']
['set', 'sshd', 'maxlines', 1]
['set', 'sshd', 'prefregex', '^<F-MLFID>(?:\\[\\])?\\s*(?:<[^.]+\\.[^.]+>\\s+)?(?:\\S+\\s+)?(?:kernel: \\[ *\\d+\\.\\d+\\]\\s+)?(?:@vserver_\\S+\\s+)?(?:(?:(?:\\[\\d+\\])?:\\s+[\\[\\(]?sshd(?:\\(\\S+\\))?[\\]\\)]?:?|[\\[\\(]?sshd(?:\\(\\S+\\))?[\\]\\)]?:?(?:\\[\\d+\\])?:?)\\s+)?(?:\\[ID \\d+ \\S+\\]\\s+)?</F-MLFID>(?:(?:error|fatal): (?:PAM: )?)?<F-CONTENT>.+</F-CONTENT>$']

[...]

Regexes can be tested against a string, or against the content of a file:

fail2ban-regex "line" "failregex"
fail2ban-regex /var/log/auth.log "failregex"

If the regex is in a file:

fail2ban-regex /var/log/auth.log /etc/fail2ban/filter.d/sshd.conf

Manually unban an IP

fail2ban-client set <jail_name> unbanip <ip_address>

Configuration customizations on pelargir

Currently I only have redefined the "sshd" jail to submit banned IP addresses to badips.com. To achieve this I have overriden the ban action for the jail like this:

root@pelargir:~# cat /etc/fail2ban/jail.local
[ssh]
action = %(action_)s
         badips[category=ssh]

badips.com

From the badips.com website

badips.com is a community based IP blacklist service. You can report malicious IPs and you can download blacklists or query our API to find out if a IP is listed.

As shown in the section that lists my fail2ban configuration customizations, it is very easy to add reporting to badips.com to fail2ban. Once you have reported your first IP address, badips.com assigns an API key to you, probably based on your own IP address.

The following command retrieves your API key. Note: The jq tool pretty-prints the JSON output from badips.com. If you haven't got jq on your system you can simply leave it out of the command:

root@pelargir:/etc/fail2ban# wget https://www.badips.com/get/key -qO - | jq '.'
{
  "err": "",
  "suc": "Your Key was already present! To overwrite, see http://www.badips.com/apidoc.",
  "key": "your-api-key"
}

This command prints the URL that you can open in your browser to get your personalized stats and graphs:

wget https://www.badips.com/get/key -qO - | jq '@uri "https://www.badips.com/stats?key=\(.key)"'