fail2ban fixups on penguin

Brian

2018-11-11 12:25

Contents

Issue: httpd logs are not being processed
Analysis
Fixup: install pyinotify and fix up jail.local

Issue: httpd logs are not being processed

The first clue I had that fail2ban wasn’t working as expected was messages similar to the following in /var/log/fail2ban.log:

INFO Jail 'sshd-ddos' started
NOTICE Jail started without 'journalmatch' set. Jail regexs will be checked against all journal entries, which is not advised for performance reasons.
INFO Jail 'apache-auth' started
NOTICE Jail started without 'journalmatch' set. Jail regexs will be checked against all journal entries, which is not advised for performance reasons.
INFO Jail 'apache-badbots' started
NOTICE Jail started without 'journalmatch' set. Jail regexs will be checked against all journal entries, which is not advised for performance reasons.
INFO Jail 'apache-noscript' started
NOTICE Jail started without 'journalmatch' set. Jail regexs will be checked against all journal entries, which is not advised for performance reasons.
INFO Jail 'apache-overflows' started
NOTICE Jail started without 'journalmatch' set. Jail regexs will be checked against all journal entries, which is not advised for performance reasons.
INFO Jail 'apache-nohome' started
NOTICE Jail started without 'journalmatch' set. Jail regexs will be checked against all journal entries, which is not advised for performance reasons.
INFO Jail 'apache-botsearch' started
NOTICE Jail started without 'journalmatch' set. Jail regexs will be checked against all journal entries, which is not advised for performance reasons.
INFO Jail 'apache-fakegooglebot' started
NOTICE Jail started without 'journalmatch' set. Jail regexs will be checked against all journal entries, which is not advised for performance reasons.
INFO Jail 'apache-modsecurity' started
NOTICE Jail started without 'journalmatch' set. Jail regexs will be checked against all journal entries, which is not advised for performance reasons.
INFO Jail 'apache-shellshock' started
NOTICE Jail started without 'journalmatch' set. Jail regexs will be checked against all journal entries, which is not advised for performance reasons.
INFO Jail 'vsftpd' started
INFO Jail 'postfix' started
INFO Jail 'sendmail-auth' started
INFO Jail 'sendmail-reject' started
INFO Jail 'dovecot' started
NOTICE Jail started without 'journalmatch' set. Jail regexs will be checked against all journal entries, which is not advised for performance reasons.
INFO Jail 'sieve' started
INFO Jail 'postfix-sasl' started
NOTICE Jail started without 'journalmatch' set. Jail regexs will be checked against all journal entries, which is not advised for performance reasons.

Analysis

Figuring out what was going on required some digging. I used the apache-auth jail for my analysis.

First, the jail is defined in /etc/fail2ban.d/jails.conf: First, the jail is defined in /etc/fail2ban.d/jails.conf:

[apache-auth]

port     = http,https
logpath  = %(apache_error_log)s

The value of the logpath parameter is the value of the variable apache_error_log, which is defined in paths-fedora.conf:

apache_error_log = /var/log/httpd/*error_log

The [apache-auth] section header is a reference to filter.d/apache-auth.conf, which contains the following:

# Fail2Ban apache-auth filter
#

[INCLUDES]

# Read common prefixes. If any customizations available -- read them from
# apache-common.local
before = apache-common.conf

[Definition]

failregex = ^%(_apache_error_client)s (AH(01797|01630): )?client denied by server configuration: (uri )?\S*(, referer: \S+)?\s*$
            ^%(_apache_error_client)s (AH01617: )?user .*? authentication failure for "\S*": Password Mismatch(, referer: \S+)?$
            ^%(_apache_error_client)s (AH01618: )?user .*? not found(: )?\S*(, referer: \S+)?\s*$
            ^%(_apache_error_client)s (AH01614: )?client used wrong authentication scheme: \S*(, referer: \S+)?\s*$
            ^%(_apache_error_client)s (AH\d+: )?Authorization of user \S+ to access \S* failed, reason: .*$
            ^%(_apache_error_client)s (AH0179[24]: )?(Digest: )?user .*?: password mismatch: \S*(, referer: \S+)?\s*$
            ^%(_apache_error_client)s (AH0179[01]: |Digest: )user `.*?' in realm `.+' (not found|denied by provider): \S*(, referer:
            ^%(_apache_error_client)s (AH01631: )?user .*?: authorization failure for "\S*":(, referer: \S+)?\s*$
            ^%(_apache_error_client)s (AH01775: )?(Digest: )?invalid nonce .* received - length is not \S+(, referer: \S+)?\s*$
            ^%(_apache_error_client)s (AH01788: )?(Digest: )?realm mismatch - got `.*?' but expected `.+'(, referer: \S+)?\s*$
            ^%(_apache_error_client)s (AH01789: )?(Digest: )?unknown algorithm `.*?' received: \S*(, referer: \S+)?\s*$
            ^%(_apache_error_client)s (AH01793: )?invalid qop `.*?' received: \S*(, referer: \S+)?\s*$
            ^%(_apache_error_client)s (AH01777: )?(Digest: )?invalid nonce .*? received - user attempted time travel(, referer: \S+)

ignoreregex =

# DEV Notes:
#
# This filter matches the authorization failures of Apache. It takes the log messages
# from the modules in aaa that return HTTP_UNAUTHORIZED, HTTP_METHOD_NOT_ALLOWED or
# HTTP_FORBIDDEN and not AUTH_GENERAL_ERROR or HTTP_INTERNAL_SERVER_ERROR.
#
# An unauthorized response 401 is the first step for a browser to instigate authentication
# however apache doesn't log this as an error. Only subsequent errors are logged in the
# error log.
# 
# Source:
#
# By searching the code in http://svn.apache.org/viewvc/httpd/httpd/trunk/modules/aaa/*
# for ap_log_rerror(APLOG_MARK, APLOG_ERR and examining resulting return code should get
# all of these expressions. Lots of submodules like mod_authz_* return back to mod_authz_core
# to return the actual failure.
#
# See also: http://wiki.apache.org/httpd/ListOfErrors
# Expressions that don't have tests and aren't common.
# more be added with  https://issues.apache.org/bugzilla/show_bug.cgi?id=55284
#     ^%(_apache_error_client)s (AH01778: )?user .*: nonce expired \([\d.]+ seconds old - max lifetime [\d.]+\) - sending new nonce\
#     ^%(_apache_error_client)s (AH01779: )?user .*: one-time-nonce mismatch - sending new nonce\s*$
#     ^%(_apache_error_client)s (AH02486: )?realm mismatch - got `.*' but no realm specified\s*$
#
# referer is always in error log messages if it exists added as per the log_error_core function in server/log.c
#
# Author: Cyril Jaquier
# Major edits by Daniel Black

The value of _apache_error_client is set in filter.d/apache-common.conf:

_apache_error_client = \[\] \[(:?error|\S+:\S+)\]( \[pid \d+(:\S+ \d+)?\])? \[client <HOST>(:\d{1,5})?\]

However, the problem isn’t in any of apache-related files. The real issue is here in /etc/fail2ban.d/jails.conf:

# "backend" specifies the backend used to get files modification.
# Available options are "pyinotify", "gamin", "polling", "systemd" and "auto".
# This option can be overridden in each jail as well.
#
# pyinotify: requires pyinotify (a file alteration monitor) to be installed.
#              If pyinotify is not installed, Fail2ban will use auto.
# gamin:     requires Gamin (a file alteration monitor) to be installed.
#              If Gamin is not installed, Fail2ban will use auto.
# polling:   uses a polling algorithm which does not require external libraries.
# systemd:   uses systemd python library to access the systemd journal.
#              Specifying "logpath" is not valid for this backend.
#              See "journalmatch" in the jails associated filter config
# auto:      will try to use the following backends, in order:
#              pyinotify, gamin, polling.
#
# Note: if systemd backend is chosen as the default but you enable a jail
#       for which logs are present only in its own log files, specify some other
#       backend for that jail (e.g. polling) and provide empty value for
#       journalmatch. See https://github.com/fail2ban/fail2ban/issues/959#issuecomment-74901200
backend = auto

See the note?

Note: if systemd backend is chosen as the default but you enable a jail for which logs are present only in its own log files, specify some other backend for that jail (e.g. polling) and provide empty value for journalmatch. See https://github.com/fail2ban/fail2ban/issues/959#issuecomment-74901200

I didn’t have pyinotify installed, so auto went with systemd. However, although the Apache httpd server is managed by systemd, it writes its own log files that systemd doesn’t read.

Fixup: install `pyinotify` and fix up `jail.local`

First, I installed the pyinotiyfy package, to allow for efficient handling of non-systemd backends.

Then I updated /etc/fail2ban.d/jail.local to update the httpd-auth jail (and serveral others):

[apache-auth]
enabled = true
backend = pyinotify
journalmatch =

fail2ban fixups on penguin

Issue: httpd logs are not being processed

Analysis

Fixup: install `pyinotify` and fix up `jail.local`

About

Recent Posts

Archives

Issue: httpd logs are not being processed

Analysis

Fixup: install pyinotify and fix up jail.local

About

Recent Posts

Archives

Tags

Fixup: install `pyinotify` and fix up `jail.local`