{"id":595,"date":"2024-01-09T18:22:46","date_gmt":"2024-01-09T16:22:46","guid":{"rendered":"https:\/\/wordpress.familie-lahme.de\/?p=595"},"modified":"2024-01-09T23:16:30","modified_gmt":"2024-01-09T22:16:30","slug":"reverse-proxy-and-fail2ban","status":"publish","type":"post","link":"https:\/\/wordpress.familie-lahme.de\/index.php\/2024\/01\/09\/reverse-proxy-and-fail2ban\/","title":{"rendered":"Reverse Proxy and Fail2Ban"},"content":{"rendered":"\n

This is a little story about my way to protect my web servers from being compromised.<\/p>\n\n\n\n

Many of you know the story on a Linux system to have fail2ban in place. It is a good and easy to run tool with a lot of preconfigured stuff and well maintained.<\/p>\n\n\n\n

As many low level home user like me, I do also run some server behind my ISP access point having just one official dynamic IP address in place. On any regular access point -like in my case a standard Fritz!Box, you can configure port forwarding. Meaning, you can route incoming traffic on a specific port to one specific private IP address.<\/p>\n\n\n\n

Now you can go the simple way and just install all of your web based applications onto that one host using host headers or subfolder.<\/p>\n\n\n\n

For example: having “wordpress<\/em>.familie-lahme.de<\/strong>” and “nextcloud<\/em>.familie-lahme.de<\/strong>” could also be solved by having “www.familie-lahme.de\/wordpress<\/em><\/strong>” or “www.familie-lahme.de\/nextcloud<\/em><\/strong>“<\/p>\n\n\n\n

I believe for most common people this is fair enough and you will obviously never get to a point, where it is really a disadvantage.<\/p>\n\n\n\n

For those coming from the enterprise business you will be familiar with the most reasons, why such a setup is devastating to a continues business. I will not walk though this huge list of possibilities, instead I will focus on the little solution I found for myself.<\/p>\n\n\n\n

First of all the<\/p>\n\n\n\n

Reverse Proxy<\/h2>\n\n\n\n

Maybe You already hear of a “proxy server<\/strong>” when talking about how to connect to a web site outside your network. The proxy server can be seen as a gate or portal between two worlds. You sent a request for any web page you would like to load, and the proxy server gets it for you. Most commonly a proxy server can reduce network traffic by a significant amount, if you imaging a company of 1000 or more employees and everybody is loading the current news paper. The proxy server can store pages it has been loading from a previous request and serve it to any same request from other computers. But it can do a lot of stuff more, if it is about security. But that is on another sheet of this book.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

Such a proxy is also called a “forward proxy”. But with this name, it seems to get obvious, there also must be a “reverse proxy”, and indeed there is.<\/p>\n\n\n\n

Now how can you describe the difference? Let’s try it this way:<\/p>\n\n\n\n

A forward proxy serves one user a lot of web pages.
A reverse proxy serves one web page to a lot of users.<\/p>\n\n\n\n

While this is a very basic approach, in common it is exactly this, but it is not a limit for none of both.<\/p>\n\n\n\n

So, let me try to explain, how a reverse proxy works.<\/p>\n\n\n\n

Let’s assume your company has a hand full of web based applications hosted inside a secure network, not facing the public internet at all. Nevertheless the clients have to access those applications from any location outside the companies network.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

We define<\/p>\n\n\n\n

“<\/strong>application-one.mycompany.com”<\/strong> as the public application name\/url
“<\/strong>application-one.internal.network”<\/strong> as the company internal application name\/url<\/p>\n\n\n\n

This is, where the Reverse Proxy can be one out of a lot of solutions. You put one single server connected to the public facing IP address and connect “the other side” to your company network. On this server you install the reverse proxy and define an external web page, lets name it “application-one.mycompany.com<\/strong>“. You also take care, that all traffic from the public internet on the relevant ports (http[s]) is routed to that server. In the second step you need to tell the proxy, where to route the requests to. So you tell it, to connect to “application-one.internal.network<\/strong>“.<\/p>\n\n\n\n

What will happen if a remote client connects to “application-one.mycompany.com”<\/strong> is, that the reverse proxy server will get the request, transforms it, and connects to the internal application with the transformed request. The application only “sees” the proxy server, as well as the client only sees the proxy server. There is no direct connection between client and application.<\/p>\n\n\n\n

This begins to make more sense, if you add more applications to the company network, but you still have only one line (ip address) to connect from outside using the reverse proxy.<\/p>\n\n\n\n

There are several more advantages if we consider availability or security. But as stated in the beginning, that’s another story.<\/p>\n\n\n\n

Now what has this all in common with fail2ban? Why is it so special?<\/p>\n\n\n\n

Let’s make a short journey, what<\/p>\n\n\n\n

Fail2Ban<\/h2>\n\n\n\n

is mend to do.<\/p>\n\n\n\n

Fail2ban is a tool to monitor log files in any custom way and to react on custom definitions to special events. In common it is used to block access to a specific IP address after a sequence of errors coming from it. For example somebody tries to brute force a password on your web page, fail2ban will block his IP after a certain amount of failed attempts, so he cannot continue.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

Fail2Ban uses the local IP tables under Linux for this. IP Tables store rules how to handle certain network requests on a host. You can have complicated stuff in there, but you also can have a very simple “REJECT” for dedicated IP Addresses. And that is, what fail2ban makes use of. For any action in the most common way, it creates a record in that tables to REJECT the incoming IP address. It is essential to know, that iptables has no relation to the data that is being transferred. It is only looking at the network connection itself.<\/p>\n\n\n\n

And this being said, you might already get a glimpse, why this can be an issue, if working with reverse proxies.<\/p>\n\n\n\n

As explained above the reverse proxy modifies some data in the header information coming from the client on the internet. One of this is the IP address. Let’s create a rough example:<\/p>\n\n\n\n

The client is somewhere on the public net having the IP 156.234.34.198
The Reverse Proxy has the IP 123.231.132.222 on the public facing side
and 10.10.10.254 on the company internal facing side
The Application server owns the IP 10.10.10.123<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

The Reverse Proxy gets a request from 156.234.34.198 on 123.231.132.222
Now it starts to modify the request data and puts its own IP into the requesting stanza.
It also puts the real client IP in a similar field.
The Application now gets a request from 10.10.10.254 on 10.10.10.123 containing the special field with the client IP<\/p>\n\n\n\n

Now lets assume for whatever reason, the client gets a fail2ban action on the application.
Fail2ban will now try to block the connection. We now have two possible reactions:<\/p>\n\n\n\n

    \n
  1. Fail2Ban sends the IP address of the proxy server to iptables. From that moment on, nobody will be able to access the application, as IP tables will now block any connection coming from the reverse proxy.<\/li>\n\n\n\n
  2. Fail2Ban sends the packaged IP of the Client from the Proxy Request to IP tables. This will not impact the clients connection, as IP tables only “sees” the IP address from the proxy server on its level.<\/li>\n<\/ol>\n\n\n\n

    Whatever we do at this point, we are unable to provide a reliable fail2ban setup.<\/p>\n\n\n\n

    So, what could be another approach to solve this?<\/p>\n\n\n\n

    Let’s see what we have to establish in any way. The goal is, that the proxy server itself get the information to block the incoming IP address from that client. So the IP tables modification has to take place on the proxy server.<\/p>\n\n\n\n

    At what level do we get the information, when to block a client?<\/p>\n\n\n\n

    Fail2Ban is parsing logfiles. We can customize how the logfiles are read.<\/p>\n\n\n\n

    Taking those two ends of the string and trying to put them together, must pin point, that in any case we need to transfer at least one of the ends to the proxy server.<\/p>\n\n\n\n

    Considering, we transfer the log files would lead to the consequence, that any further application would have to sent the logfiles to the proxy, too. This would start to stress the network and would start to stress the proxy. Anyway, if you are interested in such a solution, check for “rsyslog”. This will get you there.<\/p>\n\n\n\n

    Considering, we transfer the fail2ban action to the proxy, brought me to the most fitting solution in my situation. I found some similar approaches on the net, where such solutions have been set into place, so I started to customize mine.<\/p>\n\n\n\n

    Let’s have a look, how fail2ban acts in case of an action.<\/p>\n\n\n\n

    Fail2Ban is working in what you can call a 3 step workflow, or in one sentence:<\/p>\n\n\n\n

    Inside a Fail2Ban jail a filter is leading to actions.<\/strong><\/p>\n\n\n\n

    So for any application we want to be processed in fail2ban, we have to setup a jail.
    Within that jail, we use a filter.
    If the filter catches something, an action is triggered.<\/p>\n\n\n\n

    Let’s focus on the action to ban a client IP from access:<\/p>\n\n\n\n

    actionban = <iptables> -I f2b-<name> 1 -s <ip> -j <blocktype><\/code><\/pre>\n\n\n\n
    This is a code line from the “iptables-allports.conf”. What you see in brackets are variables. So for example the <iptables> contains the location of the iptables command, mainly “\/usr\/sbin\/iptables”. <name> is given by the jail itself. So if you name the jail “keksdose” this will be resolved to “f2b-keksdose”. <ip> is coming from the filter, reading the ip address in question. In our scenario the IP of the remote client, trying to brute force our system. And last but not least <blocktype> in this case holds the simple “REJECT” rule in place.<\/p>\n\n\n\n
    Putting this in place of the definition of <actionban> the command as resolved above is sent to the local command line of the system, resulting in a corresponding iptables record. At that stage it should be mentioned, that fail2ban is running in the context of root -so no user security layer in between.<\/p>\n\n\n\n
    So the goal we have in mind is to send this command to the proxy server.<\/p>\n\n\n\n
    The most common command for remoting on Linux system is SSH, the secure shell. And this is, what we make use of in this case, too. SSH has a neat feature, where you can send console or script commands directly to the remote command line on the system you are connecting to.<\/p>\n\n\n\n
    so something like<\/p>\n\n\n\n
    ssh myspecialuser@remotesystem 'echo \"Hallo Welt\"'<\/code><\/pre>\n\n\n\n
    will result in an output of “Hallo Welt” in the context of the remote machine. Imagine what happens, if you sent a “reboot” \ud83d\ude09<\/p>\n\n\n\n
    Now what we want is, that the “echo” part of the example is replaced by the “actionban” command from fail2ban. This is quite simple, as we can put that nearly seamlessly into place. In here I take a cleaner approach, but it results in the same :<\/p>\n\n\n\n
    [Definition]\n\n# set the ssh connection \n# we are running as root and the ssh key is in the default location\n\nPROXYSERVER = proxyserver.mycompany.net\nSSHUSER = fail2banuser\nSSHCOMMAND = ssh <SSHUSER>@<PROXYSERVER> sudo\n\n# we just need to extend the regular commands with the SSH sommand\n\nactionban = <SSHCOMMAND> <iptables> -I f2b-<name> 1 -s <ip> -j <blocktype>\n<\/code><\/pre>\n\n\n\n
    So what we do in here is exactly what I explained above. We create another variable <sshcommand> and place it in front of the regular iptables command. For better readability I have nested the proxy server and the user used for the connection. And you might have seen the little thing “sudo<\/strong>” at the end of the <sshcommand>.<\/p>\n\n\n\n
    \"\"<\/figure>\n\n\n\n
    Why “sudo”?<\/p>\n\n\n\n
    Well for obvious reasons, modifying the iptables is a security relevant thing, that should not be done by any regular user, but has to have root privileges. Nevertheless, we do not want to have a remote user connecting to the server, that is root by default. So we have to put “sudo” in place to elevate the privileges on this command.<\/p>\n\n\n\n
    BUT….. sudo is commonly asking for a password, so this could not work.<\/p>\n\n\n\n
    Well, yes, this is indeed a fact, that the sudo will ask for password, if we do not implement another step. Even this seems a little odd considering all the security thoughts, we have to “unpassword” this -and only this; command.<\/p>\n\n\n\n
    VISUDO is needed.<\/h2>\n\n\n\n
    Beside the option of putting a user completely to superuser group, you can also define single commands that a regular user is able to use. This is done by putting special configurations to the “VISUDO” file. Be reminded, this is a delicate step and needs proper handling. Doing mistakes in here, can lead to the situation, that you will not be able to access your system at all. So let’s crack it.<\/p>\n\n\n\n
    On the proxy server, open the “visudo” command as root (ex: sudo visudo). A text editor (vim\/nano) will show up, displaying the current configuration. Leave everything as it is, and scroll to the end of the file. At that point we add the config for our new user:<\/p>\n\n\n\n
    # Fail2Ban remoting\nfail2banuser ALL=(ALL) NOPASSWD:\/usr\/sbin\/iptables\n<\/code><\/pre>\n\n\n\n
    HINT: as written in the comments of the file, there has to be a blank line at the end of that file.<\/strong><\/p>\n\n\n\n
    Save and exit the file.<\/p>\n\n\n\n
    From now on, the user “fail2banuser” will be able to sudo iptables without being prompted for a password. This enables unattended remote ssh commands like we need them.<\/p>\n\n\n\n
    There are three steps, that I skipped up to now:<\/p>\n\n\n\n
      \n
    1. You need to create the fail2banuser on the proxy server (adduser)<\/li>\n\n\n\n
    2. You need to create a rsa key pair for the user root on the application server (ssh-keygen)<\/li>\n\n\n\n
    3. copy the public ssh key to the proxy server for the user fail2banuser (ssh-copy-id)<\/li>\n<\/ol>\n\n\n\n
      At the end you need to make sure, that from the context of the root user on the application server you can establish a password less connection using ssh to the proxy server.<\/p>\n\n\n\n
      If this fits, we can go for a test ride.<\/p>\n\n\n\n
      Let me take my Gitea server as example -I will obscure the real names.
      This is the jail:<\/strong><\/p>\n\n\n\n
      [gitea]\nenabled = true\nfilter = gitea\nlogpath = \/var\/log\/syslog\nmaxretry = 10\nfindtime = 3600\nbantime = 900\naction = iptables-allports-remote<\/code><\/pre>\n\n\n\n
      and here the filter:<\/strong><\/p>\n\n\n\n
      # gitea.conf\n[Definition]\nfailregex =  ^.+ gitea\\[[0-9]{1,5}\\]:.+(Failed authentication attempt for ).+ from <HOST>\nignoreregex =<\/code><\/pre>\n\n\n\n
      and last but not least, the action:<\/strong><\/p>\n\n\n\n
      [INCLUDES]\n\nbefore = iptables-common.conf\n\n[Definition]\n\n# set the ssh connection \n# we are running as root and the ssh key is in the default location\nPROXYSERVER = proxy.mycompany.net\nSSHUSER = fail2banuser\nSSHCOMMAND = ssh <SSHUSER>@<PROXYSERVER> sudo\n\n# we just need to extend the regular commands with the SSH sommand\n\nactionban = <SSHCOMMAND> <iptables> -I f2b-<name> 1 -s <ip> -j <blocktype>\n\nactionunban = <SSHCOMMAND> <iptables> -D f2b-<name> -s <ip> -j <blocktype>\n\nactionstart = <SSHCOMMAND> <iptables> -N f2b-<name>\n              <SSHCOMMAND> <iptables> -A f2b-<name> -j <returntype>\n              <SSHCOMMAND> <iptables> -I <chain> -p <protocol> -j f2b-<name>\n\nactioncheck = <SSHCOMMAND> <iptables> -n -L <chain> | grep -q 'f2b-<name>[ \\t]'\n\nactionstop = <SSHCOMMAND> <iptables> -D <chain> -p <protocol> -j f2b-<name>\n             <SSHCOMMAND> <actionflush>\n             <SSHCOMMAND> <iptables> -X f2b-<name>\n\n[Init]<\/code><\/pre>\n\n\n\n
      Please keep in mind, this is of course not the whole solution of setting up a proxy and an application server or all the surroundings of such an environment. This should just get you to the point of understanding this particular solution and its corners.<\/p>\n\n\n\n
      Make sure you read some documents of how to use fail2ban in general to get more familiar with it.
      For the reverse proxy stuff You can also find a lot of solutions for nearly any application in the wild.
      And keep in mind, Linux is fun to play around with, but you should keep an eye on your knowledge of it.
      Anyway do not let you being stop by any so call expert on the internet, saying, you are to dump to use Linux. And if you are also playing around with Windows, hey, better that! A penny is only worth a penny, because it has two sides \ud83d\ude09<\/p>\n\n\n\n

      Feel free to adapt and enhance at your will.<\/p>\n","protected":false},"excerpt":{"rendered":"
      This is a little story about my way to protect my web servers from being compromised. Many of you know the story on a Linux system to have fail2ban in place. It is a good and easy to run tool with a lot of preconfigured stuff and well maintained. As many low level home user…<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[31],"tags":[42,46,44,34,45,43,47],"class_list":["post-595","post","type-post","status-publish","format-standard","hentry","category-computer","tag-apache","tag-computer","tag-fail2ban","tag-headless","tag-linux","tag-reverse-proxy","tag-security"],"_links":{"self":[{"href":"https:\/\/wordpress.familie-lahme.de\/index.php\/wp-json\/wp\/v2\/posts\/595","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/wordpress.familie-lahme.de\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/wordpress.familie-lahme.de\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/wordpress.familie-lahme.de\/index.php\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/wordpress.familie-lahme.de\/index.php\/wp-json\/wp\/v2\/comments?post=595"}],"version-history":[{"count":8,"href":"https:\/\/wordpress.familie-lahme.de\/index.php\/wp-json\/wp\/v2\/posts\/595\/revisions"}],"predecessor-version":[{"id":610,"href":"https:\/\/wordpress.familie-lahme.de\/index.php\/wp-json\/wp\/v2\/posts\/595\/revisions\/610"}],"wp:attachment":[{"href":"https:\/\/wordpress.familie-lahme.de\/index.php\/wp-json\/wp\/v2\/media?parent=595"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/wordpress.familie-lahme.de\/index.php\/wp-json\/wp\/v2\/categories?post=595"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/wordpress.familie-lahme.de\/index.php\/wp-json\/wp\/v2\/tags?post=595"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}