{"id":683,"date":"2025-11-09T16:02:36","date_gmt":"2025-11-09T15:02:36","guid":{"rendered":"https:\/\/wordpress.familie-lahme.de\/?p=683"},"modified":"2025-11-09T16:10:57","modified_gmt":"2025-11-09T15:10:57","slug":"setting-up-letsencrypt-with-multiple-sites-and-remote-certificate-update","status":"publish","type":"post","link":"https:\/\/wordpress.familie-lahme.de\/index.php\/2025\/11\/09\/setting-up-letsencrypt-with-multiple-sites-and-remote-certificate-update\/","title":{"rendered":"Setting up LetsEncrypt with multiple sites and remote certificate update"},"content":{"rendered":"\n<h2 class=\"wp-block-heading\"><strong>Prerequisits<\/strong><\/h2>\n\n\n\n<p>Reading this you should<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>have a brief understanding about Certificates and their usage. <\/li>\n\n\n\n<li>have a brief understanding about Linux and its command line. &nbsp;<\/li>\n\n\n\n<li>know how Apache works, espacially how to run multiple sites.<\/li>\n<\/ul>\n\n\n\n<p>Nevertheless if you do not know anything about the above, I encourage You to take this document to inspire yourself. &nbsp;<\/p>\n\n\n\n<p>Never stop learning!<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Abstract<\/strong><\/h2>\n\n\n\n<div class=\"wp-block-group\"><div class=\"wp-block-group__inner-container is-layout-constrained wp-block-group-is-layout-constrained\">\n<div class=\"wp-block-group\"><div class=\"wp-block-group__inner-container is-layout-constrained wp-block-group-is-layout-constrained\">\n<p>We are hosting multiple sites on one reverse-proxy-server and a remote IMAP server that uses SSL. The CertBot can be triggered using cron to update certificates automatically if required. The certificate challenge (validation) is done using an http request to the named site with a special folder. The setup does not allow http requests to reach the internal site or application. Primarily the reverse proxy translates all http requests to https requests. This needs to be prevented for the cerbot update.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Environment<\/strong><\/h2>\n\n\n\n<p>The setup is highly internal and for private use. This is not a production environment.<br>The uplink consists of a common ISP router having just one dynamic IP v4 address. The router is configured to forward http(80) and https(443) to a dedicate internal IP address -the reverse-proxy (&#8220;hermes&#8221;). This machine processes the incoming requests and distributes to the applications and websites in the backend. The applications are separated by specific host headers like:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>mysite1.mydomain.tld<\/li>\n\n\n\n<li>mysite2.mydomain.tld<\/li>\n\n\n\n<li>a.s.o<\/li>\n<\/ul>\n\n\n\n<p>In addition there is a mail server (&#8220;horus&#8221;) with smtp(25) and IMAPS(993) for mail handling. The SSL enabled IMAPS service (dovecot) needs a valid certificate as well, but is not natively supported by LetsEncrypt certbot.<br>And here starts the story.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Apache flaw<\/strong><\/h2>\n\n\n\n<p>Apache supports serial site processing by default. In general this is done by naming the configuration files for each site with a leading number.<\/p>\n<\/div><\/div>\n<\/div><\/div>\n\n\n\n<ul class=\"wp-block-list\">\n<li>000-mysite1.conf<\/li>\n\n\n\n<li>001-mysite2.conf<\/li>\n\n\n\n<li>a.s.o.<\/li>\n<\/ul>\n\n\n\n<p>But there is a flaw in that approach if the site is name like &#8220;*:80&#8221;.<br>You must not use that value. All &#8220;*&#8221; have to be replaced by an ip address espacially the site you want to have at the top.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>bad example<\/strong><\/h2>\n\n\n\n<pre class=\"wp-block-code\"><code>&gt; cat \/etc\/apache2\/sites-enabled\/000.B-http2https-catcher.conf\n\n## this is a default HTTP catcher.\n## all HTTP requests are reroutet to HTTPS\n## only certbot is directly pushed to its folders\n\n&lt;VirtualHost *:80&gt;\n        ServerAdmin webmaster@mydomain.tld\n        DocumentRoot \/var\/www\/certbot\n        ErrorLog ${APACHE_LOG_DIR}\/certbot_error.log\n        CustomLog ${APACHE_LOG_DIR}\/certbot_access.log combined\n        &lt;Directory \/var\/www\/certbot&gt;\n                Options -Indexes\n                AllowOverride None\n                Require all granted\n\n                ## we would like to send all http requests to https but the certbot updates\n                RewriteEngine On\n#               LogLevel alert rewrite:trace8\n                RewriteCond %{REQUEST_URI} !^\/.well-known\/(acme-challenge|pki-validation)\/.*$\n                RewriteRule (.*) https:\/\/%{HTTP_HOST}%{REQUEST_URI} &#91;L]\n        &lt;\/Directory&gt;\n&lt;\/VirtualHost&gt;<\/code><\/pre>\n\n\n\n<p>This results in an &#8220;`apachectl -S&#8220;` output like this:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>&#91;.. header output ...]\nVirtualHost configuration:\n10.10.10.238:80        mysite0.mydomain.tld (\/etc\/apache2\/sites-enabled\/000.C-mysite0.mydomain.tld.conf:3)\n10.10.10.238:443       is a NameVirtualHost\n         default server mysite1.mydomain.tld (\/etc\/apache2\/sites-enabled\/001ssl-mysite1-proxy.conf:1)\n         port 443 namevhost mysite1.mydomain.tld (\/etc\/apache2\/sites-enabled\/001ssl-mysite1-proxy.conf:1)\n         port 443 namevhost mysite2.mydomain.tld (\/etc\/apache2\/sites-enabled\/002ssl-mysite2-proxy.conf:1)\n         port 443 namevhost mysite3.mydomain.tld (\/etc\/apache2\/sites-enabled\/003ssl-mysite3-proxy.conf:4)\n         port 443 namevhost mysite4.mydomain.tld (\/etc\/apache2\/sites-enabled\/004ssl-mysite4-proxy.conf:1)\n         port 443 namevhost mysite5.mydomain.tld (\/etc\/apache2\/sites-enabled\/005ssl-mysite5-proxy.conf:1)\n         port 443 namevhost mysite6.mydomain.tld (\/etc\/apache2\/sites-enabled\/006ssl-mysite6-proxy.conf:1)\n*:80                   127.0.1.1 (\/etc\/apache2\/sites-enabled\/000.B-http2https-catcher.conf:7)\n&#91;... some more output.. ]<\/code><\/pre>\n\n\n\n<p>You can see, the catcher has been moved to the very end. This results in a processing error, as the pre-processing cannot handle the certbot request and replies with &#8220;page not found&#8221; or &#8220;access denied&#8221;.<br>All other services \/ applications work flawlessly if addressed correct. The certbot error only comes to surface, after the certificates have expired and the certbot is not able to update them automatically.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>correct configuration<\/strong><\/h2>\n\n\n\n<pre class=\"wp-block-code\"><code>## this is a default HTTP catcher.\n## all HTTP requests are reroutet to HTTPS\n## only certbot is directly pushed to its folders\n\n&lt;VirtualHost 10.10.10.238:80&gt;\n&#91;... rest left unchanged ...]<\/code><\/pre>\n\n\n\n<p>This results in the correct serial reading:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>VirtualHost configuration:\n10.10.10.238:80        is a NameVirtualHost\n         default server hermes.mydomain.tld (\/etc\/apache2\/sites-enabled\/000.B-http2https-catcher.conf:5)\n         port 80 namevhost hermes.mydomain.tld (\/etc\/apache2\/sites-enabled\/000.B-http2https-catcher.conf:5)\n         port 80 namevhost mysite0.mydomain.tld (\/etc\/apache2\/sites-enabled\/000.C-mysite0.mydomain.tld.conf:2)\n10.10.10.238:443       is a NameVirtualHost\n         default server mysite1.mydomain.tld (\/etc\/apache2\/sites-enabled\/001ssl-mysite1-proxy.conf:1)\n         port 443 namevhost mysite1.mydomain.tld (\/etc\/apache2\/sites-enabled\/001ssl-mysite1-proxy.conf:1)\n         port 443 namevhost mysite2.mydomain.tld (\/etc\/apache2\/sites-enabled\/002ssl-mysite2-proxy.conf:1)\n         port 443 namevhost mysite3.mydomain.tld (\/etc\/apache2\/sites-enabled\/003ssl-mysite3-proxy.conf:4)\n         port 443 namevhost mysite4.mydomain.tld (\/etc\/apache2\/sites-enabled\/004ssl-mysite4-proxy.conf:1)\n         port 443 namevhost mysite5.mydomain.tld (\/etc\/apache2\/sites-enabled\/005ssl-mysite5-proxy.conf:1)\n         port 443 namevhost mysite6.mydomain.tld (\/etc\/apache2\/sites-enabled\/006ssl-mysite6-proxy.conf:1)<\/code><\/pre>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Catching the cerbot requests<\/strong><\/h2>\n\n\n\n<p>In the catcher config there are two essential lines:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>    RewriteCond %{REQUEST_URI} !^\/.well-known\/(acme-challenge|pki-validation)\/.*$\n    RewriteRule (.*) https:\/\/%{HTTP_HOST}%{REQUEST_URI} &#91;L]<\/code><\/pre>\n\n\n\n<p>Line one defines the condition, when to apply the rule in the second line. In general it says, apply the rule, if the condition is NOT (the exclamation mark (&#8216;!&#8217;) makes it a &#8216;NOT&#8217;) met. The condition is defined as &#8220;[anythinghere][followed by this folder combination]&#8221; It matches on:<br><code>http:\/\/imagingAnything\/<strong>.well-known\/acme-challenge<\/strong>\/imagingAnything<\/code><br>as well as on<br><code>http:\/\/imagingAnything\/<strong>.well-known\/pki-validation<\/strong>\/imagingAnything<\/code><\/p>\n\n\n\n<p>The essentials are the two folder combinations at the beginning. The rule rewrites the requested URL from http to https. From that moment on, apache processes this as a redirect feedback to the client. It can also be read as [R=302,L], where the &#8220;R&#8221; defines the redirection code. The client will then call the newly received URL and the catcher will not be triggered. In any other case the request will be processed in the http catcher folder and if certbot has triggered that request, it will lead to the certification validation file. This file only exists during runtime of the request.<\/p>\n\n\n\n<p>Just to rephrase:<br>This rule redirects all requests, that are not from certbot to their designated ssl-URLs.<br>All certbot requests are left untouched and are handled locally by this default site.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>post processing<\/strong><\/h2>\n\n\n\n<p>As written in the abstract there is also a remote IMAPS mail server (dovecot). LetsEncrypt does not support direct management of such server, but the certificate gained for a specific web site can be reused for this. It has to be copied from the web server to the IMAPS server. This step need to be done after certbot has updated the certificates. Certbot provides a folder structure to process scripts at certain times in the workflow:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>&gt; ll \/etc\/letsencrypt\/renewal-hooks\/\ntotal 20\ndrwxr-xr-x 5 root root 4096 Jul 22  2022 .\/\ndrwxr-xr-x 9 root root 4096 Nov  9 14:10 ..\/\ndrwxr-xr-x 2 root root 4096 Jul 22  2022 deploy\/\ndrwxr-xr-x 2 root root 4096 Apr 17  2024 post\/\ndrwxr-xr-x 2 root root 4096 Jul 22  2022 pre\/<\/code><\/pre>\n\n\n\n<p>In the &#8220;post&#8221; folder the script &#8220;`certtrans.sh&#8220;`:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>#!\/bin\/bash\n\nlogger \"letsencrypt: copy script started\"\n\nlogger \"letsencrypt: start: copy cert to horus\"\nscp \/etc\/letsencrypt\/live\/mysite4.mydomain.tld\/* certupload@horus.mydomain.tld:\nlogger \"letsencrypt: finished: copy cert to horus\"\n\nlogger \"letsencrypt: sending restart dovecot to horus\"\nssh certupload@horus.mydomain.tld sudo systemctl restart dovecot\n\nlogger \"letsencrypt: sending restart postfix to horus\"\nssh certupload@horus.mydomain.tld sudo systemctl restart postfix\n\nlogger \"letsencrypt: copy script finished\"<\/code><\/pre>\n\n\n\n<p>The script uses ssh copy (scp) to transfer the files. The user &#8220;certupload&#8221; is configured using PKP for authentication and the home folder is set to the certificate store on the remote machine.<\/p>\n\n\n\n<p>It also takes care to restart both mail services (dovecot \/ postfix) to load the updated certificate.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>automation schedule<\/strong><\/h2>\n\n\n\n<p>Not much to put in here. LetsEncrypt brings its own cron job with it:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>&gt; cat \/etc\/cron.d\/certbot\n\n# \/etc\/cron.d\/certbot: crontab entries for the certbot package\n#\n# Upstream recommends attempting renewal twice a day\n#\n# Eventually, this will be an opportunity to validate certificates\n# haven't been revoked, etc.  Renewal will only occur if expiration\n# is within 30 days.\n#\n# Important Note!  This cronjob will NOT be executed if you are\n# running systemd as your init system.  If you are running systemd,\n# the cronjob.timer function takes precedence over this cronjob.  For\n# more details, see the systemd.timer manpage, or use systemctl show\n# certbot.timer.\nSHELL=\/bin\/sh\nPATH=\/usr\/local\/sbin:\/usr\/local\/bin:\/sbin:\/bin:\/usr\/sbin:\/usr\/bin\n\n0 *\/12 * * * root test -x \/usr\/bin\/certbot -a \\! -d \/run\/systemd\/system &amp;&amp; perl -e 'sleep int(rand(43200))' &amp;&amp; certbot -q renew<\/code><\/pre>\n\n\n\n<h2 class=\"wp-block-heading\">post remarks<\/h2>\n\n\n\n<p>If You are doing any changes to apache config, the command prompt might return to &#8220;reload&#8221; apache. I highly recommend to &#8220;restart&#8221; apache to make sure the config is loaded:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>&#91;sudo] systemctl restart apache2<\/code><\/pre>\n\n\n\n<p>You could also read this article about my reverse-proxy:<br><a href=\"https:\/\/wordpress.familie-lahme.de\/index.php\/2024\/01\/09\/reverse-proxy-and-fail2ban\/\">https:\/\/wordpress.familie-lahme.de\/index.php\/2024\/01\/09\/reverse-proxy-and-fail2ban\/<\/a><\/p>\n\n\n\n<p><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Prerequisits Reading this you should Nevertheless if you do not know anything about the above, I encourage You to take this document to inspire yourself. &nbsp; Never stop learning! Abstract We are hosting multiple sites on one reverse-proxy-server and a remote IMAP server that uses SSL. The CertBot can be triggered using cron to update&#8230;<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[31],"tags":[42,56,58,55,45,43,57],"class_list":["post-683","post","type-post","status-publish","format-standard","hentry","category-computer","tag-apache","tag-certbot","tag-certificate","tag-letsencrypt","tag-linux","tag-reverse-proxy","tag-ssl"],"_links":{"self":[{"href":"https:\/\/wordpress.familie-lahme.de\/index.php\/wp-json\/wp\/v2\/posts\/683","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/wordpress.familie-lahme.de\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/wordpress.familie-lahme.de\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/wordpress.familie-lahme.de\/index.php\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/wordpress.familie-lahme.de\/index.php\/wp-json\/wp\/v2\/comments?post=683"}],"version-history":[{"count":14,"href":"https:\/\/wordpress.familie-lahme.de\/index.php\/wp-json\/wp\/v2\/posts\/683\/revisions"}],"predecessor-version":[{"id":699,"href":"https:\/\/wordpress.familie-lahme.de\/index.php\/wp-json\/wp\/v2\/posts\/683\/revisions\/699"}],"wp:attachment":[{"href":"https:\/\/wordpress.familie-lahme.de\/index.php\/wp-json\/wp\/v2\/media?parent=683"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/wordpress.familie-lahme.de\/index.php\/wp-json\/wp\/v2\/categories?post=683"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/wordpress.familie-lahme.de\/index.php\/wp-json\/wp\/v2\/tags?post=683"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}