Upgrading Stalwart from 0.15 to 016

The Setup

This is going to be a doozy of a post, but hopefully a wealth of information for someone out there. I think I may have ran into, literally, every issue that one could run into with the Stalwart 0.15 -> 0.16 upgrade path. I lost all my emails, went down reverse proxy rabbit holes, DKIM inconsistencies, sub-addressing changes, and just some strange behavior I was not expecting from various Stalwart features and logs. So stick around, this one will be good!

To get the stage set I woke up one morning and my Stalwart email server was down. I checked my logs and found the original welcome message like below:

════════════════════════════════════════════════════════════🔑 Stalwart bootstrap mode - temporary administrator account
   username: admin   password: XXXXXXXXXXXXXXXX
Use these credentials to complete the initial setup at the/admin web UI. Once setup is done, Stalwart will provision apermanent administrator and this temporary account will nolonger apply.════════════════════════════════════════════════════════════

After re-reading the docs I find that I made the stupid mistake of using the latest container version for my setup… Bad idea as it literally causes these issues that I just ran into. See the docs here:

But, easy enough fix. Rollback from backups, pin the v0.15 tag and look at the upgrade docs. Easy, little did I know this would turn into a 3 month process.

Preparing for the upgrade

So, being the cocky SysAdmin that I am I decided an in-place upgrade would be fine. So what if the documentation says you should run a test version first and play around. So what if it said it’s literally easier to do this because you can apply a snapshot of the changes to your new container during the prod upgrade.

Attempt #1

So I go to my favorite coffee shop after work, get a nice iced matcha latte and break out the upgrade instructions from Stalwart for the 0.16 upgrade.

So I back up my volumes pre-migration, I run the migration script they provide and get my export.json along with everything else. I mount my 2 new volumes, copy my files, change the permissions, point them to the new expected paths /etc/stalwart and /var/lib/stalwart. I bump the tag on the container to the v0.16 and hold my breath as I restart.

The container comes up in Recovery mode, I use the stalwart-cli to apply my settings, reboot it out of recovery mode, login with my temp admin account, and…. Everything is wiped. After re-reading the docs I realize they do a very bare-bones migration so besides my user account, domain, and DKIM keys pretty much everything is wiped. Reference my previous post about setting up my AWS relay as that was gone. My custom sub-addressing was gone, my proxy setup for my Nginx reverse proxy was gone, my app passwords were gone. Everything was gone!

At this point I realize I probably should’ve heeded the advice of the stalwart devs and actually took some time walking through this. I look at my watch and realize that I’m now 3 hours deep, the coffee shop closes in 15 minutes and I just barely completed the actual migration without actually getting anything done. This plus I’ve now had 3 hours of downtime for my email server, which isn’t ideal. I let out a sigh, and revert to the working v0.15 setup from my backup and pack things up, rethinking what I’m doing as this will likely turn into an all day upgrade instead of a quick few hours.

Attempt #2 (Mirrored Upgrade Path)

A couple months later. I sit down at noon on the 4th of July holiday weekend with a cup of coffee and take a deep breath as I get ready to dive into this thing. The v0.15 is no longer getting security upgrades and I really need to get this completed.

So, I actually follow the guidance and setup another container along side the current one. No downtime or breaking Production this time. So I do the steps above, a lot quicker this time, and end up with the same “broken” stalwart install that I did before. I can confirm that I login with my account, password, and TOTP. So that’s at least good that those settings made it. I see none of my app passwords are saved which sucks, but I figure I can just recreate and move on from that.

I continue with opening up the current v0.15 web ui next to the new 0.16 web ui and begin trying to copy and remember all of the settings I had before. I had to regenerate my AWS SES SMTP account because I did not save the settings from before. I got my routes and strategies setup. I add my proxy settings for my reverse proxy

Missing emails

But I run into my first problem, I have no emails. I have had over a year and close to 1000 archived emails from my life on this server, so I really need these emails to work!

I go down a couple rabbit holes to get out of this, I see in the docs:

Okay, easy. I struggle to find the “Recalculate disk quotas” task as that doesn’t exist. I eventually try “Tasks>Scheduled Tasks>Create Task” and then select Task Type: Perform account maintenance operations. and Maintenance type: Recalculate storage quota usage for the account. I set up the task, click on “Create” get the toast notification that the task was successfully created and… nothing happens. Not task was created in the “Scheduled Tasks” section. I try again, and green toast notification that it was created successfully, but no task. Weird, I pull up the logs, turn on debug and see that the task is submitted, but it’s never actually created. This is where I find my first bug in this new version. When you submit a task you have the option to submit a date and time. I filled out the date using the selector, but not the time and the form accepts it as a valid request. But nothing happens in the backend. Turns out, you actually need a time for the event to be created successfully, the field is a required form before submittal but since you can submit without a time it gets this weird condition where the task is accepted, but not created and notifies the user that the task was created successfully. So.

Tip #1: When submitting tasks, make sure to add a time as it’s a hard requirement.

After figuring out the time issue I think I have this. Run the task again with time this time and… the task does successfully run. But no emails still. My account shows that it has 500 Bytes of emails used. *Sigh* Time to look up alternatives.

So I go ahead and check the data volume. It’s around 200MB so my emails are still there. After some more googling and reading issues on the GitHub page I think the issue is my account ID and the ID associated with the emails are mis-mapped. I tried a few more things in the UI and just.. couldn’t get it to work. Luckly I have my emails backed up on multiple places. So.

Tip #2: Always keep multiple local copies of your emails. Whether it’s on your phone, desktop. laptop, etc.

I tried everything I could think of and eventually gave up and used the tool imapsync to copy my local emails to my account. The downside is that I am technically duplicating the blobs on the server since I will now have 2 copies of every email that I have. But I hope (and still haven’t confirmed) that by running the delete orphan blobs job on my server that it will remove the original ones that couldn’t be mapped to my original server.

Finalizing the Installation

So at this point I consider myself “good”. I have my emails, I can login, everything is looking good. Now it’s time to begin validation testing! So I roll up my sleeves and send some validation emails to my gmail and Protonmail accounts.

Duplicate DKIM Signatures

I am greeted with a Mailer Daemon error :)

554 Transaction failed: Duplicate header

Hmm.. duplicate headers? After looking through the headers of the emails it turns out that the only duplicate header is the DKIM-Signature one. But… that’s intentional right? I have a RSA and a Ed25519 key, so they should both be on there, right? Well it turns out that Stalwart changed this process from 0.15.X to 0.16.X where it will attached all DKIM signatures present on the domain. Where before it would just attach one. So, I tried deleting the Ed25519 key since RSA would still have the best legacy support. I try sending another test email after confirming I just have the RSA key present and… I get a 554 Transaction failed: Duplicate header. Sigh Okay, maybe it’s just another duplicate header I missed before. I check my email again and.. There’s 2 RSA DKIM keys… That’s weird, I can only have 1 on Stalwart. I check my SES Relay and sure enough I have Easy DKIM enabled on the SES side. So any outgoing mail would be signed my Stalwart and signed by SES since it’s relaying my emails. This leaves me with 2 choices, just use SES Easy DKIM or turn that off and configure Stalwart to only use RSA to ensure there’s 1 DKIM-Signature configured. Ultimately I decided, since SES is relaying all of my outgoing emails that I will just let it handle the DKIM signatures and delete the DKIM keys on my server.

Honestly, I did this from a maintenance point of view. SES makes it easy, so that’s what I typically do. I had to manually add the DKIM DNS entries for the domain, but that was only one time. Stalwart 0.16 has a cool feature that manages this for you, but I had to disable it. I may revisit this if I decide to change to a different SMTP relay due to security reasons. I’m not happy I keep ceding more control over my stuff to Amazon.. But at this point it’s easier and I just have to accept it.

Also, just to clarify this point too because I didn’t know about why this is a problem. Spoiler alert, it’s not a problem. The RFC-6376 fully supports multiple DKIM signatures for various reasons:

• The email went through multiple relays getting a signature from each one
• The sender is retiring an old DKIM key and it still uses the old and new keys for the time being

So at this point, you’re probably like me asking “Why is this not working when it’s supported”. Answer: Because Google and Protonmail (the only two I tested) don’t feel like supporting it! This setup is 100% valid it’s just that the receiving party has to feel like supporting it and from what I can tell most of the mainstream email providers just don’t feel like it… So, maybe one day. But so far a single RSA key and no way to rotate to a new one without completely nuking the old RSA key is the way.

Changing TLS Certificates for Stalwart services

I love Let’s Encrypt. I actually used a single certificate (maybe this is bad of me) for all of the services that I host. I use it for my SWAG (Nginx) reverse proxy setup. I also used it for Stalwart for JMAP, HTTPS, IMAPS, Submission, etc. It’s the same cert for everything!! It worked really well. I actually configured Stalwart to point to a file and then setup a shared volume between Stalwart and SWAG that worked brilliantly. But Stalwart in 0.16.X decided to run the service under UID 2000 instead of 1000. Because of this my permissions no longer made the SWAG/Stalwart shared certificate setup work. I tried trying to change GID to work, but eventually just ended up calling it and decided to setup the TLS certificate service on Stalwart instead. This involved changing some of my SWAG configurations. I had to remove all stream setups and just directly expose the email ports to the internet since Stalwart was handling TLS now instead of the proxy. So I removed all of those. Finally I have to tweak my site setup for my email domain. I still proxy JMAP and HTTP requests over SWAG in order to get my domain to work properly. SWAG owns port 443 so that needs to be open at all times and I use port 80 to renew SWAGs certificate so SWAG also owns port 80. But that was a simple change, just forward to the Stalwart host and expect HTTPS instead of just HTTP and also do not validate the SSL. Boom! Done!

As for Stalwart I had to look at my TLS options because SWAG owns port 443 and 80 so I could not use the normal HTTP verification for Stalwart and SWAG. Enter: ACME DNS-01! This solution actually works really well since I use Cloudflare to manage my DNS. Stalwart already has an API key for my Cloudflare, so it was really easy to setup DNS-01 authentication in addition to the other DNS record management. Click the box on the domain to also manage DNS-01 and a few tweaks on the TLS settings and I’m up and running with my TLS there. On to the next problem!

Subaddressing Formatting

So another thing they changed, I actually followed the instructions in 0.15.X for setting up my subaddressing because I actually like it this way. They change how the subaddressing works in the settings so I had to re-set it up because I realized that my subaddresses were not being delivered to me at all. The 0.15.X instructions recommended using .@ and I really liked that setup. I know the standard is +@, but I think that spammers can probably figure out that method since it’s the standard “plus” addressing that everyone uses. Mine is a little more unique where the username and the alias are swapped and I use a period instead of a plus. Dots are pretty common in usernames (think first.last@domain) so I think that makes it a lot harder for spammers to handle since they could be butchering valid usernames by splitting on the dot and getting a bunch of failed delivery bounce backs. But, 0.16.x does now support subdomains, but only in that plus format. My issue is I already have been rolling and have a ton of alias in the alias.user format so I can’t really easily change that. Instead I settle on using a Recipient Rewriting rule using the following Regex Rule:

if: matches('^([^.]+)\.([^.]+)@(.+)$', rcpt)then: $2 + '+' + $1 + '@' + $3else: false

The way the rule works is if is a period separating the email address in this format .. it will rewrite it to the expected “plus formatting” like +@. For example: [email protected] would be rewritten to [email protected] this matching the subaddressing rule that Stalwart is now expecting. It’s maybe a little dirty to deal with it, but at least this way my existing subaddressing setup works like it did before.

Trusted Proxy setup

So, once again I had issues with trusting my reverse proxy just like when I configured Stalwart before. They have a Trusted Proxy setting which tricks me every time. I enable it and sure enough my web interface breaks every. time. Because the Trusted Proxy expects the Proxy Protocol to be used which I can’t turn on. So every time I enable it I have to go back into the CLI and disable it because I also keep 8080 disabled, even internally, for security reasons. So I have to use the CLI to turn off Trusted Proxy and whitelist the proxy IP address so it doesn’t get blocked for spam (since all failures are associated with the proxy IP and not the remote IP). I finally get it configured with enabling the ``Obtain remote IP from Forwarded headerwhich fun fact: The logs don't show the actualremoteIPin the logs! So I spent a few hours trying to figure out why the remoteIP was still showing up as the proxy IP instead of the *actual* remote IP address. It turns out that the remoteIP *is* the *actual* remote IP but unless you have verbose logging enabled on Stalwart it will only show the *first* remoteIP which is the proxy and not the *second* remoteIP which is the *actual* remote IP... I know it is confusing.. Anyway, after doing this my IP bans work, and theX-Forwarded-For` headers to work they just don’t show up in the logs..

CalDAV and CardDAV Notes

This is the final, major, issue that I had for setting up Stalwart 0.16.X. I decided to migrate all my stuff from Baïkal to Stalwart since it seems to be much more supported and I don’t have to host a second, now redundant, service to do this capability. Moving CalDAV and CardDav is actually really easy luckily. Just download a .ics or .card file with all of your contacts and an .ics for each calendar you have and you can import them in the new service. So, I went in pulled the files I needed from Baïkal and shut it down. After doing so I setup Stalwart to have have the Calendars and Contact books that I needed to move. I added them to DAVx⁵ and imported the .ics and .card files successfully. I was actually super excited because this meant I could start to send and receive email acceptances with Stalwart, where as Baïkal had no email integrations to send to participants.

So I, excitedly, added my Protonmail account to an invite and… got nothing. Strange.. I have it setup properly, I think. I try again with gmail and also see nothing. It’s here I proceed to absolutely fill my calendar with a bunch of Junk calendar invites trying to test and figure out why this isn’t working. I try from my Evolution client on my Desktop and.. the verification email sends! So why is my phone not working properly? I go in and check the now verbose logs and see that the new event is created, but no email is set to send to the participants. So.. Does it work for my desktop and not for my phone? Well I tried using different calendars on my phone, but still no luck. After a lot of troubleshooting it turns out that DAVx⁵ needs your full email address as the account name because it uses this as the ORGANIZER field and if it doesn’t exactly match your email the ORGANIZER field won’t trigger iMIP email to be sent properly. Evolution and other desktop clients use this, but DAVx⁵ does not use the account username as the ORGANIZER name but instead uses the Account name for some reason… Anyway, I had to rename the account name and I got the outgoing calendar invites to work properly!

Summary of Upgrade Issues

So the TL;DR for this is: just about everything that could’ve gone wrong for me did haha.

• I initially wiped my settings and DB trying to upgrade the first time (Always backup before an upgrade!).
• After getting the upgrade to work successfully I lost access to all my emails.
• After getting all my emails I couldn’t send any because of duplicate DKIM Signatures that used to work, but changed with 0.16.
• My reverse proxy IP would be banned for spamming because I couldn’t determine if remoteIPs were configured correctly and incorrectly using Trusted Proxy and bricking the Web UI.
• I couldn’t receive emails from my subaddresses because I use a custom format that’s no longer supported out of the box (or at least no longer documented in their documentation).
• My TLS Certificate setup is no longer supported with the new 0.16.X and I have to use Stalwart ACME to manage them instead of Let’s Encrypt with worked in the past fine for months.

But after dealing with ALL of those issues I now have a stable, working, and honestly more usable Stalwart email server than before. The online UI pieces that are slowly being added are a nice to have. I’m considering purchasing an Enterprise license for the features behind the paywall due to the added functionalities behind the “paywall”. I also have the new iMIP email notifications and outgoing and incoming emails for my events which I also did not have. I look forward to continued Stalwart upgrades and hope they aren’t as painful as this one. I expected to complete this upgrade in a few hours, maybe a day. But this turned into a multi-day troubleshooting session that would just come up as I would try to use a new feature and run into issues.