msmtp and AppArmor: investigating a curious access issue

Background

I am currently refactoring all of my email configuration for command line usage. My current stack currently uses the following software:

• mutt to read emails;
• neovim to write them (called by mutt);
• isync (mbsync) to synchronize them; and
• msmtp to send them.

While moving reading man pages, I have noticed that msmtp(1) specifies that msmtp supports the $XDG_CONFIG_HOME/msmtp/config configuration file path, in addition to ~/.msmtprc that I have been using.

The $XDG_CONFIG_HOME usually expands to ~/.config/ directory. It is a standard used by a lot of programs to store their configuration files instead of putting them in arbitrary files directly in the $HOME directory.

I always used ~/.msmtprc before, but regrouping files into $XDG_CONFIG_HOME is appealing. It permits to have a cleaner $HOME directory structure. This is nice to have when you accidentally list all files in there.

One important fact to note is that I use GNU Stow in order to create symlinks for managing my dotfiles. All of them are regrouped per-software in my ~/dotfiles git repository.

Here is a non-exhaustive tree of this directory:

$ tree -a ~/dotfiles
/home/user/dotfiles
├── mutt
│   ├── .muttrc
│   ├── .msmtprc
│   ├── .mutt
│   │   ├── common.rc
│   │   ├── ...

When I am on a system that I want to configure to use mutt (and all the related software like msmtp), I just run the following commands to symlink all the configuration files into my home directory:

$ cd dotfiles/
$ stow mutt
$ ls -l ~/.muttrc
... /home/user/.muttrc -> dotfiles/mutt/.muttrc
$ ls -l ~/.msmtprc
... /home/user/.msmtprc -> dotfiles/mutt/.msmtprc

Moving the msmtp configuration

This configuration file is moved so that it uses the $XDG_CONFIG_HOME path:

$ cd ~/dotfiles
$ mkdir -p mutt/.config/msmtp/
$ git mv mutt/.msmtprc mutt/.config/msmtp/config
$ rm ~/.msmtprc
$ stow mutt
$ ls -ld ~/.config/msmtp
... /home/user/.config/msmtp -> ../dotfiles/mutt/.config/msmtp

After moving this file, msmtp is not able to read its configuration file anymore. This is the message displayed when running the following example command to send a test email:

$ echo "hello, world" | msmtp -t example@example.org
msmtp: account default not found: no configuration file available

In order to verify that msmtp is trying to open the ~/.config/msmtp/config file like the manual page states, let’s use strace to see which files are accessed:

$ echo -e "foo" | strace msmtp -t example@example.org
...

openat(AT_FDCWD, "/etc/msmtprc", O_RDONLY)
= -1 (No such file or directory)

stat("/home/user/.msmtprc", 0x...)
= -1 ENOENT (No such file or directory)

stat("/home/user/.config/msmtp/config",
     {st_mode=S_IFREG|0644, st_size=551, ...})
= 0

geteuid()
= 1000

openat(AT_FDCWD, "/home/user/.config/msmtp/config", O_RDONLY)
= -1 EACCES (Permission denied)

We can see that msmtp is trying different configuration files. It can see our configuration file (stat), but it fails when trying to open it. Then it finally prints the error message and exits with non-zero error.

The mode of the file detected by stat is 644 so there is no reason msmtp should not be able to open the file… except if something else is preventing it to open it!

Investigating AppArmor rules

And this is when you have to consider the rest of my system. I am using Debian testing, which is preparing for the bullseye release. But since buster release, AppArmor is activated by default.

AppArmor is a software that confines programs according to a set of rules. So this would certainly explain why msmtp cannot access the configuration file, while I can access it with any other text editor.

By looking into the rules directory, we can indeed see that there is a set of rules activated:

$ ls -lh /etc/apparmor.d/usr.bin.msmtp
-rw-r--r-- 1 root root 1,5K 16 Dec  14:37 /etc/apparmor.d/usr.bin.msmtp

Here is its (edited) content:

$ cat /etc/apparmor.d/usr.bin.msmtp
...

profile msmtp /usr/bin/msmtp flags=(attach_disconnected) {
  ...

  /usr/bin/msmtp          mr,
  /etc/aliases            r,
  /etc/msmtprc            r,
  /etc/mailname           r,
  /etc/netrc              r,
  owner @{HOME}/.msmtp*   r,
  owner @{HOME}/.netrc    r,
  owner @{HOME}/.tls-crls r,

  owner @{HOME}/.msmtp*.log wk,
  /var/log/msmtp            wk,

  owner @{HOME}/**/*msmtprc        r,
  owner @{HOME}/.config/msmtp/*    r,
  owner @{HOME}/.cache/msmtp/*     r,
  owner @{HOME}/.cache/msmtp/*.log wk,

  ...

  #include <local/usr.bin.msmtp>
}

We can see here the list of files that /usr/bin/msmtp program is allowed to access on the system. The files we are interested in are the configuration files in the home directory of the user, represented by @{HOME}.

We can see that the @{HOME}/**/*msmtprc rule matches the previous path that we were using (via the symlink): ~/dotfiles/mutt/.msmtprc.

However, the rule using the XDG_CONFIG_HOME is not using a ** directory wildcard. This means that the program can read the file in ~/.config, but only if it is a plain file. If it is a symlink redirecting to somewhere else in the home directory, this rule prevents msmtp from reading the link.

The solution seems simple: introduce a directory wildcard for the ~/.config path as well. Here is the modification that I applied onto this file as root:

$ diff -u /etc/apparmor.d/usr.bin.msmtp.orig /etc/apparmor.d/usr.bin.msmtp
--- /etc/apparmor.d/usr.bin.msmtp.orig      2020-12-16 13:25:37.618559611 +0100
+++ /etc/apparmor.d/usr.bin.msmtp   2020-12-22 16:14:55.067110824 +0100
@@ -23,7 +23,7 @@
   /var/log/msmtp            wk,

   owner @{HOME}/**/*msmtprc        r,
-  owner @{HOME}/.config/msmtp/*    r,
+  owner @{HOME}/**/.config/msmtp/* r,
   owner @{HOME}/.cache/msmtp/*     r,
   owner @{HOME}/.cache/msmtp/*.log wk,

We need to restart AppArmor in order for the changes to be applied:

$ sudo apparmor_parser -R /etc/apparmor.d/usr.bin.msmtp

And now, if we try to call msmtp again with our symlinked configuration file, everything works as expected:

$ echo -e "foo" | msmtp -t example@example.org

(msmtp will fail because of the example address, but works with a real one)

Fixing the bug in Debian

So, now that we have identified the problem, it would be nice to fix it so that other people do not have to conduct the same kind of investigation (although it was fun to do!).

First, we need to find out which package is installing these rules for msmtp:

$ apt-file search /etc/apparmor.d/usr.bin.msmtp
msmtp: /etc/apparmor.d/usr.bin.msmtp

We can see that the rules are provided by the msmtp package itself. We could directly file a bug using the reportbug tool, but let’s be nice and first prepare a patch for the source package so that the maintainer can merge the fix directly.

Let's download the source package of msmtp:

$ mkdir -p ~/tmp/msmtp
$ cd ~/tmp/msmtp
$ apt source msmtp
Reading package lists... Done
NOTICE: 'msmtp' packaging is maintained in the 'Git' version
control system at:
https://salsa.debian.org/kolter/msmtp.git
Please use:
git clone https://salsa.debian.org/kolter/msmtp.git
to retrieve the latest (possibly unreleased) updates to the package.
Need to get 377 kB of source archives.
...

We see an interesting notice that tells us that this package is maintained via Git. We should probably clone the repository instead of using the source package, since the issue may have already been fixed:

$ git clone https://salsa.debian.org/kolter/msmtp.git
...
$ cd msmtp
$ find . -name 'usr.bin.msmtp'
./debian/apparmor/usr.bin.msmtp
$ diff -u /etc/apparmor.d/usr.bin.msmtp.orig ./debian/apparmor/usr.bin.msmtp

Using diff, we can see that there is no difference between the file installed on our system and the latest rules. So we can just replace this file with the one that we have patched, and create a commit out of it:

$ cp /etc/apparmor.d/usr.bin.msmtp ./debian/apparmor/usr.bin.msmtp
$ git commit ./debian/apparmor/usr.bin.msmtp
...
$ git format-patch -1

From here we can create a bug for the package with reportbug and attach this patch as a proposed solution:

$ reportbug --mutt msmtp
...
12 bug reports found:

Bugs with severity important
   1) #917260  msmtp: hangs after message termination [<CRLF>.<CRLF>]

Bugs with severity normal
   2) #683892  msmtp attempts to connect to gnome-keyring
   3) #933551  msmtp can't write logs to an encfs file
   4) #942457  msmtp: Running msmtp as a user and Apparmor
   5) #944188  /etc/msmtprc password disclosure
   6) #945024  msmtp: Doesn't generate a Message-Id
   7) #969198  msmtp-mta package should use update-alternatives for symlinking to msmtp
   8) #975333  msmtp: AppArmor profile breaks --file, logfile, and passwordeval

Bugs with severity minor
   9) #487555  Shouldn't complain about permissions unless .msmtprc contains passwords

Bugs with severity wishlist
  10) #569122  $HOME/.msmtprc is incompatible with vixie cron.
  11) #810727  msmtp: should be able to pull hostname from server greeting (for gssapi)
  12) #834167  Should allow pinning server key
(1-12/12) Is the bug you found listed above [y|N|b|m|r|q|s|f|e|?]?

We can see that there are already two bugs regarding msmtp and apparmor.

In #975333, it seems to be agreed that the AppArmor rules are breaking many workflows, and that users should be able to easily disable them for msmtp. This is currently being worked on by the maintainer, if I understood the discussion correctly.

In #942457, there are some complaints about how using symlinks and dotfiles does not always work. Some people resort to copy the file instead of linking to it. This looks like a bug to attach our patch to.

I have sent a new answer to this bug report, confirming that this bug is still occurring. The patch has also been proposed for the current package maintainer to fix the issue described in this article.

Conclusion

msmtp is a nice piece of software. It allows me to send mails via mutt or the command line easily. However the AppArmor rules proposed in the Debian package are a little bit restrictive. They should be improved (the intent of this article), or easily disabled while we are still figuring this out.

Happy mailing!