Explain how sieve can work instead of maildrop

[Infern1]

Current situation

Maildrop moves messages coming from foo@mail.com to the folder Foo

Desired situation

Migrate those rules to Sieve, so they can be easily managed in roundcube

How can the same achieved by Sieve from Dovecot?

I understand incoming mail flow as:
Haraka -> qmail-smtpd -> Maildrop -> vpopmail deliver to Maildir => user reads mail with IMAP

I don't see where Sieve with lmtp replaces maildrop

[msimerson]

I understand incoming mail flow as:
Haraka -> qmail-smtpd -> vdelivermail -> maildrop -> deliver to Maildir => user reads mail with IMAP

In 2017 I updated the qmail-deliverabled plugin and Haraka to support this delivery path:

Haraka -> dovecot (LMTP) -> Maildir

It works for standard email accounts and it knows the difference by using the qmail-deliverabled plugin. In that scenario, dovecot is the LDA and sieve scripts can act on messages the same as maildrop in the former. I've been meaning to also update the Haraka aliases plugin so that aliases and forwards also get handled in Haraka.

[Infern1]

So the answer is, if I want to use LMTP with Haraka I should manually change qmail-deliverable.ini the default provision script won't do this for me.

Any example would be good, with some pros and cons, so people can decide

[msimerson]

Correct, the underlying assumption of the provision scripts is that your configs are holy and sacred and should not be trampled upon. Here's how I tell Haraka to deliver via LMTP:

# less qmail-deliverable.ini 
check_outbound=true
host=172.16.15.8
queue=smtp_forward
next_hop=lmtp://172.16.15.15

The key is the next_hop setting. QMD returns different codes for Maildir, email lists, etc. Based on that, if the email address is a Maildir, then Haraka delivers it directly to Dovecot LDA.

Both roundcube and snappymail have plugins for editing sieve scripts. You do have to set up sieve though in dovecot. Here are some relevant bits from my dovecot local.conf file:

protocols = imap pop3 lmtp sieve submission

protocol lmtp {
  mail_fsync = optimized
  mail_plugins = $mail_plugins sieve
}

service lmtp {
  user = vpopmail
  inet_listener lmtp {
    port = 24
  }

  unix_listener lmtp {
    #mode = 0666
  }
}

service managesieve-login {
  inet_listener sieve {
    port = 4190
  }
}

[Infern1]

Correct, the underlying assumption of the provision scripts is that your configs are holy and sacred and should not be trampled upon.

That's what I like about the provision script, however sometimes with upgrades I might miss new functionality.
I don't mind editing by hand, however it would if people are aware of this new optional stuff.

Here's how I tell Haraka to deliver via LMTP:

# less qmail-deliverable.ini 
check_outbound=true
host=172.16.15.8
queue=smtp_forward
next_hop=lmtp://172.16.15.15

The key is the next_hop setting. QMD returns different codes for Maildir, email lists, etc. Based on that, if the email address is a Maildir, then Haraka delivers it directly to Dovecot LDA.

That is straightforward, any reason not to make it default?

Both roundcube and snappymail have plugins for editing sieve scripts. You do have to set up sieve though in dovecot. Here are some relevant bits from my dovecot local.conf file:

I think this works out of the box

[msimerson]

That is straightforward, any reason not to make it default?

Probably not. I like giving changes some time to "burn in" and surface edge cases. I see that I enabled that a few years ago and haven't had any issues.

I think this works out of the box

Yep, it does appear I added that quite a long time ago as well.

[Infern1]

Yes, this change makes Sieve Filter work!

So default /usr/local/etc/mail/mailfilter is not used when using lmtp ? (Which is not really a problem for me)

.qmail forward are processed as normal I assume

[Infern1]

Did you test Sieve filter with forward to a different address?

2023-05-25T14:54:31.723Z [NOTICE] [2DF21179-C2E3-468B-BB41-C80685909F39.1] [core] sender <rob.lensen@foobar.com> code=CONT msg=""
2023-05-25T14:54:31.729Z [NOTICE] [2DF21179-C2E3-468B-BB41-C80685909F39.1] [core] recipient <somefamilly@yahoo.com> code=OK msg="not local" sender=rob.lensen@foobar.com
2023-05-25T14:54:31.731Z [INFO] [2DF21179-C2E3-468B-BB41-C80685909F39.1] [core] client half closed connection ip=172.16.15.15
2023-05-25T14:54:31.733Z [NOTICE] [2DF21179-C2E3-468B-BB41-C80685909F39.1] [core] disconnect ip=172.16.15.15 rdns=dovecot helo=dovecot relay=N early=N esmtp=N tls=N pipe=N errors=0 txns=1 rcpts=0/0/1 msgs=0/0/0 bytes=0 lr="550 I cannot deliver mail for <somefamilly@yahoo.com>" time=0.062
2023-05-25T14:54:31.736Z [NOTICE] [CD6D503D-8C40-48DA-9813-28CBA06A70C6.1.1] [outbound] recipient <simone@bla.nu> deferred: 451 4.2.0 <simone@bla.nu> Execution of Sieve filters was aborted due to temporary failure
2023-05-25T14:54:31.737Z [INFO] [CD6D503D-8C40-48DA-9813-28CBA06A70C6.1.1] [outbound] Temp failing 1685026471525_1685026471525_0_91990_IgoAyW_1_haraka for 64 seconds: Some recipients temp failed: <simone@bla.nu>

Fixed by adding to local.conf

# less local.conf |grep sub
submission_host = haraka:25

[msimerson]

Hmm, I have a similar addition. I should look into what the difference is.

/data/dovecot/etc # grep sub local.conf 
protocols = imap pop3 lmtp sieve submission
service submission {
submission_relay_host = haraka
submission_relay_ssl = no

[Infern1]

Hmm, I have a similar addition. I should look into what the difference is.

/data/dovecot/etc # grep sub local.conf 
protocols = imap pop3 lmtp sieve submission
service submission {
submission_relay_host = haraka
submission_relay_ssl = no

I think it is exactly the same functionality, however not coming from the provision scripts

[Infern1]

Hmm, I have a similar addition. I should look into what the difference is.

/data/dovecot/etc # grep sub local.conf 
protocols = imap pop3 lmtp sieve submission
service submission {
submission_relay_host = haraka
submission_relay_ssl = no

can you post whole your local.conf cannot get forwards to work correctly or does haraka also need a change?

sieve: info: started log at 2023-06-23 14:16:52 +0200.
error: msgid=<CAEgN0qL0MaS1WY_LeNQuktkyK_LaZiLjSmjZbMeKvnhuX0JS3A@mail.gmail.com>: redirect action: failed to redirect message to <r .XX@xx.com>: Sendmail program returned error (temporary failure).

2023-06-23T12:16:52.209Z [INFO] [BF87DC59-22F7-472F-A079-466AD19EE03C.1.1] [outbound]  hook=get_mx plugin=qmail-deliverable function=hook_get_mx params=bsdfreaks.nl retval=OK msg="[object Object]"
2023-06-23T12:16:52.233Z [INFO] [BF87DC59-22F7-472F-A079-466AD19EE03C.1.1] [outbound] secured verified=false cipher=TLS_AES_256_GCM_SHA384 version=TLSv1.3 error=ERR_TLS_CERT_ALTNAME_INVALID cn=bsdfreaks.nl organization="" issuer="Let's Encrypt" expires="Aug  9 21:53:51 2023 GMT" fingerprint=AE:09:22:FA:E2:3E:AB:69:17:D6:9E:4D:E0:85:13:1B:07:07:BC:72
2023-06-23T12:16:52.339Z [NOTICE] [2AB0B96F-C1BF-4A52-B360-8F5FD0D53E89] [core] connect ip=172.16.15.15 port=30969 local_ip=172.16.15.9 local_port=25
2023-06-23T12:16:52.345Z [INFO] [2AB0B96F-C1BF-4A52-B360-8F5FD0D53E89] [helo.checks] skip:proto_mismatch(private)
2023-06-23T12:16:52.424Z [NOTICE] [2AB0B96F-C1BF-4A52-B360-8F5FD0D53E89.1] [core] sender <r.j.xx@gmail.com> code=CONT msg=""
2023-06-23T12:16:52.429Z [NOTICE] [2AB0B96F-C1BF-4A52-B360-8F5FD0D53E89.1] [core] recipient <r.XX@xx.com> code=OK msg="not local" sender=r.j.xx@gmail.com
2023-06-23T12:16:52.431Z [INFO] [2AB0B96F-C1BF-4A52-B360-8F5FD0D53E89.1] [core] client half closed connection ip=172.16.15.15
2023-06-23T12:16:52.431Z [NOTICE] [2AB0B96F-C1BF-4A52-B360-8F5FD0D53E89.1] [core] disconnect ip=172.16.15.15 rdns=dovecot helo=dovecot relay=N early=N esmtp=N tls=N pipe=N errors=0 txns=1 rcpts=0/0/1 msgs=0/0/0 bytes=0 lr="550 I cannot deliver mail for <rob.lensen@linde.com>" time=0.094
2023-06-23T12:16:52.434Z [NOTICE] [BF87DC59-22F7-472F-A079-466AD19EE03C.1.1] [outbound] recipient <rob@bsdfreaks.nl> deferred: 451 4.2.0 <rob@bsdfreaks.nl> Execution of Sieve filters was aborted due to temporary failure
2023-06-23T12:16:52.434Z [INFO] [BF87DC59-22F7-472F-A079-466AD19EE03C.1.1] [outbound] Temp failing 1687522547321_1687522611581_1_2723_bi9p7d_204_haraka for 128 seconds: Some recipients temp failed: <rob@bsdfreaks.nl>

[msimerson]

Forwards won't yet work via LMTP. QMD only designates vpopmail directories as LMTP eligible, everything else is delivered in the usual way. I did recently fix a couple bugs related to this in Haraka and smtp_forward plugins.

https://gist.github.com/msimerson/e0b5f92582c1a9046ef5fdf27ff200c8

[msimerson]

I've been poking at this some more.

When Haraka delivers to a dovecot Maildir via LMTP, it bypasses qmail, maildrop, and vpopmail. This is desirable if you're moving to sieve as it completely bypasses maildrop and issue #534. It may be surprising to learn that email forwards and vacations defined in qmailadmin no longer work. The mechanism on the backend that implements that is qmail-local delivery rules. If a user has logged into qmailadmin and set up a forward, that forward works by adding an entry such as &user@example.com to the .qmail in their virtual mail directory. If the mailbox is forwarded with 'Save a Copy' checked, then the .qmail file will have two lines. With LMTP delivery to dovecot, both are ignored.

Dovecot LDA can replicate the functionality we've been using maildrop for. The quota plugin replicates the deliverquota program from maildrop. Sieve can implement mail filtering, forwards, and vacation autoreplies.

[tniethammer]

Hi toghether have there been some progress? It looks sieve filtering is working. But I have an issue with vacation messages to External hosts. I thing its due dma and hostname haraka which is wrong for external hosts.

Dec 12 22:49:52 dovecot dma[101a6][82493]: new mail from user=vpopmail uid=89 envelope_from=dovecot@mx40.projektb.net
Dec 12 22:49:52 dovecot dma[101a6][82493]: mail to=tobias.niethammer@gmx.net queued as 101a6.276b4261b000
Dec 12 22:49:52 dovecot dma[101a6.276b4261b000][82494]: tobias.niethammer@gmx.net trying delivery
Dec 12 22:49:52 dovecot dma[101a6.276b4261b000][82494]: using smarthost (haraka:25)
Dec 12 22:49:52 dovecot dma[101a6.276b4261b000][82494]: trying remote delivery to haraka [172.16.16.9] pref 0
Dec 12 22:49:54 dovecot dma[101a6.276b4261b000][82494]: remote delivery to haraka [172.16.16.9] failed after RCPT TO: 550 [CFB63D57-B89@haraka] I cannot deliver mail for tobias.niethammer@gmx.net
Dec 12 22:49:54 dovecot dma[101a6.276b4261b000][82494]: delivery failed, bouncing as 10177
Dec 12 22:49:54 dovecot dma[10177]: new mail from user=vpopmail uid=89 envelope_from=<>
Dec 12 22:49:54 dovecot dma[10177]: mail to=dovecot@mx40.projektb.net queued as 10177.276b4261b0a0
Dec 12 22:49:54 dovecot dma[10177.276b4261b0a0][82494]: dovecot@mx40.projektb.net trying delivery
Dec 12 22:49:54 dovecot dma[10177.276b4261b0a0][82494]: using smarthost (haraka:25)
Dec 12 22:49:54 dovecot dma[10177.276b4261b0a0][82494]: trying remote delivery to haraka [172.16.16.9] pref 0
Dec 12 22:49:59 dovecot dma[10177.276b4261b0a0][82494]: remote delivery to haraka [172.16.16.9] failed after RCPT TO: 550 [6169DF76-F4B@haraka] I cannot deliver mail for dovecot@mx40.projektb.net
Dec 12 22:49:59 dovecot dma[10177.276b4261b0a0][82494]: can not bounce a bounce message, discarding

any suggestions to go forward ?

[tniethammer]

hm well it looks like that the ^M are wrong and need to be removed, any idear how to do this? and why ist this not correct?
when I open the file on windows with notepad++ it shows the correct line endings CRLF

I'm just an bit confused ...
if i show with hexdump I see also at the end 0d 0a 0d 0a
so should be fine ?

If I change the dma spool file to the endings: 0d 0a 0a
it works, but there are some Header informations inside the body

X-Spam-Checker-Version: SpamAssassin 4.0.1 (2024-03-26) on spamassassin
X-Spam-Status: No, score=0.0 required=5.0 tests=SPF_PASS,UNPARSEABLE_RELAY
autolearn=no autolearn_force=no version=4.0.1

[msimerson]

Yep, the ^M characters were the clue I saw that something was amiss. I know that Notepad won't save with CRLF line endings, I don't know about Notepad++. You may need to use an alternative editor (like VS Code) that provides the option. Or run the file through a string editor like sed to replace the ^M (CR) line endings with \r.

[tniethammer]

Hi Matt,
I do not have any progress ... ist that sieve vacation thing working at your toaster?
the sieve file is generated with roundcube and has the right line endings if the line endings are wrong it gets an error anyway there is also no possiblility to change the line endings per parameter.

is it maybe an problem with dma ? the mail message is this:

Received: from vpopmail (uid 89)
(envelope-from dovecot@mx40.projektb.net)
id 101ba
by mx40.projektb.net (DragonFly Mail Agent v0.11+ on dovecot);
Wed, 18 Dec 2024 21:42:07 +0100
X-Sieve: Pigeonhole Sieve 0.5.21.1 (49005e73)^M
Message-ID: dovecot-sieve-1734554527-21447-0@dovecot^M
Date: Wed, 18 Dec 2024 21:42:07 +0100^M
From: test02@projektb-mail.net^M
To: tobias.niethammer@gmx.net^M
Subject: Out of Office^M
In-Reply-To: 048501db518d$4b4df380$e1e9da80$@gmx.net^M
References: 048501db518d$4b4df380$e1e9da80$@gmx.net^M
Auto-Submitted: auto-replied (vacation)^M
Precedence: bulk^M
X-Auto-Response-Suppress: All^M
MIME-Version: 1.0^M
Content-Type: text/plain; charset=utf-8^M
Content-Transfer-Encoding: 8bit^M
^M
I am currently out of the office and will not be able to respond to your email.^M
^M
I will get back to you as soon as possible upon my return.^M
^M
Best regards,^M
Your Name^M

here in notepad++ it shows the line endings

if you have an working setup how have you configured dovecot? also via dma?

[msimerson]

I haven't tried setting up vacation. It's not a feature I encourage anyone to use.

[tniethammer]

ok I undestand you, but customers want to have a posibility to set an message for any reasons...

I opened an ticket at dovecot, but they think its an mta issue i did a lot more testing https://dovecot.org/mailman3/archives/list/dovecot@dovecot.org/thread/FAUUAMCIK7VQC4TO72XDEBAGCOL3OXYA/

If i Change the following DMA spool file from:
###########################

Received: from vpopmail (uid 89)
(envelope-from dovecot@mx40.projektb.net)
id 1008d
by mx40.projektb.net (DragonFly Mail Agent v0.11+ on dovecot);
Fri, 20 Dec 2024 20:27:42 +0100
X-Sieve: Pigeonhole Sieve 0.5.21.1 (49005e73)^M
Message-ID: dovecot-sieve-1734722862-691938-0@dovecot^M
Date: Fri, 20 Dec 2024 20:27:42 +0100^M
From: test02@projektb-mail.net^M
To: tobias.niethammer@gmx.net^M
Subject: test Urlaub^M
In-Reply-To: 01f701db5315$3b6f4c50$b24de4f0$@gmx.net^M
References: 01f701db5315$3b6f4c50$b24de4f0$@gmx.net^M
Auto-Submitted: auto-replied (vacation)^M
Precedence: bulk^M
X-Auto-Response-Suppress: All^M
MIME-Version: 1.0^M
Content-Type: text/plain; charset=utf-8^M
Content-Transfer-Encoding: 8bit^M
^M
test 1^M
test 2^M
test 3^M
test4^M
^M
#########################################################

to
#########################################################

Received: from vpopmail (uid 89)
(envelope-from dovecot@mx40.projektb.net)
id 1008d
by mx40.projektb.net (DragonFly Mail Agent v0.11+ on dovecot);
Fri, 20 Dec 2024 20:27:42 +0100
X-Sieve: Pigeonhole Sieve 0.5.21.1 (49005e73)^M
Message-ID: dovecot-sieve-1734722862-691938-0@dovecot^M
Date: Fri, 20 Dec 2024 20:27:42 +0100^M
From: test02@projektb-mail.net^M
To: tobias.niethammer@gmx.net^M
Subject: test Urlaub^M
In-Reply-To: 01f701db5315$3b6f4c50$b24de4f0$@gmx.net^M
References: 01f701db5315$3b6f4c50$b24de4f0$@gmx.net^M
Auto-Submitted: auto-replied (vacation)^M
Precedence: bulk^M
X-Auto-Response-Suppress: All^M
MIME-Version: 1.0^M
Content-Type: text/plain; charset=utf-8^M
Content-Transfer-Encoding: 8bit^M

test 1^M
test 2^M
test 3^M
test4^M
^M
#########################################################

I removed the CRLF after the Content-Transfer-Encoding line with an LF only than the mail gets correct processed and transfered. so from Point of Dovecot CRLF is right, could it be that there is sth wrong with Haraka to Accept there CRLF? I also have seen your pull on Haraka haraka/Haraka#3287 could it be related to this?

if I use the dovecot sieve forward this works without issue but there is then also an LF after the last header line