Port25 PowerMTA bounce handler for Interspire and MailWizz
:beer: Please support me: Although all my software is free, it is always appreciated if you can support my efforts on Github with a contribution via Paypal - this allows me to write cool projects like this in my personal time and hopefully help you or your business.
I receive frequent requests to provide consulting- or installation-services for Port25, the scripts etc. While I appreciate your donations to keep my contributions going, I can unfortunately not assist you with fixing your configuration / installation.
The majority of issues I receive are based on:
The scripts are open-source and free. There is nothing more annoying when I receive emails / issues from your angry customers that they paid ridiculous amounts of money for bounce-handling and my scripts are not working. If you resell my scripts and then change license information / remove any attribution to my work you are violating the license of this project.
If you resell my scripts, keep copyright intact and then perhaps also extend the decency to provide a Paypal contribution. I am more than happy to fix bugs and make improvements, but will not do so on the back of your commercial ventures.
Transactional- and promotional mail to our customers is an important mechanism to stay in touch and hopefully improve revenue due to the value and information we provide in our communication with customers.
But let's be honest: Bounce handling and keeping your mail-lists is tough. Free solutions do not seem to exist and professional solutions/services are expensive. I have never really understood this, as bounce-handling is generally quite easy:
Once our MTA (mail transfer agent) such as PowerMTA records a hard-bounce we need to remove the recipient email from our system as it not just wastes resources attempting to to send mail to a non-existant email, it will eventually hurt sender reputation.
The big challenge comes in that campaign tools (especially when hosted in-house) either require "dedicated bounce email accounts" or require software development skills to write code to integrate with the campaign APIs to manage unsubscribes.
This Port25 PowerMTA bounce handler addresses this as it can run standalone (you provide a CSV) or it can integrate with PowerMTA acct-file
pipe feature and unsubscribes via API calls to MailWizz, Interspire and any other provider in near-realtime.
:warning: I run OS X and Linux and I have not tested any of it on Windows. I suppose that Windows will play nice with PHP, but if you find any issues, please log it and I will fix with your help.
The installation is quite simple:
bouncehandler
under /opt/pmta/
so that Port25 can execute it.All configuration is controlled within setup.php
and the following options can be adjusted:
DEFAULT OPTIONS:
1
to log to the console. If you leave it at 0
it will log to /var/log/pmta/pmta-bounce-handler.log
(or if the file can not be written, it will create pmta-bounce-handler.log
in the directory where the script is run)$bounceCategories = array("bad-mailbox","bad-domain","routing-errors","inactive-mailbox");
. If the array is empty, all records are processed as bounces. CAREFUL*: PowerMTA also records transient/soft-bounces and you would inadvertently unsubscribe users due to temporary delivery errors (i.e. mailbox full).$softbounceCategories = array("bad-configuration", "bad-connection", "content-related", "invalid-sender", "other", "policy-related", "quota-issues", "spam-related", "virus-related");
. If the array is empty, all bounces are classified as hard-bounces.INTERSPIRE CONFIGURATION: If you do not use Interspire, leave constants as undefined:
xml.php
, typically http://www.example.com/xml.php
admin
)MAILWIZZ CONFIGURATION If you do not use MailWizz, leave constants as undefined:
http://www.example.com/api
For me a mail-bounce means that we will never, ever be able to deliver to the recipient again for various reasons (mailbox does not exist, domain does not exist etc). There are many cases where free-mail (such as Nokia Mail) cease to exist and in such cases the recipient should be unsubscribed from ALL CONTACT LISTS. Email marketers freak out about such "bold" move, but to be honest if an email bounces in one contact list, it is pointless to hold on to that subscription in other contact lists.
My approach is that if a delivery permanently fails, we unsubscribe the recipient from all systems. On our transactional enviroment (where users continue to log in with their non-functioning email-address/userid) we show the user a notice that we had delivery issues and ask the user to reconfirm his email. This is an elegant and efficient way to reconfirm email details.
Interspire allows the management of multiple contact lists which are used for various campaigns. Naturally an email-recipient can be subscribed to one or more contact lists.
When we bounce an email-recipient in Interspire, we subscribe the email-recipient from all contact-lists, irrespective from where the bounce originates from. Interspire does not allow a "bulk-unsubscribe via email-address" and we have to run several API calls to unsubscribe a single recipient.
:warning: We cache the Interspire contact lists upon startup. In a future release I will adjust this to periodically refresh the cache to avoid a scenario where newly created lists are not refreshed.
MailWizz functions similar to Interspire and we also unsubscribe an email-recipient from all contact lists.
In order for the port25-bouncehandler to properly identify MailWizz campaigns, you need to configure a custom-header in MailWizz via "Servers -> Delivery Servers: Additional Headers" as depicted below (this needs to be done per delivery server):
You can manage bulk-unsubscribes via standalone. The only pre-requisite is a CSV file which contains an email-address in the first column of the file.
:warning: With Standalone processing we will always unsubscribe from all configured providers, so make sure that your CSV file is correct. We do however log from which lists a recipient is unsubscribed, so in case something goes wrong, you can find out which addresses are affected.
Command-line options
pmta
needs to have write-access to the file.To run the standalone processing you simply pipe the CSV file into the bounce-handler:
cat bounce.csv | /usr/bin/php ./bouncehandler/bouncehandler.php
This will then output progress into the console or log-file:
[29/May/2016:09:46:44] Port25 PowerMTA bounce-handler
[29/May/2016:09:46:44] (C) 2016 Gerd Naschenweng http://github.com/magicdude4eva
[29/May/2016:09:46:44] ------------------------------------------------------------------
[29/May/2016:09:46:44] Handling bounce categories=bad-mailbox,bad-domain,routing-errors,inactive-mailbox
[29/May/2016:09:46:44] Bounce-provider: Interspire, initialising
[29/May/2016:09:46:44] Endpoint-URL=http://interspire.example.com/xml.php
[29/May/2016:09:46:45] Interspire enabled with lists=112,3,32,95,81,108,109,115,116,117
[29/May/2016:09:46:45] Bounce-provider: Interspire, complete
[29/May/2016:09:46:45] Bounce-provider: MailWizz, initialising
[29/May/2016:09:46:45] Endpoint-URL=http://mailwizz.example.com/api
[29/May/2016:09:46:45] MailWizz enabled!
[29/May/2016:09:46:45] Bounce-provider: MailWizz, complete
[29/May/2016:09:46:45] Starting bounce processing
[29/May/2016:11:22:09] Starting bounce processing
[29/May/2016:11:22:10] MailWizz: unsubscribe for [email protected]:
[29/May/2016:11:22:10] - skipped: A Mailwizz list #1
[29/May/2016:11:22:10] - skipped: A Mailwizz list #2
[29/May/2016:11:22:11] Interspire: Skipping recipient [email protected] - no subscribed lists returned
....
[29/May/2016:11:22:12] Completed bounce processing! Total records=4, processed=4, skipped=0
The Standalone processing can also be used to process PowerMTA files without using the acct-file
-pipe-processing. It is perhaps something you should look at before automating it to test that integration works and that your bounce file is correct.
Running a PowerMTA file in standalone processing is the same command as above:
cat pmta-bounce-file.csv | /usr/bin/php ./bouncehandler/bouncehandler.php --debug
Note that if your setup does not follow the recommendations for the acct-file
and your columns are in different sequence, you will have to change the PORT25_OFFSET_*
-constants in bouncehandler.php
.
The processing of a PowerMTA file is quite similar, with the only addition that the record and bounce-category is also logged:
...
[29/May/2016:11:28:04] Bounce: bad-domain [email protected] via vmta-my-mta01/[email protected]
[29/May/2016:11:28:06] MailWizz: Skipping [email protected], already unsubscribed!
[29/May/2016:11:28:06] Bounce: bad-mailbox [email protected] via vmta-my-mta01/[email protected]
[29/May/2016:11:28:06] MailWizz: unsubscribe for [email protected]:
[29/May/2016:11:28:06] - skipped: MailWizz list name
[29/May/2016:11:28:06] Bounce: bad-domain [email protected] via vmta-my-mta01/[email protected]
[29/May/2016:11:28:07] Interspire: Skipping recipient [email protected] - no subscribed lists returned
...
:warning: I can not help you with configuration/setup issues when it comes to PowerMTA. I have extensive experience, but each setup varies, although the basic principles remain the same. This README is comprehensive enough that you will be able to configure your environment properly if you read this document twice.
If you do not follow the default setup below, I suggest you read the PowerMTA user-guide (specifically: 3.3.7 Accounting Directives
and 11. The Accounting and Statistics
).
My suggestion is that you first start off with manual processing, by just adding the following section to your /etc/pmta/config file.
<acct-file /var/log/pmta/bounce.csv>
delete-after 60d
move-interval 5m
max-size 500M
records b
record-fields b timeQueued,bounceCat,vmta,orig,rcpt,srcMta,dlvSourceIp,jobId,dsnStatus,dsnMta,dsnDiag
</acct-file>
This will generate an accounting file for just bounced records and will look something like this:
type,timeQueued,bounceCat,vmta,orig,rcpt,srcMta,dlvSourceIp,jobId,dsnStatus,dsnMta
b,2016-05-29 01:10:51+0200,bad-domain,vmta-mymta04,[email protected],[email protected],sourcemta.domain.com (0.0.0.0),1.1.1.1,,5.1.2 (bad destination system: no such domain),
b,2016-05-29 01:10:23+0200,bad-mailbox,vmta-mymta03,[email protected],[email protected],sourcemta.domain.com (0.0.0.0),1.1.1.1,,5.1.1 (bad destination mailbox address),mx1.destindomain.com (2.2.2.2)
Our bounce-processing relies on the sequence of record-fields in the bounce-file and if you change the order, you will need to change the index of those fields in the constants PORT25_OFFSET_*
as defined in bouncehandler.php
:
PowerMTA field | mapped record | Description |
---|---|---|
type | bounceRecord[0] | type - always b |
timeQueued | bounceRecord[1] | Time message was queued to disk |
bounceCat | bounceRecord[2] | likely category of the bounce (see Section 1.5), following the recipient which it refers |
vmta | bounceRecord[3] | VirtualMTA selected for this message, if any |
orig | bounceRecord[4] | originator (from MAIL FROM: |
rcpt | bounceRecord[5] | recipient (RCPT TO: |
srcMta | bounceRecord[6] | source from which the message was received. the MTA name (from the HELO/EHLO command) for messages received through SMTP |
dlvSourceIp | bounceRecord[7] | local IP address PowerMTA used for delivery |
jobId | bounceRecord[8] | job ID for the message, if any |
dsnStatus | bounceRecord[9] | DSN status for the recipient to which it refers |
dsnMta | bounceRecord[10] | DSN remote MTA for the recipient to which it refers |
dsnDiag | bounceRecord[11] | DSN diagnostic string for the recpient to which it refers |
You can then run the bounce-processing manually:
cat /var/log/pmta/bounce-2016-05-29-0000.csv | /usr/bin/php /opt/pmta/bouncehandler/bouncehandler.php --debug
Once you are comfortable with this you can then switch into automatic processing.
Switching to automatic processing is quite simple, you adjust your current record to the following:
<acct-file |/usr/bin/php /opt/pmta/bouncehandler/bouncehandler.php>
records b
record-fields b timeQueued,bounceCat,vmta,orig,rcpt,srcMta,dlvSourceIp,jobId,dsnStatus,dsnMta,dsnDiag
</acct-file>
Ensure that the bouncehandler logs the startup-messages in it's log-file. If this does not happen, then PowerMTA is not able to run the PHP due to possible permission errors (ownership of the /opt/pmta/bouncehandler folder or no executable permissions of the bouncehandler.php
).
Port25 is capable of processing feedback loop (FBL) reports. In our case we have automated the FBL processing, where Port25 receives the FBL report, then pipes it into our bouncehandler.php which then calls a feedback-loop processor. We automatically remove any reported email from all systems and notify our Postmaster team via email.
For the setup to work, the following is required:
Create a FBL domain fbl.example.com
Create a MX record for fbl.example.com
which points to your Port25 server - i.e. fbl.example.com MX 1 mailserver.example.com
Configure the feedback-loop-processor
in Port25 and list any addresses you accept for FBL reports:
<feedback-loop-processor>
deliver-unmatched-email no
deliver-matched-email no # default: no
forward-errors-to [email protected]
forward-unmatched-to [email protected]
<address-list>
address /[email protected]/
</address-list>
</feedback-loop-processor>
relay-domain fbl.bidorbuy.co.za
--logfile
to write to a different log-file:<acct-file /var/log/pmta/fbl.csv>
records feedback-loop
map-header-to-field f header_X-HmXmrOriginalRecipient rcpt # hotmail recipient
record-fields f *, header_subject, header_BatchId, header_Message-Id, header_List-Unsubscribe, header_List-Id, header_X-Mw-Subscriber-Uid, header_X-Mailer-LID, header_X-Mailer-RecptId
</acct-file>
<acct-file |/usr/bin/php /opt/pmta/bouncehandler/bouncehandler.php --logfile=/var/log/pmta/fbl-processor.log>
records feedback-loop
map-header-to-field f header_X-HmXmrOriginalRecipient rcpt # hotmail recipient
record-fields f *, header_subject, header_BatchId, header_Message-Id, header_List-Unsubscribe, header_List-Id, header_X-Mw-Subscriber-Uid, header_X-Mailer-LID, header_X-Mailer-RecptId
</acct-file>
feedback-loop-processor.php
to according to your requirements[email protected]
with the various FBL lists Word To The Wise - ISP Summary Information
In addition to the standard FBL fields, we write out the following fields:
rcpt
x-job-id
List-Id
- most mailers such as MailWizz use the standard headerSome FBL providers (such as OpenSRS/ReturnPath) will anonymise the recipient-email address and in those cases we need to use the Subscriber-ID/List-ID to unsubscribe the user.
The unsubscribe handling currently only supports MailWizz, but can be extended to anything else. In order for MailWizz to work, you will need install and configure the mailwizz-unsubscribe-extension
.
Additionally, you need to make the following DNS and Port25 changes:
Create a FBL domain fbl-unsub.example.com
Create a MX record for fbl-unsub.example.com
which points to your Port25 server - i.e. fbl-unsub.example.com MX 1 mailserver.example.com
Configure the fbl-unsub.example.com
domain in Port25 to process unsubscribes:
relay-domain fbl-unsub.example.com
<domain fbl-unsub.example.com>
type pipe
command "/usr/bin/php /opt/pmta/bouncehandler/unsubhandler.php --logfile=/var/log/pmta/unsubhandler.log --from=""$from"" --to=""$to"""
</domain>
The MailWizz extension modifies the List-Unsubscribe
to includes mailto and HTTP tags to allow providers such as Google, Microsoft, Yahoo to offer their users an automatic unsubscribe without leaving the email client:
List-Unsubscribe:
<mailto:[SUBSCRIBER_UID].[LIST_UID].[CAMPAIGN_UID]@fbl-unsub.example.com?subject=unsubscribe>,
<http://YOURMAILWIZZDOMAIN.COM/lists/[LIST_UID]/unsubscribe/[CAMPAIGN_UID]/[SUBSCRIBER_UID]/unsubscribe-direct?source=email-client-unsubscribe-button>
When a user clicks on the "Unsubscribe link", the mail-client will send an email to the mailto
address. Port25 (via the configuration of the MX record) will accept the email, extract the from
and to
details and then invokce the unsubhandler.php
piping the content of the email received into the handler.
The unsubhandler.php
then extracts the subscriber-UID and list-UID from the to
address and then calls the MailWizz API to unsubscribe the user. Once a unsubscribe was successful, an email is sent to you (you will need to adjust unsubhandler.php
to change this).
If you do not use MailWizz and generate your own List-Unsubscribe headers, I found the sequence of mailto
(first) and http
(second) to be very important. Google outright refuses to process unsubscribe requests if mailto
is not first.
If you are not using DKIM to sign your emails, you should revisit your email sender practises. DKIM is absolutely necessary for Google Postmaster Tools to function and both Feedback-ID
and List-Unsubscribe
need to be signed.
I have placed default options into a wild-card domain so that it applies to all recipients (also note the optimistic TLS for encrypted outbound mail):
<domain *>
...
dkim-sign yes # DKIM signing on messages
dkim-algorithm rsa-sha256 # recommended by RFC4871, default rsa-sha1
dkim-body-canon simple # recommended by DKIMCore (default relaxed)
dkim-headers Feedback-ID,List-Unsubscribe # additionally sign the Feedback-ID header
use-starttls yes # Specifies whether PowerMTA should use the STARTTLS extension
require-starttls no # We use optimistic TLS - i.e. only use it if the recipient supports it
...
</domain>
The configuration described above is running on a vanilla Centos7 installation with the Webtatic PHP repository and our current version is PHP 5.6.24.
From user feedback so far I noticed the following issues:
The bouncehandler receives piped data via Port25 and as such needs to run as the same user as Port25. Typically PowerMTA runs as user pmta
, so make sure to set the following permissions:
chown -R pmta:pmta /opt/pmta/bouncehandler
On CentOS7, PHP resides in /usr/bin/php
and the above configuration is based on that. I have seen some customer configurations where PHP resides in /usr/local/bin/php
and you would have to adjust the Port25 configuration to reflect that.
In a Linux environment, RRD is easy to configure and we use it to log unsubscribes and stats - this is pretty much experimental at the moment.
If you do not need it, just remove the following line from setup.php
:
// RRD Graphs - requires installation of php-rrdtool - if not defined, it will not be enabled
define("RRD_FILE", "/var/log/pmta/pmta.rrd");
The installation of RRD varies depending on your distribution. In our case of CentOS 7, it was as simple as:
yum install rrdtool-devel php56w-devel php56w-pear
In some environments you will need to either additionally install php-pecl-rrd
or php-rrdtool
Once done, you have to make the following change in your /etc/php.ini
; Used for Port25 RRD reporting
extension=rrd.so
I have been asked a number of times what infrastructure we are running, and while I can not go into all the specifics, any decent Linux admin will be able to replicate it:
pmtastats
is just an awesome tool for this).We use Logstash and Graylog for log collection and monitoring. Logstash has a versatile CSV filter which allows the parsing and processing of PowerMTA accounting files into Graylog or Kibana - the GIST can be viewed via the GIST: Port25 Logstash & Graylog integration
I often get asked "Can you help me integrate your scripts with product XYZ, it has APIs?". Although I enjoy coding, I don't think it is a good idea to write scripts for an environment which I do not use or monitor. The scripts I provide have been used in our production environment for several years and work reliable because we monitor them and tweak them when necessary.
My script setup is however very pluggable and any PHP developer will be able to write a processor by following these steps:
:beer: Please support me: If the above helped you in any way, then follow me on Twitter or send me some coins:
(CRO) cro1w2kvwrzp23aq54n3amwav4yy4a9ahq2kz2wtmj (Memo: 644996249) or 0xb83c3Fe378F5224fAdD7a0f8a7dD33a6C96C422C (Cronos)
(USDC) 0xb83c3Fe378F5224fAdD7a0f8a7dD33a6C96C422C
(BTC) 3628nqihXvw2RXsKtTR36dN6WvYzaHyr52
(ETH) 0xb83c3Fe378F5224fAdD7a0f8a7dD33a6C96C422C
(BAT) 0xb83c3Fe378F5224fAdD7a0f8a7dD33a6C96C422C
(LTC) MQxRAfhVU84KDVUqnZ5eV9MGyyaBEcQeDf
(Ripple) rKV8HEL3vLc6q9waTiJcewdRdSFyx67QFb (Tag: 1172047832)
(XLM) GB67TJFJO3GUA432EJ4JTODHFYSBTM44P4XQCDOFTXJNNPV2UKUJYVBF (Memo ID: 1406379394)
Go to Curve.com to add your Crypto.com card to ApplePay and signup to Crypto.com for a staking and free Crypto debit card.
Use Binance Exchange to trade #altcoins. Sign up with Coinbase and instantly get $10 in BTC. I also accept old-school PayPal.
If you have no crypto, follow me at least on Twitter.