Page of

Mailman 3 Setup Using Ubuntu Bionic

Created on 2018-12-20

Last Updated on 2020-06-16

If this guide needs editing, contact Kelly Close at LRE Water kelly.close@lrewater.com

NOTE 6/16/2020 - I set up another Ubuntu 18.04 Mailman3 system today following this guide which is now 1-1/2 years old. It still works! I tried to improve or clarify things that caught me up but after several re-starts it’s done and working, so I’m confident these instructions work. There are however some really delicate steps that need to be followed to the letter, in sequence and syntax. I strongly recommend making regular images of your system in progress if you are able, so you can roll back to the last working version if things get out of line, just like back in the 90’s when you would save your game after achieving challenging milestones so you didn’t have to start over next time you died! In fact, setting up this server really was a lot like playing an old game of Legend of Zelda. “It’s dangerous to go alone! Take this...”

Table of Contents

Server Credentials Quick Reference 3

Setting Up the New Server - UBUNTU 18.04 4

Assumptions and First Steps 4

Set the “Hosts” and “Hostname” 4

Installing the LAMP stack (Linux, Apache, MySQL and PHP) 5

Apache 5

PHP 6

MySQL (users planning to use PostgreSQL as the back end may opt to skip this) 6

Installing and Securing phpMyAdmin (only relevant if installing MySQL) 8

Installing PostgreSQL (for using PostgreSQL on the back end of Mailman 3) 9

Set Encoding Language 9

Install PostgreSQL 9

Configure the pg_hba.conf file 10

Install Additional PostgreSQL Libraries 11

Additional Server Tweaks 11

Adding Ubuntu User Roles and Groups (best practices) 11

Changing PHP Maximum Upload Limits To Enable Larger Uploads 11

Setting the Time Zone on New Servers 12

Setting up the DNS records for Postfix 12

Installing and Configuring Postfix: 12

UPDATE 6/16/2020 - Amazon Web Services (AWS) EC2 and Postfix 12

Install Postfix 13

Recommended responses to prompts: 13

Tweak the Postfix Configuration 13

Map Mail Addresses to Linux Accounts 14

Adjust the Firewall 14

Setting the Environment and install and configure the mail client 14

Initialize the Maildir and Test the Client 15

Managing Mail with the Client 16

Sending Mail with the Client 16

Install Ubuntu mailman3-full package 18

Follow README instructions for final configuration steps 19

README.Debian in /usr/share/mailman3/ 19

README.Debian in /usr/share/mailman3-web/ 20

Set up matching keys to authorize Hyperkitty 21

A final check of configurations for SSL setups 21

Other configuration tweaks 22

Set the time zone 22

Set site owner 22

Installing SSL Certificate 22

First Set Global Server Name Directive 22

SSL Cert Install process 23

Go to yourdomain.org/mailman3/postorius/lists 24

Add a redirect from the root of your URL to the lists path 24

Hyperkitty Archiver permissions and troubleshooting 24

Keys and Passwords (Updated 2020-6-20) 24

Base URL setting (added 2020-6-20) 25

Non-https port 80 listener for Hyperkitty (added 2020-6-20) 25

UPDATE 6/16/2020 for hanging browser on login 25

Added 6/16/2020 - Mail Security and Deliverability Configurations 26

Configuring messages with Templates 27

Copying lists over from a different Mailman3 server (PostgreSQL specific) 28

Clear the lock file 29

Extra step to apply a new domain name to your migrated lists 29

Server Credentials Quick Reference

(make a copy of this document and fill in your own information as you go for later reference)

Host IP Address _____________________

Additional Hosting Notes _____________________________________________________

Apache URL ______________________________
mailman listens by default on yourdomain.xxx/mailman3

Sudo SSH login user: ____________ password: ________________________

MySQL Root login user: root_______ password: ________________________

Mailman admin login user: ____________ password: ________________________

PostgreSQL super user user: _postgres____ password: ________________________

Postgresql port: 5432

Postgresql Mailman user user: _mailman3___ password: ________________________

Postorius super user user: mailman3web password: ________________________

Setting Up the New Server - UBUNTU 18.04

Assumptions and First Steps

This guide is based on starting with a fresh Ubuntu 18.04 LTS server install. Most recently I used the Ubuntu Server 18.04 image provided by Amazon Web Services - other images could vary in what optional Ubuntu packages are already have installed.
Text editing commands shown are using the “nano” editor,
Assign a fixed IP address to the server (e.g. 12.34.56.789) and establish a URL that points to that IP address with www and @ A records. Allow 48 hours for the domain records to propagate before beginning.
You need Port 25 open for Postfix. But many vendors, including Amazon, will block this port on your account by default, so even if you open it on your firewall settings, Postfix won’t work until you contact your cloud vendor and ask them to open the port. To get AWS’s default block on Port 25 lifted so Postfix can work, you will need to fill out this form and request the restriction be removed. Explain what you are doing and they will most likely lift it, but it can take 24 - 48 hours, so give yourself some lead time. https://console.aws.amazon.com/support/contacts?#/rdns-limits
Connect through SSH to the new server using the assigned IP address as root - this guide recommends the Bitvise SSH Client (https://www.bitvise.com/download-area).

Set the “Hosts” and “Hostname”

Type the following command to edit the host name file:

sudo nano /etc/hostname

By default the new server will be named “ubuntu” or possibly the IP address if one was autoassigned on launch (AWS). Change that if desired (must be all lowercase letters)
Nano tip: Save (cntrl+o) and exit (cntrl+x).
Type the following command to edit the hosts file:

sudo nano /etc/hosts

When this file opens it will most likely have a single entry that says something like: “127.0.0.1 localhost”. Now add a line after that line that says “127.0.1.1 nameofserver”, where the nameofserver is whatever name you just put in the hostname file.
Save (cntrl+o) and exit (cntrl+x) the file.
Restart the server. To do this from the command line:

sudo reboot

If you are using a virtual server you will need to reconnect your SSH client after a minute or so (give the system time to restart).

Installing the LAMP stack (Linux, Apache, MySQL and PHP)

Linux is already installed, so it’s just the next three.
This is a more complete reference for installing LAMP on Ubuntu Bionic:

https://www.digitalocean.com/community/tutorials/how-to-install-linux-apache-mysql-php-lamp-stack-ubuntu-18-04

Apache

Type on the SSH command line to first get all the packages current and then install:

sudo apt update
sudo apt -y upgrade (if prompted to keep existing versions of anything, always elect to install maintainer’s version, as this is a new setup and you want the most recent version of everything)
sudo apt install apache2

Adjust the firewall to allow web traffic (this is new for Ubuntu 18.04 -- for extra info view the Digital Ocean installation guide in the link above)

sudo ufw app list

You should see “Apache Full” in the list printed.

sudo ufw app info "Apache Full"

You should see “80,443/tcp” listed for the ports

sudo ufw allow in "Apache Full"

You may also want to enable the apache Rewrite module - I do this because I install Drupal sites on most of my servers and it is required to support the “clean url’s” feature but other things may need it as well. Restart apache afterward

sudo a2enmod rewrite
sudo service apache2 restart OR sudo systemctl restart apache2 OR sudo /etc/init.d/apache2 restart (NOTE: all three restart methods work on the image configuration I am using however other images may vary. I like the first method because it’s easiest to type)
To verify the module installed, type sudo apache2ctl -M and rewrite_module (shared) should appear in the list
To disable the rewrite module you can run sudo a2dismod rewrite

To verify that Apache is installed successfully, go to your server’s IP address in a browser window (eg. http://12.34.56.789). The page should display the words “It works!"

PHP

Type on the SSH command line:

sudo apt install php libapache2-mod-php php-mysql

Modify the way that Apache serves files when a directory is required (this is new for Ubuntu 18.04 -- for extra info view the Digital Ocean installation guide in the link above):

sudo nano /etc/apache2/mods-enabled/dir.conf
Add the following lines to the dir.conf file to move the PHP index file to the first position after the DirectoryIndex specification (and then save the file and exit):

<IfModule mod_dir.c>
DirectoryIndex index.php index.html index.cgi index.pl index.xhtml index.htm
</IfModule>

To see that PHP is working , create a new file with the following command:

sudo nano /var/www/html/info.php

Add the following lines to the file and then save and exit:

<?php

phpinfo();

Restart apache so that all of the changes take effect:

sudo service apache2 restart

Finish up by visiting your php info page (make sure you replace the example ip address with your correct one): http://5.152.179.191/info.php. If everything worked correctly, you should see a screen that describes the PHP version details. After ensuring everything worked correctly, remove info.php from the server by running the following command:

sudo rm /var/www/html/info.php

MySQL (users planning to use PostgreSQL as the back end may opt to skip this)

Type on the SSH command line:

sudo apt install mysql-server
sudo mysql_secure_installation

Say “no” to validating the password plugin because we’ll be using phpMyAdmin (for more info on why, view the Digital Ocean installation guide linked above)
During the initial installation, MySQL will ask you to set a root password. Record the password you set in the quick reference section for later reference.
A prompt during the secure setup will ask you for your current root password. Type in the password you just chose previously.
Say Yes to all the other options (Remove anonymous users? Y; Disallow root login remote? Y; Remove test database and access to it? Y; Reload privilege tables now? Y). At the end, MySQL will reload and implement the new changes.
Because we’re using a password to connect to MySQL as root (not the auth_socket plugin by default), we need to switch the authentication method (this is new for Ubuntu 18.04 -- for extra info view the Digital Ocean installation guide in the link above). Start MySQL and type the following commands:

sudo mysql to start mysql
mysql > SELECT user,authentication_string,plugin,host FROM mysql.user;

You should see that the root user authenticates using the auth_socket plugin

mysql > ALTER USER 'root'@'localhost' IDENTIFIED WITH mysql_native_password BY 'password'; where ‘password’ is the password you chose for the mysql root user
mysql > FLUSH PRIVILEGES;

You can check that the authentication method has changed by retyping:
mysql > SELECT user,authentication_string,plugin,host FROM mysql.user;

mysql > exit

NOTE: Once you have password authentication enabled, you will use a different command to access the MySQL shell:

mysql -u root -p

Installing and Securing phpMyAdmin (only relevant if installing MySQL)

The following is a more complete reference for installing/securing phpMyAdmin: https://www.digitalocean.com/community/tutorials/how-to-install-and-secure-phpmyadmin-on-ubuntu-18-04

sudo apt install phpmyadmin php-mbstring php-gettext

During the installation, phpMyAdmin will walk you through a basic configuration. Once the process starts up, follow these steps:

Select Apache2 for the server (note that sometimes initially apache2 is highlighted but not selected. You may need to hit ‘SPACE’, then ‘TAB’, and then ‘ENTER’ to select apache2)
Choose YES when asked whether to configure the database for phpmyadmin with dbconfig-common
Enter your MySQL root password for any password prompt - keeping phpMyAdmin and MySQL admin passwords the same makes things much simpler.

Enable the mbstring PHP extension (new for Ubuntu 18.04):

sudo phpenmod mbstring

Restart apache:

sudo service apache2 restart

One last step is needed in Ubuntu 18.04 to set up MySQL and phpMyAdmin

Create a symbolic link in the /var/www/http/ folder for phpmyadmin by typing:

sudo ln -s /usr/share/phpmyadmin /var/www/html/phpmyadmin

(note the above command is lower case letter ‘el’ and n (not a capital i)

You should be able to access phpmyadmin by visiting http://5.152.179.191/phpmyadmin.

Installing PostgreSQL (for using PostgreSQL on the back end of Mailman 3)

Set Encoding Language

Very Important: Before installing PostgreSQL, set the server’s default locale encoding language to UTF8 so that all of the postgres cluster databases are in UTF8 not ASCII2.

Type locale to verify that the server is already set to UTF-8. If it is not, it will display a bunch of current local settings equal to “C”
If needed, update locale settings by typing sudo nano /etc/default/locale and then updating the settings for “LC_ALL” and “LANG” to “en_US.UTF-8” (without quotes)
Reboot the server (sudo reboot). If using a virtual server you will need to reconnect your SSL terminal after a little wait (at least a minute). Once reconnected, type locale to see the change

Install PostgreSQL

To install postgreSQL use the SSH command line and type:

sudo apt install postgresql postgresql-contrib

Change the PostgreSQL postgres user password; we will not be able to access the server otherwise. Switch to the “postgres” Linux user (this was set up during the install above) and start PostgreSQL and set a password for the postgres user:

sudo -u postgres psql postgres
\password postgres

Provide the password when prompted, and record in the quick reference section.
PgAdmin requires the installation of an add-on for full functionality. The "adminpack" add-on, which it calls Server Instrumentation, is part of postgresql-contrib, so it’s already on the server because of the install command above, but it must be activated within postgreSQL:

Type the following command and when finished type Cntrl+d to exit again.

CREATE EXTENSION adminpack;

Type Cntrl+d to exit PostgreSQL

Configure the pg_hba.conf file

To allow pgAdmin to connect to the server with a password, edit the pg_hba.conf file and change the authentication method from peer to md5:

sudo nano /etc/postgresql/10/main/pg_hba.conf

Change any METHOD that is set to “peer” to “md5” (note, this changes all “ident” login support to “password” login support)
If using a remote (cloud) server, add your own IP address pg_hba.conf file.

# Your Connection

host all all 98.76.54.321/32 md5

To allow connection from any IP address by specific PostgreSQL users (careful!):

host all postgres 0.0.0.0/0 md5

After saving and exiting, you should reload the server configuration changes by typing:

sudo /etc/init.d/postgresql reload

Set PostgreSQL to listen for traffic

sudo nano /etc/postgresql/10/main/postgresql.conf

Uncomment and modify the the listen_addresses line to be:

listen_addresses = '*'

After saving and exiting from that file, make sure to reload and restart PostgreSQL:

sudo /etc/init.d/postgresql restart

You should now be able to log-on to the server using pgAdmin! As of the writing of this guide, the version of PostgreSQL installed by Ubuntu Bionic requires pgAdmin IV. Connect using the IP address assigned to your server, port 5432 (or other if you changed this), username: postgres and password: (the password you assigned)
NOTE that it is NOT necessary at this point to create a database user for Mailman 3, as the Mailman setup later on will do that for you.

Install Additional PostgreSQL Libraries

You will also need to add support for Python (Mailman3 requires that) and possibly other thing. There are lots of PostgreSQL add-in packages and libraries but the two additional ones I always add are for PHP and PostGIS.

Enable Python libraries for PostgreSQL (Required by Mailman3)

This library is used when using Python to communicate with PostgreSQL

sudo apt install python-psycopg2

Enable PHP libraries for PostgreSQL (optional)

This library is used when using PHP to communicate with PostgreSQL:

sudo apt install php7.2-pgsql (Updated for Ubuntu 18.04)

Enable the postGIS libraries for PG 10

sudo apt install postgresql-10-postgis-2.4
sudo apt install postgresql-10-postgis-scripts

Additional Server Tweaks

Adding Ubuntu User Roles and Groups (best practices)

Add Ubuntu users so others helping to administer the server don’t need to use root:

sudo adduser nameofnewuser

When adding new users, you will provide passwords. Users may change them later.
Add any users to the sudo group that you would like to have sudo privileges by typing:

sudo usermod -a -G sudo nameofuser

Verify what groups each person currently belongs to or to view what groups users belong to on another server (as an example), type the following. This list is view only, do not attempt to edit groups through this file.

sudo nano /etc/group

Changing PHP Maximum Upload Limits To Enable Larger Uploads

Tweak apache environment variables to allow upload of bigger files (particularly when restoring mysql dbs) - the default is 2MB. This guide recommends the values 202MB, 201MB and 200MB for memory_limit, post_max_size and upload_max_filesize respectively. This guide has more detail: http://stackoverflow.com/questions/3958615/import-file-size-limit-in-phpmyadmin
sudo nano /etc/php/7.2/apache2/php.ini

Search for entry (ctrl+W) post_max_size - enter 201M
Search for memory_limit and enter 202MB
Search for upload_max_filesize and enter 200MB

Save this file (ctrl-O) and exit nano (ctrl-X)
Restart apache: sudo service apache2 restart

(at this point, best practices dictate logging out of the server, and back in as a non-root user in the sudo group, created above)

Setting the Time Zone on New Servers

To set the correct time zone on a server, type the following command

sudo dpkg-reconfigure tzdata

In CO, USA, follow prompts selecting “US” and “Mountain” - elsewhere, choose wisely!

Setting up the DNS records for Postfix

Will be using Postfix on this server - to properly configure Postfix, you will need a Fully Qualified Domain Name (https://en.wikipedia.org/wiki/Fully_qualified_domain_name) pointed at the server (A records for @ and www) - for help with this you may need to contact your Domain host.

Installing and Configuring Postfix:

UPDATE 6/16/2020 - Amazon Web Services (AWS) EC2 and Postfix

You need Port 25 open for Postfix. But many vendors, including Amazon, will block this port on your account by default, so even if you open it on your firewall settings, Postfix won’t work until you contact your cloud vendor and ask them to open the port.
To get AWS’s default block on Port 25 lifted so Postfix can work, you will need to fill out this form and request the restriction be removed. https://console.aws.amazon.com/support/contacts?#/rdns-limits
Explain what you are doing and they will most likely lift it, but it can take 24 - 48 hours, so give yourself some lead time.

Source: https://www.digitalocean.com/community/tutorials/how-to-install-and-configure-postfix-on-ubuntu-18-04 Postfix is a popular open-source Mail Transfer Agent (MTA) that can be used to route and deliver email on a Linux system. It is estimated that around 25% of public mail servers on the internet run Postfix. In order to properly configure Postfix, you will need a Fully Qualified Domain Name pointed at your Ubuntu 18.04 server. You can find help on setting up your domain name with DigitalOcean by following this guide. If you plan on accepting mail, you will need to make sure you have an MX record pointing to your mail server as well. For the purposes of this tutorial, we will assume that you are configuring a host that has the FQDN of mail.example.com.

Install Postfix

Run the following commands to install (priority set to force more prompts at setup - this is very important!)

sudo DEBIAN_PRIORITY=low apt install postfix

Recommended responses to prompts:

General type of mail configuration?: Internet Site
System mail name: (yourdomain.xxx)
Root and postmaster mail recipient: ubuntu (this needs to be a user on your system; the ubuntu user is there by default on Ubuntu 18.04 so it’s convenient but any user may be used, considering it will be the owner of Postfix so it should be a permanent user with sudo privileges. Sending postfix address will be ubuntu@yourdomain.xxx)
Other destinations to accept mail for: accept defaults
Force synchronous updates on mail queue?: No
Local networks: accept defaults
Mailbox size limit: 0
Local address extension character: +
Internet protocols to use: all

To return and re-adjust these settings type:

sudo dpkg-reconfigure postfix

Tweak the Postfix Configuration

Set the home_mailbox variable to Maildir/ which will create a directory within the user's home directory (user specified above). The postconf command is used to set the configuration setting by typing:

sudo postconf -e 'home_mailbox= Maildir/'

Set the location of the virtual_alias_maps table:

sudo postconf -e 'virtual_alias_maps= hash:/etc/postfix/virtual'

Map Mail Addresses to Linux Accounts

The virtual alias map table allows you to list any other addresses that you wish to accept email for and map them to the primary address (set above).

sudo nano /etc/postfix/virtual

For example, if you would like to accept email at contact@example.com and admin@example.com and would like to have those emails delivered to the ubuntu Linux user, you could set up your file like this:

contact@example.com ubuntu
admin@example.com ubuntu

Save and close the file. Apply the mapping by typing:

sudo postmap /etc/postfix/virtual

Restart Postfix to be sure that all of our changes have been applied:

sudo systemctl restart postfix

Adjust the Firewall

If you are running the UFW firewall, as configured in the initial server setup guide, we'll have to allow an exception for Postfix.

sudo ufw allow Postfix

Setting the Environment and install and configure the mail client

To add the variable to these files, type:

echo 'export MAIL=~/Maildir' | sudo tee -a /etc/bash.bashrc | sudo tee -a /etc/profile.d/mail.sh

To read the variable into current session, source the /etc/profile.d/mail.sh file:

source /etc/profile.d/mail.sh

To install the s-nail package, type:

sudo apt install s-nail

Adjust a few settings. Open the /etc/s-nail.rc file in your editor:

sudo nano /etc/s-nail.rc

Search the file for “set emptystart” (CTRL-W in nano to search), and add the following options below it:

set emptystart
set folder=Maildir
set record=+sent

This will allow the client to open even with an empty inbox. It will also set the Maildir directory to the internal folder variable and then use this to create a sent mbox file within that, for storing sent mail. Save and close the file when you are finished (CTRL-O, CTRL-X).

Initialize the Maildir and Test the Client

Send the email by piping a string to the s-nail command. Adjust the command to mark your Linux user as the recipient:

echo 'init' | s-nail -s 'init' -Snorecord ubuntu

You may get the following response:

Can't canonicalize "/home/ubuntu/Maildir"

This is normal and may only appear when sending this first message. We can check to make sure the directory was created by looking for our ~/Maildir directory:

ls -R ~/Maildir

You should see the directory structure has been created and that a new message file is in the ~/Maildir/new directory:

/home/sammy/Maildir/:
cur new tmp

/home/sammy/Maildir/cur:

/home/sammy/Maildir/new:
1463177269.Vfd01I40e4dM691221.mail.example.com

/home/sammy/Maildir/tmp:

It looks like our mail has been delivered!

Managing Mail with the Client

Use the client to check your mail:

s-nail

You should see your new message waiting:

s-nail version v14.8.6. Type ? for help.
"/home/sammy/Maildir": 1 message 1 new
>N 1 sammy@example.com Wed Dec 31 19:00 14/369 init

Hitting ENTER should display your message:

[-- Message 1 -- 14 lines, 369 bytes --]:
From sammy@example.com Wed Dec 31 19:00:00 1969
Date: Fri, 13 May 2016 18:07:49 -0400
To: sammy@example.com
Subject: init
Message-Id: <20160513220749.A278F228D9@mail.example.com>
From: sammy@example.com

init

You can get back to your message list by typing h, and then ENTER:

s-nail version v14.8.6. Type ? for help.
"/home/sammy/Maildir": 1 message 1 new
>R 1 sammy@example.com Wed Dec 31 19:00 14/369 init

Since this message isn't very useful, we can delete it with d, and then ENTER:

Quit to get back to the terminal by typing q and then ENTER:

Sending Mail with the Client

You can test sending mail by typing a message in a text editor:

nano ~/test_message

Enter some text you'd like to email:

Hello,

This is a test. Please confirm receipt!

Using the cat command, we can pipe the message to the s-nail process. This will send the message as your Linux user by default. You can adjust the "From" field with the -r flag if you want to modify that value to something else:

cat ~/test_message | s-nail -s 'Test email subject line' -r from_field_account user@email.com

The options above are:

-s: The subject line of the email
-r: An optional change to the "From:" field of the email. By default, the Linux user you are logged in as will be used to populate this field. The -r option allows you to override this.
user@email.com: The account to send the email to. Change this to be a valid account you have access to.

You can view your sent messages within your s-nail client. Start the interactive client again by typing:

s-nail

Afterwards, view your sent messages by typing:

file +sent

You can manage sent mail using the same commands you use for incoming mail.

NOTE - This is a good time to back up the server in case you want to roll back to here!

Install Ubuntu mailman3-full package

This package installed the mailman3 core, mailman3-web interface, all needed dependencies including python (not the python library for PostgreSQL previously installed above however) and set up most of the configuration needed to integrate hyperkitty and postorious.

This guide assumes use of PostgreSQL for the database back end to Mailman 3

NOTE: Setting the priority in the command below to Low is ESSENTIAL, otherwise all prompts are not provided during install and it becomes a config file nightmare to get things working! Below are recommended replies to prompts (the ones I used that worked!) - note than many of these prompts appear to be asked multiple times but the text on the prompt pages are subtly different, some for mailman3, some for postorius, some for hyperkitty.

Be careful and be consistent. If you can make it through this part cleanly, it’s pretty clear sailing but one little typo here is hard to recover from. Also note, the prompts don’t necessarily show up in the order below, as they repeat for the different components.

(Save your image before proceeding if possible!)

sudo DEBIAN_PRIORITY=low apt install mailman3-full
Specified Internet site (SMTP) mail server and domain mhfd-lists.org
<Yes> to configure database for mailman3 with dbconfig-common
<Yes> to configure database for mailman3-web with dbconfig-common
Specified pgsql database
Connection method: TCP/IP
Host name: localhost
Port number: 5432
Method for authentication: password
Database name for Mailman: mailman3
User for mailman3: mailman3 (do not include the @localhost)
User mailman3 password: yourmailman3password
Database name for MailmanWeb: mailman3web
User for mailman3web: mailman3web (o not include the @localhost)
User mailman3web password: yourmailman3webpassword
Reply YES to restart web server
Postorius sending host: mhfd-lists.org
Postorius superuser and password to: admin yourpostoriusadminpassword
Postorius Email address (this should be a real, working email address that you have access to and does not need to be associated with your server domain): xxxx@yyyy.zzz
PostgreSQL administrative user name: postgres
PostgreSQL administrative user password: yourpostgrespassword

To re-configure later use sudo dpkg-reconfigure mailman3 (select NOT to reinstall db unless that process was unsuccessful the first time through)

The Mailman 3 deployment folder on Ubuntu Bionic is /etc/mailman3/...
NOTE: the config files live here

The Django-project folder is /usr/share/mailman3-web

NOTE: the settings_local.py file here is a symboloic link to mailman3-web.py in the deployment folder, so all configuring can be done from the Mailman 3 deployment folder!

Follow README instructions for final configuration steps

The mailman3-full packages drops README files into /usr/share/doc/mailman3 and mailman3-web. These describe additional configuration steps to enable the Mailman3 setup. Relevant excerpts are included below for easier reference.

README.Debian in /usr/share/mailman3/

For Mailman3 to work properly, you need to configure your mail transport agent (MTA) to deliver list mail to Mailman3
The following settings must be added to the end of /etc/postfix/main.cf - note this is 4 lines of code, even though two of them wrap in this document.
Type sudo nano /etc/postfix/main.cf to open and put this content at the end of the file:

# Mailman related settings

owner_request_special = no

transport_maps = hash:/var/lib/mailman3/data/postfix_lmtp

(note the following MUST be on all one line in the .cf file - do not include this note)

local_recipient_maps = proxy:unix:passwd.byname $alias_maps hash:/var/lib/mailman3/data/postfix_lmtp

(note the following MUST be on all one line in the .cf file - do not include this note)

relay_domains = ${{$compatibility_level} < {2} ? {$mydestination} : {}} hash:/var/lib/mailman3/data/postfix_domains

Your section of code will look something like this, 4 lines:

Restart postfix to make changes take effect: sudo service postfix restart

At this point, verify that the postfix lookup tables (aka hash or alias files) were created. Look in the folder /var/lib/mailman3/data/ and verify the hash (aka alias) files are there. You should see:

If there are no files in that folder then something is probably wrong with the lines of code just added to the postfix main.cf file - verify those first. And then to fix the issue type:

sudo -u list mailman aliases (the -u parameter ensures the new files created will be owned by the “list” user which is needed for mailman)

This should create the 4 needed files in /var/lib/mailman3/data/.

NOTE: The following was already done when installing on 6/16/202

README.Debian in /usr/share/mailman3-web/

First step references the /mailman3/ readme file (covered above)
Next, the Hyperkitty configuration snippet needs to be added to the Mailman3 configuration file in /etc/mailman3/mailman.cfg. VERIFY:
sudo nano /etc/mailman3/mailman.cfg

Verify these lines are present at the bottom of the mailman.cfg file:

[archiver.hyperkitty]

class: mailman_hyperkitty.Archiver

enable: yes

configuration: /etc/mailman3/mailman-hyperkitty.cfg

If not there run sudo dpkg-reconfigure mailman3
(if running this, select NOT to reinstall db and YES to add snippet)
(restarting of the mailman service may fail after this if the bottom config steps haven’t been done yet - just move on)

NOTE: The following was already done when installing on 6/16/2020

A superuser must be created for the Postorius web interface. This may have been done during configuration of the mailman3-web package, but can be done by running the reconfiguring command:

sudo dpkg-reconfigure mailman3-web - if running this, select:

NO to reinstall db,
accept mhfd-lists.org for domain,
set domain name for sender email address to mhfd-lists.org (instead of localhost.local),
set postorius superuser to:
User name: yourusername
Password: yourpassword
Email address: youremail@yourdomain.xxx
Selected apache2 for web server,
YES to restart web server

Set up matching keys to authorize Hyperkitty

There are two configuration files that have a key in them that must match for Hyperhitty to authorize and be able to store archives (the paths should be as follows):
/etc/mailman3/mailman-hyperkitty.cfg (set the api_key without any quotes)

/etc/mailman3/mailman-web.py (set the ‘SECRET_KEY’ - include single quotes around the key)

NOTE that the long string provided with setup should be replaced with a key of your own choosing. Avoid special characters - my install did not work until I set this key myself using just letters, numbers and some of the basic specials like underscore(_) and bang (!). I think it was a forward slash (/) in the original string that was messing mine up.

A final check of configurations for SSL setups

If you are using a system configured with https:// (SSL certificates) you may run into HyperKitty permissions when it tries to archive messages. Here is a list of configuration checks to review that will save you some time up front. There are more details on how to troubleshoot HyperKitty permission errors in a later section in this doc.

mailman-hyperkitty.cfg:

[general]

base_url: https://lists.mailman3.org/archives

api_key: xxx

settings_local.py:

MAILMAN_ARCHIVER_KEY = 'xxx'

ALLOWED_HOSTS = ['localhost',

'127.0.0.1',

'lists.mailman3.org',

'mail.mailman3.org',

'mailman.iad1.psf.io',

'mail.falconframework.org',

'104.239.228.201',

] (Or set ALLOWED_HOSTS to * but being explicit is more secure)

MAILMAN_ARCHIVER_FROM = ('127.0.0.1',

'::1',

'::ffff:127.0.0.1',

'104.239.228.201',

)

ACCOUNT_DEFAULT_HTTP_PROTOCOL = 'https'

Other configuration tweaks

Set the time zone

In /etc/mailman3/mailman-web.py change environment variable

Set TIME_ZONE = ‘America/Denver’ (mountain standard time - or as appropriate for you! Here is a list of possibilities https://stackoverflow.com/questions/13866926/is-there-a-list-of-pytz-timezones)

Set site owner

In /etc/mailman3/mailman.cfg

Set site_owner: (recommend using same address provided for Postorius superuser email above - this provides a backup for system messages)

Installing SSL Certificate

Can use “Let’s Encrypt” to install a free SSL Certificate on the server - follow the Digital Ocean guide here: https://www.digitalocean.com/community/tutorials/how-to-secure-apache-with-let-s-encrypt-on-ubuntu-18-04

NOTE: This guide does NOT use a virtual apache host setup; using the default apache setup is appropriate if the server will only be used for a single Mailman 3 site. To accomplish this, we needed to create a global server name directive (below) to eliminate the apache warning “Could not reliably determine the server’s fully qualified domain name”. (Error can be seen or checked that it’s gone by running sudo apache2ctl configtest).

First Set Global Server Name Directive

Create a global server name directive

Add the “ServerName localhost” directive to the /etc/apache2/apache2.conf file
Restart apache: sudo service apache2 restart

Verified this worked by typing: sudo apache2ctl configtest

SSL Cert Install process

First, add the repository: sudo add-apt-repository ppa:certbot/certbot

Then install Certbot's Apache package with apt: sudo apt install python-certbot-apache

Certbot is now ready to use. Be sure your domain has had time to propagate before configuring.

SSL Cert Configuration process:

Note the example below associates the SSL with both the www form and non www form of the URL. You can include as many subdomains as you want.

sudo certbot --apache -d xxxxxxxxxxx.xxx -d www.xxxxxxxxxxxx.xxx

Enter email address (used for urgent renewal and security notices) (Enter 'c' to

cancel): yourprojectemail@domain.xxx

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Please read the Terms of Service at

https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf. You must

agree in order to register with the ACME server at

https://acme-v02.api.letsencrypt.org/directory

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

(A)gree/(C)ancel: A

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Would you be willing to share your email address with the Electronic Frontier

Foundation, a founding partner of the Let's Encrypt project and the non-profit

organization that develops Certbot? We'd like to send you email about our work

encrypting the web, EFF news, campaigns, and ways to support digital freedom.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

(Y)es/(N)o: N

Obtaining a new certificate

.....stuff....

Which virtual host would you like to choose?

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

1: 000-default.conf | | | Enabled

2: 000-default-le-ssl.conf | udfcd-lists.org | HTTPS | Enabled

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Select the appropriate number [1-2] then [enter] (press 'c' to cancel): 2

Please choose whether or not to redirect HTTP traffic to HTTPS, removing HTTP access.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

1: No redirect - Make no further changes to the webserver configuration.

2: Redirect - Make all requests redirect to secure HTTPS access...

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Select the appropriate number [1-2] then [enter] (press 'c' to cancel): 2

...final confirmation stuff will print out on the screen, with info on how to test the certificate

Go to yourdomain.org/mailman3/postorius/lists

You should be able to use the US, log in and start setting up lists!
Make sure you’re on the https://... version (clear browser cache if needed)
Click Login and use the superuser created above to get into the system
Create lists!

Add a redirect from the root of your URL to the lists path

There are more sophisticated ways to do this but this is a quick way to make sure users that try to go to your domain yourdomain.com end up at the lists. Mailman 3 listens on the mailman3 subfolder and redirects on its own to yourdomain.org/mailman3/postorius/lists. This redirects from the root in a way that (in my experience) all browsers support without complaint.

The root of your web folder on your server (/var/www/html/ on Ubununtu) will have an index.html file in it that comes with apache. Copy that to a backup (sudo cp index.html index_apache_backup.html) and replace it with one that has only the content shown below.

sudo nano index.html (to open the file - then hold down CTRL-K to delete all existing content)

Type in the content:

<head>

</head>

CTRL-O (to save) CTRL-X (to exit)

Hyperkitty Archiver permissions and troubleshooting

Keys and Passwords (Updated 2020-6-20)

There are a number of keys and passwords in the 3 configuration files found in the deployment folder:

mailman-hyperkitty.cfg

api_key = xxx

mailman-web.py

SECRET_KEY = ‘aaaaaa’

MAILMAN_REST_API_PASS = ‘bbbbbb’

MAILMAN_ARCHIVER_KEY = ‘xxx’

DATABASES = .... ‘PASSWORD’ (for mailman3web database user): ‘yyy’

mailman.cfg

admin_pass: bbbbbb

url: postgres://mailman3:zzz@localhost/mailman3 (mailman3 database user)

Most of these populate on their own correctly during the setup steps but you will probably need to set the xxx passwords, or at least one of them (copy from the other file if it’s there in one file and not the other, or set your own in both places). It is crucial these match or your messages will not archive! You will see a HyperKitty Authorization error in /var/log/mailman3/mailman.log if these don’t match.

Base URL setting (added 2020-6-20)

Using an SSL certificate means you are also using Virtual Hosts. Using certbot as described in this guide sets all that up for you but mailman3 needs a little more help to navigate.

In the

base_url: http://udfcd-lists.org/mailman3/hyperkitty/

#base_url: http://localhost/mailman3/hyperkitty/

Non-https port 80 listener for Hyperkitty (added 2020-6-20)

If the 403 Forbidden error still persists AFTER making sure the archiver keys match and the base url is set to the actual domain and the server public ip is in the MAIL_FROM variable, then you probably need a “non-https port 80 listener” to let the local conversations with hyperkitty happen.

Add the following to the top of the file /etc/apache2/sites-available/000-default.conf

# non-https port 80 listener for localhost so hyperkitty can archive

ServerName localhost

Include conf-available/mailman3.conf

</VirtualHost>

After saving the file and exiting, reload apache2.

UPDATE 6/16/2020 for hanging browser on login

The 4 lines that you added to the postfix main.cf file are critical to success. If you reach the end of this setup and cannot log into your new Mailman website (if trying to log in results in the browser hanging and eventually going to a Server Error 500 page) check your logs with this command to see if the “hash tables” were created:

sudo tail /var/log/mail.log

If you see a line saying something like “error: open database /var/lib/mailman3/data/postfix_lmtp.db: No such file or directory” then your hash tables did not get created. Verify the 4 lines below in your /etc/postfix/main.cf file and fix if necessary. REstart postfix. Then follow these steps to fix the rest of the problem:

Create the hash tables manually: sudo -u list mailman aliases

(note that as of 2020-6-16 these hash files did not automatically get created, even with the correct lines of code in the Postfix config file so this manual step was required regardless)

Added 6/16/2020 - Mail Security and Deliverability Configurations

To improve the chances that messages sent from the server will make it past people’s spam filters, configure some additional mail server protocols.

Reverse DNS

Create the following TXT record in the domains DNS records. This allows receiving mail servers to verify that the mssage really came from the udfcd-lists.org server (and wasn’t spoofed). Almost all receiving mail services look for this now, and without this record messages from this mail server will get caught by spam filters routinely.

Note the IP address shown below should be a backwards version of the ip address for the Mailman3 server (the place that udfcd-lists.org is pointing to in the DNS A record).

191.179.152.5.in-addr.arpa.udfcd-lists.org 14400 IN TXT yourdomain.com

SPF

Create the following TXT records in the udfcd-lists.org DNS records. This is another common filter applied by receiving mail services - without this record messages from this mail server will get caught by spam filters routinely.

Note the IP address shown below should match the ip address for the Mailman3 server (the place that udfcd-lists.org is pointing to in the DNS A record).

DKIM

Install and configure DKIM on the Ubuntu server - great walkthru here that includes special instructions for Ubuntu 18: https://www.linuxbabe.com/mail-server/setting-up-dkim-and-spf

This also involves putting a public key (paired with a private key on the server) into a DNS record: (note - remember the semi-colon at the end of the key!

DMARC

This has not yet been configured. Use this guide if it should be:

Configuring messages with Templates

Text files can be provided to the Mailman3 system and tied to template settings so that content is automatically appended to messages. Templates can also be used to define content in auto generated messages such as welcome messages.

Official documentation:

https://docs.mailman3.org/projects/mailman/en/latest/src/mailman/rest/docs/templates.html

Excerpt from docs: The naming scheme for templates includes colon-separated sections following the form <context>:<recipient>:<type>:<name> where context would be “domain” or “list”, <recipient> would be “admin”, “user”, or “member”, and <type> can be a number of different things like “footer”. All template names used internally by Mailman are given in the docs linked above.

Key information for Ubuntu 18.04:

The default list templates live here, and affect all lists:

usr/lib/python3/dist-packages/mailman/templates/en/

Custom templates by list live here:

(note /var/lib/mailman3 is the “var_dir” variable set in /etc/mailman3/mailman.cfg)

/var/lib/mailman3/templates/lists

Subfolders under this folder match specific lists and within each list specific folder are .txt files. These files are named using the scheme above indicating how they are applied.

Database records tell Mailman which template .txt files go with which list. The table public.template stores these records.

In the version of Postorius installed with mailman3-full as of 2020-6-16 these must be added manually:

(Note - these records *should* be added automatically once they have been added on the file system but my experience as of 2020-6-16 was that they were not. I added the records to the database table in PostgreSQL (I use pgAdmin IV to interact with the db) and then the overrides worked.)

Copying lists over from a different Mailman3 server (PostgreSQL specific)

It’s actually super easy to migrate all your lists from one Mailman3 server to another. You can also move the lists over to a different server that uses a different domain with an extra step (note: I have seen a reference to there being a python script included with the mailman install that will rename all lists to a new domain but I couldn’t find it - add a comment here if you know where that is!

Shut down mailman3 on your old server

This will prevent there from being a lock file in your backups

First - backup the mailman3 and mailman3web databases from your existing server.

sudo service mailman3 stop

Also zip up the /var/lib/mailman3 folders

sudo tar -cvzf var_lib_mailman3.gz
You may wish to exclude the cache folder contents, or delete them from your .gz file as it is just unneeded bulk.

Rename the 2 databases on your new server that were installed with your setup

On your new server, those databases will be locked if you have mailman3 running so stop the mailman3 service before renaming

Create 2 new databases in your new server PostgreSQL instance, mailman3 and mailman3web

they can be owned by postgres, assuming that is your administrative user

Restore the backups to the newly created databases
Transfer the .gz file you created on your old server to your new server, dropping it into the /var/lib/mailman3 folder
Unzip the gz file

sudo tar -xvf var_lib_mailman3.gz

Clear the lock file

If there is anything in /var/lib/mailman/locks, remove it
sudo rm /var/lib/mailman/locks/*

Extra step to apply a new domain name to your migrated lists

You will need to use a PostgreSQL SQL window to apply these changes from within PostgreSQL. Ideally you will have pgAdmin installed and be able to do this through the GUI but it can be done at the pgsql command line too:

Tables in mailman3 that reference the domain need to be updated:

domain - update field mail_host
mailinglist - update fields list_id and mailhost
member - update field list_id
bounceevent - update fields list_id and email
ban - update field list_id

For each of these, update the appropriate fields using search/replace.

EXAMPLE:

UPDATE member SET list_id=replace(list_id,’udfcd-lists’,’mhfd-lists’);

Tables in mailman3web that reference the domain need to be updated:

django_mailman3_maildomain - update field mail_domain
django_site - update fields domain and name
hyperkitty_mailinglist - update field name and list_id

For each of these update the appropriate fields using search/replace.

EXAMPLE:

UPDATE member SET list_id=replace(list_id,’udfcd-lists’,’mhfd-lists’);