Postfix relay through Yahoo! (SSL)

Abstract

We take email for granted. It seems so easy. Just click the send button. But spam has made sending and receiving email a lot more complicated. This is especially true for people who like to run their own networks at home. If you are like me, you might have 1 or more physical machines running multiple virtual machines, not to mention multiple containers as well. All this network infrastructure needs to communicate with you somehow. After all, how do you keep tabs on whether or not your CRON jobs are successful? Email is the way. But sending email out from a home network without it being blocked at various points along the way can get complicated. We can go through the cost of buying a domain and email service, but who wants the cost? So how do we get the email through? This post explains how to configure Postfix to relay email through a Yahoo! account over SSL.

Disclaimer

This post is solely informative. Critically think before using any information presented. Learn from it but ultimately make your own decisions at your own risk.

Requirements

I did all of the work for this post using the following major technologies. You may be able to do the same thing with different technologies or versions, but no guarantees.

• A Yahoo! account
• Ubuntu 18.04.1 LTS
• Postfix 3.3.0

Install Postfix

I’m not going to go into a lot of explanation. I’m simply going to state the steps and configuration I did which got everything working.

First thing you need to do is install Postfix.

$ sudo apt-get update
$ sudo apt-get install postfix mailutils

NOTE For the Postfix installation, when it asks for the “System mail name” value, it should be the same as the name of the server.

Configure Yahoo! Account

The Yahoo! Account’s security setting must be set to allow password authentication. By default, it is not enabled. Go to the Yahoo! Account Security page and make sure “Password is enabled” is set as showing in Figure 1.

Figure 1 - Yahoo! Account Security

Yahoo! Account Security

Configure Postfix Authentication for Yahoo!

Configure Postfix to login to Yahoo!’s SMTP server. This configures the server name, port, account id, and clear text password.

# Create the password file
$ cd /etc/postfix/sasl
$ touch sasl_passwd_yahoo
$ chmod 600 sasl_passwd_yahoo

Now edit the sasl_passwd_yahoo file using your favorite editor. Make it look like Listing 1, replacing ACCOUNT_NAME and CLEAR_TEXT_PASSWD appropriately.

Listing 1 - sasl_passwd_yahoo

[smtp.mail.yahoo.com]:465 ACCOUNT_NAME@yahoo.com:CLEAR_TEXT_PASSWD

Now hash the sasl_passwd_yahoo file into a Postfix .db file

$ postmap sasl_passwd_yahoo

Configure Postfix Email Mapping for Yahoo!

If user mike attempts to send an email from a server with the name bluegreensky, the default Postfix FROM address will be mike@bluegreensky. This is not good because if you try to relay this email through Yahoo!, it will be blocked since the FROM address does not match the Yahoo! account. To get around this, configure a regular expression mapping file that will change all local email addresses (like mike@bluegreensky) to the Yahoo! account email address.

First, create a generic map file that’s empty

$ mkdir /etc/postfix/map
$ cd /etc/postfix/map
$ touch generic_map
$ chmod 600 generic_map
# Hash the file into a *.db file
$ postmap /etc/postfix/map/generic_map

Next, create a yahoo map file that will change local email address to the Yahoo! account email address.

$ mkdir /etc/postfix/map
$ cd /etc/postfix/map
$ touch regex_map_yahoo 
$ chmod 600 regex_map_yahoo 

Now edit the regex_map_yahoo file using your favorite editor. Make it look like Listing 2, replacing HOSTNAME - just HOSTNAME, not @HOSTNAME…don’t lose the @ character - and ACCOUNT_NAME appropriately.

Listing 2 - regex_map_yahoo

/.+@HOSTNAME/    ACCOUNT_NAME@yahoo.com

Now hash the regex_map_yahoo file into a Postfix .db file

$ postmap regex_map_yahoo

Configure Postfix SSL for Yahoo!

SSL must be used to connect to Yahoo! SMTP servers. During Postfix installation, a main.cf is created. It must be edited

$ cd /etc/postfix

Now edit the main.cf file using your favorite editor. In Listing 3 are the values you must add or update in main.cf. For each of the name/value pairs below, search main.cf to see if it already exists. If so, use my value below. If not, add my value to the end of main.cf.

Listing 3 - main.cf

# Yahoo!
relayhost = [smtp.mail.yahoo.com]:465
smtp_use_tls = yes
smtp_sasl_auth_enable = yes
smtp_sasl_security_options =
smtp_sasl_password_maps = hash:/etc/postfix/sasl/sasl_passwd_yahoo
smtp_tls_CAfile = /etc/ssl/certs/ca-certificates.crt
smtp_tls_wrappermode = yes
smtp_tls_security_level = encrypt
smtp_generic_maps = hash:/etc/postfix/map/generic_map, regexp:/etc/postfix/map/regex_map_yahoo

Restart Postfix

Restart Postfix so it picks up all the new configuration.

bash $ service postfix restart

Testing Postfix

If all goes well, Postfix restarted without any errors and is now configured to relay email from the server through the Yahoo! account. I typically test this with both mail and at.

# Test sending mail directly
$ echo "test email message" | mail -s "test email from server" some_email_address@someprovider.com

# Test sending mail through scheduler at
$ echo "echo \"gosh golly, it is AT\"" | at now

Do this testing, and see if you get the email your are expecting.

NOTE When testing with at, you will want to create a ~/.forward file with your email address so the results of the scheduled job are not delivered to your server’s local account’s inbox.

Summary

This post shows how to configure Postfix to relay email through a Yahoo! account over SSL. Sending email from a home network and have it not be blocked is tough. This configuration got me working. I hope it works for you.

Derby Database Backup

Abstract

I have already posted a number of blogs about Derby:

• Multiple Derby Network Servers on the same Host
• Apache Derby Database Users and Permissions
• Integration Testing with Maven and an In-Memory Derby Database

This wasn’t intended to be a series. But over the years I’ve been using Derby more and more. Recently, I started using Derby as my database of choice for my Microservice architecture. These are personal-use applications, so Derby is more than sufficient. Even though these are personal-use applications, I require multiple servers with limited user permissions and - most importantly - backup. I’d hate to lose my data! The purpose of this post is to demonstrate how to backup a Derby database.

Disclaimer

This post is solely informative. Critically think before using any information presented. Learn from it but ultimately make your own decisions at your own risk.

Requirements

I did all of the work for this post using the following major technologies. You may be able to do the same thing with different technologies or versions, but no guarantees.

• Apache Derby 10.14.2.0
• OpenJDK 64-Bit Server VM Zulu11.1+23 (build 11-ea+22, mixed mode)

Download

There are no downloads with this blog post. The scripts are shown in full.

Derby System Utility

Backing up a Derby database is really quite simple. Derby has a built-in system utility for performing the backup. The utility is SYSCS_UTIL.SYSCS_BACKUP_DATABASE('/location/of/the/backup/'). When called, Derby will lock the database and perform the copy operation to the file system location you specify as the parameter to SYSCS_BACKUP_DATABASE. Now that we know the system utility to do the backup, let’s look at a bash script to automate it.

Backup Script

Listing 1 is a bash script which can be easily modified to backup any Derby database on any network server.

Listing 1 - derby-mydatabase-backup.sh

#!/bin/bash

# Define a bunch of variables which will be used within this script.
# The names of the variables should be self-explanatory.
DERBY_HOME=/opt/db-derby-10.14.2.0-bin/
NETWORK_SERVER_HOST=localhost
NETWORK_SERVER_PORT=1527
DATABASE_NAME=mydatabase
DATABASE_USER=sa
DATABASE_PASSWORD=abc123
JDBC_URL="jdbc:derby://$NETWORK_SERVER_HOST:$NETWORK_SERVER_PORT/$DATABASE_NAME"
BACKUP_DIRECTORY="/tmp/$DATABASE_NAME-backup/$NETWORK_SERVER_PORT"
BACKUP_SCRIPT="$BACKUP_DIRECTORY/backup.sql"

# Remove old backup if it exists. It is not a good idea to
# perform a backup on top of an existing backup.
rm -rf $BACKUP_DIRECTORY
mkdir -p $BACKUP_DIRECTORY
cd $BACKUP_DIRECTORY

# Use the `echo` command to dynamically create an SQL file.
# This SQL file will be used by Derby `ij` to connect to
# the database and perform the backup.
echo "connect '$JDBC_URL' user '$DATABASE_USER' password '$DATABASE_PASSWORD';" >> $BACKUP_SCRIPT
echo "CALL SYSCS_UTIL.SYSCS_BACKUP_DATABASE('$BACKUP_DIRECTORY');" >> $BACKUP_SCRIPT
echo "exit;" >> $BACKUP_SCRIPT

# Run the Derby `ij` application, passing it the SQL file
# that was just dynamically created. `ij` will read the 
# SQL file, executing its commands. This will then
# cause `ij` to connect to the database and call the 
# system utility to perform the backup.
$DERBY_HOME/bin/ij $BACKUP_SCRIPT

Let’s take a look at this script in more detail.

Lines 5–15 setup a number of variables used within the script. Some variables are used to set the values of other variables. There is nothing too complicated here. The names of the variables are self-explanatory.

Lines 17–19 is file system maintenance. It is not a good idea to perform a backup on top of an existing backup. So these lines remove an existing backup (if it exists) and creates a new, empty, backup directory.

Lines 24–26 are then responsible for creating the backup.sql script file. This script file contains the SQL commands to perform the backup. Line 24 is the connect command so Derby ij can connect to the database you want to backup. Line 25 is where the magic happens with a call to the SYSCS_BACKUP_DATABASE system utility. The location of the backup directory is passed as a parameter to the utility. When this SQL command is executed, Derby will lock the database and perform the backup. Line 26 is the exit command to exit ij.

Line 33 is then finally where everything happens. The Derby ij command is called with the location of the dynamically created backup.sql file passed to ij as a command-line parameter. When bash executes line 33, and if everything goes well, the Derby database will be backed up.

NOTE If you are running the Derby network server with a Java security policy, you may run into some problems with this script. the Java SecurityManager may prevent the network connection to the database or the SecurityManager my encounter permission problems writing to the backup directory.

Summary

Backing up a Derby database is pretty easy. Just call SYSCS_UTIL.SYSCS_BACKUP_DATABASE('/location/of/the/backup/').

References

Backing Up a Database. (2013, January 24). db.apache.org. Retrieved from https://db.apache.org/derby/docs/10.0/manuals/admin/hubprnt43.html.

Multiple Derby Network Servers on the same Host

Abstract

Suppose you want to start multiple Derby network servers on the same host. They need to be listening on different ports and ideally store their data in different locations. The listings below show Linux bash and Windows batch scripts to configure starting a Derby network server. In this example, the Derby network server will listen on port 1110. Each Derby network server will also have its own file system location to store its databases.

Disclaimer

This post is solely informative. Critically think before using any information presented. Learn from it but ultimately make your own decisions at your own risk.

Requirements

I did all of the work for this post using the following major technologies. You may be able to do the same thing with different technologies or versions, but no guarantees.

• Apache Derby 10.14.1.0
• Java zulu11.1+23-ea-jdk11

I am not going to go through the process of downloading and installing these technologies. I’ll leave that as an exercise for you.

Linux bash scripts

Here are the Linux bash scripts to configure starting a Derby network server. We’ll look at the details of each file. First we’ll set the environment.

Listing 1 - setenv.sh

#!/bin/bash
 
export DERBY_HOME=/home/derby/opt/derby
export PATH="$DERBY_HOME/bin:$PATH"
echo "DERBY_HOME=$DERBY_HOME"

export JAVA_HOME=/home/derby/opt/java
echo "JAVA_HOME=$JAVA_HOME"

export NS_HOME=/var/local/derby/1110
mkdir -p $NS_HOME
echo "NS_HOME=$NS_HOME"

export NS_PORT=1110
echo "NS_PORT=$NS_PORT"

export NS_HOST=0.0.0.0
echo "NS_HOST=$NS_HOST"
 
export DERBY_OPTS=""
export DERBY_OPTS="$DERBY_OPTS -Dderby.drda.host=$NS_HOST"
export DERBY_OPTS="$DERBY_OPTS -Dderby.drda.portNumber=$NS_PORT"
export DERBY_OPTS="$DERBY_OPTS -Dderby.system.home=$NS_HOME"

DERBY_HOME is self explanatory: it’s where Derby is unzipped (installed). Add Derby’s bin directory to the PATH.

JAVA_HOME is self explanatory: it’s where Java is unzipped (installed). Add Java’s bin directory to the PATH.

NS_HOME is “Network Server Home”. This is the directory the Derby network server will use to store its configuration and databases. Whenever a new database is created on this Derby network server, a new sub-directory will be created under NS_HOME for the new database. This allows multiple Derby network servers running on the same host to keep their data separate.

NS_PORT is self explanatory: it’s the port the Derby network server uses to listen for connections. This allows multiple Derby network servers to run on the same host.

NS_HOST sets the network interface used by the Derby network server when listening for connections. By default, the Derby network server only listens for connections on the loopback address of 127.0.0.1. This default means clients must run on the same host as the network server - not very useful. By setting the host to 0.0.0.0, the Derby network server will listen for connections on any network interface on the host. If your Derby network server host has multiple network interfaces, NS_HOST should be set to the IP of one of those interfaces. Setting this value allows clients to be remote and run on any host.

DERBY_OPTS is the system property used to get all of the configuration options to Derby. It’s value is created by concatenating together the appropriate Derby system properties with their associated values.

Now that the environment is set, we can start the server.

Listing 2 - start.sh

#!/bin/bash

# Directory of the script
SD=$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )

# Source in common variables
source $SD/setenv.sh

startNetworkServer -noSecurityManager

SD is Script Directory. The evaluation determines the fully-qualified file system location of the start.sh script and assigns it to SD. This is useful when referencing other scripts.

source self explanatory: it sources in the environment configuration to run the Derby network server.

startNetworkServer -noSecurityManager starts the Derby network server. The DERBY_OPTS variable - set in setenv.sh - is used to configure the network server. The -noSecurityManager command line option needs some explanation. By default, Derby’s Java process runs with a limited security policy. I’ve found that this limited security policy gets in the way of database operations which I normally would just expect to work. By specifying -noSecurityManager, you run the Derby network server without any security policy. This may not be the ideal way for you to run a Derby network server. However, this blog is limited in scope to running the server. Search for my other blog on how to secure a Derby network server.

Now that the server is running, we need to stop it.

Listing 3 - stop.sh

#!/bin/bash

# Directory of the script
SD=$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )

# Source in common variables
source $SD/setenv.sh

stopNetworkServer

All of this is self explanatory. No further comments are needed for this script.

Windows batch scripts

Here are the Windows batch scripts to configure starting a Derby network server. We will not look at these files in additional details; details are included with the Linux bash scripts.

Listing 1 - setenv.cmd

@echo off
 
set DERBY_HOME=C:\Applications\Derby
set PATH=%DERBY_HOME%\bin;%PATH%
echo DERBY_HOME=%DERBY_HOME%

set JAVA_HOME=C:\Applications\Java
echo JAVA_HOME=%JAVA_HOME%

set NS_HOME=C:\Users\Derby\1110
md %NS_HOME% 2>NUL
echo NS_HOME=%NS_HOME%

set NS_PORT=1110
echo NS_PORT=%NS_PORT%

set NS_HOST=0.0.0.0
echo NS_HOST=%NS_HOST%
 
set DERBY_OPTS=-Dderby.drda.host=%NS_HOST%
set DERBY_OPTS=%DERBY_OPTS% -Dderby.drda.portNumber=%NS_PORT%
set DERBY_OPTS=%DERBY_OPTS% -Dderby.system.home=%NS_HOME%

Listing 2 - start.cmd

@echo off
call setenv.cmd
startNetworkServer -noSecurityManager

Listing 3 - stop.cmd

@echo off
call setenv.cmd
stopNetworkServer

Summary

That’s it. I hope you enjoyed learning how to run multiple Derby network servers on the same host.