Author Archives: xianator

How to Download Emails with Image Attachments from Gmail with PHP

If you’ve read “AWS Scripted”, you will have noticed that I recommend disabling file uploads because they pose a major security threat. I also suggest using email with an attachment as an alternative. Whilst not as clean as a file upload, and certainly more inconvenient for your users, this method is inherently much more secure. You could also use the code below to read emails without attachments from an account.

As an alternative, you could set up a completely separate server which only deals with file uploads. This is ok but I would not recommend keeping any sensitive data on it. And if your users are uploading things like identity documents, it would be a disaster if these were compromised. So at the very least, you need to send sensitive data off-server. However, you also need to make sure that wherever you send the data is also secure and not hackable from the file upload server. What you need is a ‘one-way’ passage to send the data down. You could achieve this by saving attachments to a database, but only allowing write access to the table from the credentials on the file upload server. You would enable read access only from a secure administration server with which you process the data.

In any case, below you will find PHP code to download emails and attachments from Gmail. Here is a possible scenario: you need an identity document from your user; you ask them to send an email with scan of said document attached to an email address you control; you have set up this email address (something like docs@yourcomany.com) to forward emails to a highly secure Gmail account (see below); you periodically run this script to fetch emails and attachments from Gmail and send them to your database; you have an Admin interface which allows you to view and process received emails. This is very secure because there is no public access to this server, only Admin access, which you could control with strong passwords, AWS Security Groups or firewalls.

Emails are inherently messy. There are lots of ways attachments can be attached, you’re dealing with MIME types and assorted paraphernalia. It took me a long time to get the code below working, and a couple of times I had to rewrite because a new style of email came in. Be aware that you might get an unintelligible email format and might need to add processing code.

Gmail is a good choice for this exercise because it is very secure if you turn on 2-Step Verification and it gives you lots of free space to store emails and attachments. You can keep the Gmail emails as a record or delete the emails as you prefer, but I would advise deleting only once human has verified the contents of an email. In the script below, we move the read emails to a ‘processed’ folder.

Gmail used to allow you to send and receive email from different email addresses than your default account address. But they have now disabled this very useful feature (it meant you could use Gmail to handle inbound and outbound emails from addresses on your domain). But you can still forward emails to your Gmail account, so with your Domain Registrar set up a new email address and forward it to your Gmail Account. Note that, in their infinite wisdom, Google don’t display emails sent from the receiving email account to the forwarding one – you just get one copy in your Sent Mail folder. But if you send an email to the forwarding address from a different account, it will appear.

On your Gmail account, you should definitely enable 2-Step Verification. However, this means you will need to create an app specific password to be used in the script below. Note that this password should be kept very secure, as it bypasses 2-Step Verification, so a Secure Laptop strategy is in order (see ‘AWS Scripted’ for more information on this). To create your app specific password:

  • go to your Google Account settings (click your picture top right and select ‘Account’)
  • select the ‘Security’ tab
  • click ‘Settings’ for ‘App passwords’
  • in the ‘App passwords’ box, select ‘Mail’ from the first drop-down
  • select Other (custom name) from the ‘Select Device’ drop-down
  • type a name, such as ‘PHP’
  • click ‘Generate’
  • copy the password shown into your PHP script

Also, in Gmail, you will need to create a new Label ‘processed’. On the left menu click ‘More’, then ‘Create new label’, then enter ‘processed’ as the label name and leave ‘Nest label under’ unchecked. Then press ‘Create’.

OK, so here is the PHP script which does the downloading:

<?php

// gmail credentials
$gmail_username='<your gmail email address>';
$gmail_password='<your gmail app password>';

// can be called without signing in
$signin=1;
require "../secure.php";
	
function get_date() {
	return date_format(date_create(), 'Y-m-d H:i:s');
	}

function get_decode_value($message, $encoding) {
	switch($encoding) {
		case 0:case 1:$message = imap_8bit($message); break;
		case 2:$message = imap_binary($message); break;
		case 3:case 5:$message=imap_base64($message); break;
		case 4:$message = imap_qprint($message); break;
		}
	return $message;
	}

function connectgmail($nusername, $npassword) {
	// connect to gmail
	$hostname = '{imap.gmail.com:993/imap/ssl}';
	// try to connect
	$inbox = imap_open($hostname, $nusername, $npassword) or die('Cannot connect to Gmail: ' . imap_last_error());
	return $inbox;
	}

function getattachment($ninbox, $nemail_number, $nid) {
	$message = array();
	$message["attachment"]["type"][0] = "text";
	$message["attachment"]["type"][1] = "multipart";
	$message["attachment"]["type"][2] = "message";
	$message["attachment"]["type"][3] = "application";
	$message["attachment"]["type"][4] = "audio";
	$message["attachment"]["type"][5] = "image";
	$message["attachment"]["type"][6] = "video";
	$message["attachment"]["type"][7] = "other";
	$structure = imap_fetchstructure($ninbox, $nemail_number);
	if (!isset($structure->parts))
		return "";
	$parts = $structure->parts;
	$message["pid"][1] = (1);
	if (count($parts)<=1)
		return "";
	$part = $parts[1];
	$message["type"][1] = $message["attachment"]["type"][$part->type] . "/" . strtolower($part->subtype);
	$message["subtype"][1] = strtolower($part->subtype);
	if (!isset($part->dparameters[0]->value))
		return "";
	$filename = $part->dparameters[0]->value;
	$body = imap_fetchbody($ninbox, $nemail_number, 2);
	$filename=$nid.".".substr($filename, strrpos($filename, '.')+1);
	$dst = "../attachments/".$filename;
	$fp = fopen($dst, 'w');
	$data = get_decode_value($body, $part->type);
	fputs($fp, $data);
	fclose($fp);
	return $filename;
	}

function getinlineattachment($ninbox, $nemail_number, $nid) {
	$body = imap_fetchbody($ninbox, $nemail_number, 1);
	$imgdata=base64_decode($body);
	$imgtype=getImageMimeType($imgdata);
	if ($imgtype=="")
		return "";
	$filename=$nid.".".$imgtype;
	$dst = "../attachments/".$filename;
	$fp = fopen($dst, 'w');
	fputs($fp, $imgdata);
	fclose($fp);
	return $filename;
	}

function getinlineattachment2($ninbox, $nemail_number, $nid) {
	$body = imap_fetchbody($ninbox, $nemail_number, 2);
	$dr=strpos($body, "Content-Transfer-Encoding: base64");
	$body=substr($body, $dr);
	$dr=strpos($body, "\r\n\r\n");
	$body=substr($body, $dr);
	$imgdata=base64_decode($body);
	$imgtype=getImageMimeType($imgdata);
	if ($imgtype=="")
		return "";
	$filename=$nid.".".$imgtype;
	$dst = "../attachments/".$filename;
	$fp = fopen($dst, 'w');
	fputs($fp, $imgdata);
	fclose($fp);
	return $filename;
	}

function getBytesFromHexString($hexdata) {
	for($count = 0; $count < strlen($hexdata); $count+=2)
		$bytes[] = chr(hexdec(substr($hexdata, $count, 2)));
	return implode($bytes);
	}

function getImageMimeType($imagedata) {
	$imagemimetypes = array("jpg" => "FFD8", "png" => "89504E470D0A1A0A", "gif" => "474946", "bmp" => "424D", "tif" => "4949", "tif" => "4D4D");
	foreach ($imagemimetypes as $mime => $hexbytes) {
		$bytes = getBytesFromHexString($hexbytes);
		if (substr($imagedata, 0, strlen($bytes)) == $bytes)
			return $mime;
		}
	return NULL;
	}

// set your gmail credentials here: email address and app password
$inbox=connectgmail($gmail_username, $gmail_password);

$emails = imap_search($inbox,'ALL');
if($emails) {
	sort($emails);
	foreach($emails as $email_number) {
		// get details
		$overview = imap_headerinfo($inbox, $email_number);
		$emailfrom=$overview->fromaddress;
		$emailsubject=$overview->subject;
		$emaildate=$overview->date;
		$emailmessage=imap_fetchbody($inbox, $email_number, 1.1);
		// message to be outputted
		echo get_date()." found email ID: ".$email_number." from: ".htmlentities($emailfrom)." subject: ".$emailsubject." message: ".$emailmessage."<br>";
		// clean data
		$emailfrom=substr($emailfrom, 0, 255);
		$emailsubject=substr($emailsubject, 0, 255);
		// insert into db and get id
		$result=doSQL("INSERT INTO receivedemails (emailfrom, emailsubject, emaildate, emailmessage) VALUES (?, ?, ?, ?);", $emailfrom, $emailsubject, $emaildate, $emailmessage) or die("Error inserting to receivedemails<br>".mysqli_error($db));
		$newid=$db->insert_id;
		// try to download the image as an attachment
		$hasattachment=1;
		$filename="";
		try {
			$filename=getattachment($inbox, $email_number, $newid);
			if ($filename=="")
				$hasattachment=0;
			}
		catch (Exception $e) {
			$hasattachment=0;
			}
		if ($hasattachment==0) {
			// try to download the image as an inline 1
			$filename=getinlineattachment($inbox, $email_number, $newid);
			if ($filename=="")
				$hasattachment=0;
			else
				$hasattachment=1;
			}
		if ($hasattachment==0) {
			// try to download the image as an inline 2
			$filename=getinlineattachment2($inbox, $email_number, $newid);
			if ($filename=="")
				$hasattachment=0;
			else
				$hasattachment=1;
			}
		$result=doSQL("update receivedemails set hasattachment=?, filename=? where receivedemailID=?;", $hasattachment, $filename, $newid) or die("Error updating receivedemails\n".mysqli_error($db));
		echo "attachment: ".(($hasattachment==1)?"yes":"no")."<br>";
		}

	// move to processed
	$elist="";
	foreach($emails as $email_number)
		$elist=$elist.$email_number.",";
	$elist=substr($elist, 0, -1);
	imap_mail_move($inbox, $elist, "processed");
	echo get_date()." email IDs: ".$elist." moved to processed<br>";
	}

imap_close($inbox);

?>

done

You’ll also want to be able to view received emails from the database, so I have provided a simple, configurable viewing script. The following zip contains a skeleton Admin website with all the required files:

imap_htdocs.zip (8kB)

You can use the files as a jumping board or to set up the website on an AWS server do the following:

  • launch a new AWS Instance using the latest Amazon Linux AMI
  • SSH to the new box
  • get root power:
    sudo su
    
  • install MySQL: follow the instructions in How to Install MySQL on an AWS Instance or:
    yum -y install mysql mysql-server
    chgrp -R mysql /var/lib/mysql
    chmod -R 770 /var/lib/mysql
    chkconfig --levels 235 mysqld on
    service mysqld start
    mysqladmin -u root password 0123456789
    
  • install Apache/PHP: follow the instructions in How to Install Apache/PHP on an AWS Instance or:
    yum -y install php php-mysql httpd
    chkconfig --levels 235 httpd on
    service httpd start
    
  • open up port 80 inbound in your AWS Security Group (from MyIP is more secure)
  • we will also be needing the PHP Imap extensions, install with:
    yum -y install php-imap
    service httpd restart
    
  • download and install the above zip to the box by running
    # move to webroot
    cd /var/www/html
    # download the zip
    wget http://www.quickstepapps.com/wp-content/uploads/2014/10/imap_htdocs.zip
    # unzip it
    unzip imap_htdocs.zip
    # delete zip
    rm -f imap_htdocs.zip
    # set webroot permissions
    find /var/www/html -type d -exec chown root:apache {} +
    find /var/www/html -type d -exec chmod 550 {} +
    find /var/www/html -type f -exec chown root:apache {} +
    find /var/www/html -type f -exec chmod 440 {} +
    # write access to attachments directory
    chmod 770 /var/www/html/attachments
    
  • install the database with:
    mysql --host=localhost --user=root --password=0123456789 --execute="source db.sql"
    # delete the script (if you want)
    rm -f db.sql
    
  • enter your Gmail credentials in the file sched/fetchemails.php with your favourite editor. This is in the two variables gmail_username and gmail_password right at the top.
  • now if you point your browser to your Public IP, you should see the password prompt: the password is 'admin'.

Test the system by sending an email to your Gmail address (or company address that is forwarding to Gmail) with an image attached. Then Click 'Check Now' on the Admin Website. When it's done, Click 'Emails' on the original page and you should see your email and image!

If you wanted to schedule the fetchemails.php page to be called every hour, you could use:

line="0 * * * * wget -O - http://localhost/sched/fetchemails.php"
(crontab -u root -l; echo "$line" ) | crontab -u root -

Don’t you just love email?

How to Install Apache/PHP on an AWS Instance

OK, this is pretty easy. Once you have your Amazon Linux AMI based Instance and you’ve SSHed to the box:

# install php, mysql connectivity and apache
yum -y install php php-mysql httpd
# start apache
service httpd start
# set apache to start on boot
chkconfig --levels 235 httpd on

You’ll need to open up port 80 inbound in your AWS Security Group. Then point your browser to the Public IP of the instance and you will see the Apache ‘It Works’ page.

Some useful commands:

# not most of these commands have to be run as root
# don't listen to the 'sudo su' naysayers, it's hip to be root!

# show the latest errors
tail /var/log/httpd/error_log

# show the latest accesses
tail /var/log/httpd/access_log

# move to the webroot
cd /var/www/html

# edit the httpd.conf config file
vi /etc/httpd/conf/httpd.conf

# edit the php.ini config file
vi /etc/php.ini

# restart apache (after any config changes or yum updates)
service httpd restart

# if you're not using monit (which you should be)
# this sets apache to autostart on boot
chkconfig --levels 235 httpd on

How to Configure iptables on an AWS Instance

If you are looking to improve security on your AWS Amazon Linux AMI Instance, iptables can be a good start. iptables is a kernel-based firewall service, very fast and free. Note that Inbound Security is handled by AWS Security Groups, so iptables is not strictly necessary to control Inbound packets. But if you turn iptables on, you will need to allow Inbound packets in addition to the SG rules.

Where iptables can help is with Outbound Security. If someone were to somehow get access to your servers, their main priority will be escalating privileges to root. Normally to do this you would need to download a ‘hacker pack’. You can prevent this with iptables. If, however, an attacker has root permissions on your box, iptables is useless because it can just be turned off.

Here is a simple starter script to be run on the AWS Instance. This assumes that you don’t do any Outbound Connecting from Apache via PHP (eg curl), but that you do connect to an RDS Instance. You will need to ‘sudo su’ before running the script, or run the script with sudo.

#!/bin/bash

# setup basic chains and allow all or we might get locked out while the rules are running...
iptables -P INPUT ACCEPT
iptables -P FORWARD ACCEPT
iptables -P OUTPUT ACCEPT

# clear rules
iptables -F

# allow HTTP inbound and replies
iptables -A INPUT -p tcp --dport 80 -m state --state NEW,ESTABLISHED -j ACCEPT
iptables -A OUTPUT -p tcp --sport 80 -m state --state ESTABLISHED -j ACCEPT

# allow HTTPS inbound and replies
iptables -A INPUT -p tcp --dport 443 -m state --state NEW,ESTABLISHED -j ACCEPT
iptables -A OUTPUT -p tcp --sport 443 -m state --state ESTABLISHED -j ACCEPT

# limit ssh connects to 10 every 10 seconds
# change the port 22 if ssh is listening on a different port (which it should be)
# in the instance's AWS Security Group, you should limit SSH access to just your IP
# however, this will severely impede a password crack attempt should the SG rule be misconfigured
iptables -A INPUT -p tcp --dport 22 -m state --state NEW -m recent --set
iptables -A INPUT -p tcp --dport 22 -m state --state NEW -m recent --update --seconds 10 --hitcount 10 -j DROP

# allow SSH inbound and replies
# change the port 22 if ssh is listening on a different port (which it should be)
iptables -A INPUT -p tcp --dport 22 -m state --state NEW,ESTABLISHED -j ACCEPT
iptables -A OUTPUT -p tcp --sport 22 -m state --state ESTABLISHED -j ACCEPT

# root can initiate HTTP outbound (for yum)
iptables -A OUTPUT -p tcp --dport 80 -m owner --uid-owner root -m state --state NEW,ESTABLISHED -j ACCEPT
# anyone can receive replies (ok since connections can't be initiated)
iptables -A INPUT -p tcp --sport 80 -m state --state ESTABLISHED -j ACCEPT

# root can do DNS searches (if your Subnet is 10.0.0.0/24 AWS DNS seems to be on 10.0.0.2)
# if your subnet is different, change 10.0.0.2 to your value (eg a 172.31.1.0/24 Subnet would be 172.31.1.2)
# see http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/vpc-dns.html
# DNS = start subnet range "plus two"
iptables -A OUTPUT -p udp --dport 53 -m owner --uid-owner root -d 10.0.0.2/32 -j ACCEPT
iptables -A INPUT -p udp --sport 53 -s 10.0.0.2/32 -j ACCEPT

# apache user can talk to rds server on 10.0.0.200:3306
iptables -A OUTPUT -p tcp --dport 3306 -m owner --uid-owner apache -d 10.0.0.200 -j ACCEPT
iptables -A INPUT -p tcp --sport 3306 -s 10.0.0.200 -j ACCEPT

# now drop everything else
iptables -P INPUT DROP
iptables -P FORWARD DROP
iptables -P OUTPUT DROP

# save config
/sbin/service iptables save

You need to be really careful with the SSH rules – if you get them wrong, you will be locked out of your box, permanently, no fix available. Also, if you are not using SSL (or SSL is terminated on your ELB), you should remove the 2 SSL rules.

DNS access is worth a mention. To determine your DNS servers, “add 2” to your Subnet, eg 10.0.0.0/24 would be 10.0.0.2/32. Another option is to type ‘nslookup’ on the server and then something like ‘google.com’ – the first line outputted will be the DNS IP address.

If you need to open up connectivity between servers in the Cloud VPC, here are some example rules, Inbound and Outbound:

# allow inbound access from 10.0.0.10 to port 2182
# change the IP and port as required
iptables -A INPUT -p tcp --dport 2182 -m state --state NEW,ESTABLISHED -s 10.0.0.10 -j ACCEPT
iptables -A OUTPUT -p tcp --sport 2182 -m state --state ESTABLISHED -j ACCEPT

# allow outbound access to 10.0.0.10 port 514 (eg for rsyslog)
# change the IP and port as required
iptables -A OUTPUT -p tcp --dport 514 -d 10.0.0.10 -j ACCEPT
iptables -A INPUT -p tcp --sport 514 -s 10.0.0.10 -j ACCEPT

# for something like rsyslog, which runs as root, we can be more restrictive
# allow outbound access ONLY TO root to 10.0.0.10:514 (eg for rsyslog)
# not the addition of '-m owner --uid-owner root'
# change the IP and port as required
iptables -A OUTPUT -p tcp --dport 514 -m owner --uid-owner root -d 10.0.0.10 -j ACCEPT
iptables -A INPUT -p tcp --sport 514 -s 10.0.0.10 -j ACCEPT

Here are some handy iptables commands (all need to be run as root, so do a ‘sudo su’ or prefix with ‘sudo’):

# stop iptables
service iptables stop

# start iptables
service iptables start

# cool stuff
watch 'iptables -nvL'

Note that a major problem with iptables is that domain names are resolved when the rule is added and not for every packet. This is probably a good thing because it would mean a lot of DNS lookups… However, if in the rules above you use a domain name (eg an instance DNS name rather than an IP) be aware that if the underlying IP changes, iptables will fail. This applies particularly to RDS services – you can’t fix the IP of an RDS Instance and Amazon recommend connecting via the DNS name, but iptables will resolve the DNS to an IP and if it changes connectivity will be lost. Bear this in mind if you loose database connectivity – to fix, rerun the iptables script.

Frankly, unless you really know what you are doing, I don’t recommend using iptables. The DNS problem mentioned above means your site could experience massive failure if any DNS/IP pairs change. It’s also very unforgiving and I have on several occasions had to terminate and rebuild servers because of a misconfiguration – so MAKE SURE you back everything up before you start playing around with iptables! But, for a simple server which doesn’t communicate much outbound, or only does so to fixed IP addresses, iptables can add very strong extra security.

How to Install MySQL on an AWS Instance

Amazon Web Services RDS provides everything you need to launch and use MySQL in the Cloud. However, for some applications, a dedicated database server may be overkill and too costly. In this article, I detail how to install MySQL on any Amazon Linux instance. In keeping with our ‘everything must be scripted’ philosophy, you will find ‘everything scripted’.

First, on the AWS instance, we ‘sudo su’. Then we run the yum commands to install MySQL Server and the MySQL command line tools, change some file permissions, set auto start on boot and then start the server:

# get root power
sudo su
# install mysql server and tools
yum -y install mysql mysql-server
# change permissions
chgrp -R mysql /var/lib/mysql
chmod -R 770 /var/lib/mysql
# set mysql to start on boot
chkconfig --levels 235 mysqld on
# start mysql
service mysqld start

Next, we change the root password (here 0123456789 but you should use your own):

mysqladmin -u root password 0123456789

Here’s how to create a new user and grant full privileges. First we connect to MySQL:

mysql --user=root --password=0123456789

This will give us a direct sql command line. Type the following to create the user tester with password tester1234:

create user 'tester'@'localhost' identified by 'tester1234';
grant all privileges on *.* to 'tester'@'localhost' with grant option;

This would allow the tester user to connect only from localhost (which is fine if you are only connecting to the database from eg PHP running on the same server). But if you want other servers to be able to connect, or you want to connect up to the database with MySQL Administrator or Query Browser from outside the Cloud, you will need the following:

create user 'tester'@'%' identified by 'tester1234';
grant all privileges on *.* to 'tester'@'%' with grant option;

Now tester can connect from any IP address. Don’t forget to open up the required ports in your AWS Security Group if you need access from anywhere not on the actual server. For connections from outside your AWS VPC, you should only allow access from your own IP address.

Say you have a database script you want to run, called db.sql and found in /home/ec2-user, you can do so with:

cd /home/ec2-user
mysql --host=localhost --user=tester --password=tester1234 --execute="source db.sql"

For more sql examples and PHP integration, please check Chapter 11 – MySQL Database in AWS Scripted. For how to automate the upload and execution of scripts, please check Chapter 13 – Uploading a New Database in AWS Scripted.