Implementing an Amazon SES proxy server using PHP

Understanding this article requires you to have a certain foundation in using SES. If you don’t understand, you can read the discussion in this question. http://segmentfault.com/q/1010000000095210

SES’s full name is Simple Email Service, which is a basic email service launched by Amazon. As part of AWS basic services, it inherits the traditional advantages of AWS -- cheap.

Yes, it’s really cheap. This is why I don't use mailgun or any other more awesome email service. If you send 100,000 emails per month, you will basically only need to pay about ten dollars. Compared with other services that often start at hundreds of dollars, this price advantage is huge. Therefore, with this I can tolerate its many shortcomings.

But as the number of people using SES in China increased, he was suddenly blocked one day at the end of last year, which was fatal. So, I started trying to build a proxy on my own server overseas to continue using this service. At the same time, this also provides an opportunity for me to improve its API to implement some more valuable functions, such as mass mailing.

So I didn’t use an overseas server to directly make a reverse proxy to play. This only solved the superficial problem, but my need to expand functions was impossible to achieve. Therefore, I set two basic goals for designing this SES agent

Fully compatible with the original API interface, which means that the original code basically does not need to be changed to use the proxy Implement mass mailing function

Achieving the first point is actually very simple. In fact, it is to use PHP to implement a reverse proxy, receive the parameters sent, and then use the curl component to send it to the real SES server after assembly, and then directly output it to the client after obtaining the receipt. . This is a standard agency process. My code is given below, and I have commented on the important parts

It should be noted that these codes need to be placed in the root directory of the domain name. Of course, second-level domain names can also be used

include __DIR__ . '/includes.php';

// Here are a few more important headers, others do not need to be paid attention to
$headers = array(
 'Date: ' . get_header ('Date'),
 'Host: ' . SES_HOST,
 'X-Amzn-Authorization: ' . get_header('X-Amzn-Authorization')
);

// Then assemble the url again to request this correct SES server
$url = 'https://' . SES_HOST . '/'
 . (empty($_SERVER['QUERY_STRING']) ? '' : '?' . $_SERVER['QUERY_STRING']);

 $ch = curl_init();
curl_setopt($ch, CURLOPT_USERAGENT, 'SimpleEmailService/php');
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 1);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 1);

// needs to be processed The only ones are the `POST` and `DELETE` methods. There are many `GET` methods so I won’t implement them one by one
// In fact, they are all methods to obtain the current information. You can view this information directly in the background
switch ($_SERVER ['REQUEST_METHOD']) {
 case 'GET':
 break;
 case 'POST':
 global $HTTP_RAW_POST_DATA;
 $data = empty($HTTP_RAW_POST_DATA) ? file_get_contents('php://input')
 : $ HTTP_RAW_POST_DATA;
 $headers[] = 'Content-Type: application/x-www-form-urlencoded';
 parse_data($data);

 curl_setopt($ch, CURLOPT_CUSTOMREQUEST, 'POST');
 curl_setopt($ch , CURLOPT_POSTFIELDS, $data);
 break;
 case 'DELETE':
 curl_setopt($ch, CURLOPT_CUSTOMREQUEST, 'DELETE');
 break;
 default:
 break;
}

curl_setopt($ch, CURLOPT_HTTPHEADER, $ headers);
curl_setopt($ch, CURLOPT_HEADER, false);

curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);

$response = curl_exec($ch);
$content_type = curl_getinfo($ch, CURLINFO_CONTENT_TYPE);
$status = curl_getinfo($ch, CURLINFO_HTTP_CODE);

curl_close($ch);

header('Content-Type : ' . $content_type, true, $status);
echo $response;

This code is very simple, but there are some tricks that need to be paid attention to. Among them, I used a private function called parse_data when processing the POST method. This function is actually the key to implementing mass mailing.

Speaking of this, I have to mention the SES email sending API. SES only provides a simple email sending API, which supports multiple sending objects. But when you send to multiple recipients, it will also In the recipient column, you can see the addresses of other recipients. Of course, it also supports the carbon copy function of cc or bcc, but when you use this carbon copy function to send group emails, the recipients will see that they are among the carbon copy objects, not among the recipients. For a regular website, these are obviously intolerable.

So we need a real concurrent interface to send emails. You must know that the quota assigned to me by SES is to send 28 emails per second (each person has a different quota). If fully utilized, 100,000 emails can be sent per hour, which is completely It can meet the needs of medium-sized websites.

So I came up with an idea, Without changing the client interface at all, I unpacked an email with multiple recipients sent over the proxy server into multiple emails with a single recipient. emails, and then send these emails to SES using an asynchronous queue. This is what the parse_data function does. Below I will directly give the code in includes.php, which contains all the private functions to be used. Please modify the previous define according to your own needs

define('REDIS_HOST', '127.0.0.1');
define('REDIS_PORT', 6379);
define('SES_HOST', 'email.us-east-1.amazonaws.com');
define('SES_KEY', '');
define('SES_SECRET', '');

/**
 * get_header
 *
 * @param mixed $name
 * @access public
 * @return void
*/
function get_header($name) {
    $name = 'HTTP_' . strtoupper(str_replace('-', '_', $name));
    return isset($_SERVER[$name]) ? $_SERVER[$name] : '';
}

/**
 * my_parse_str
 *
 * @param mixed $query
 * @param mixed $params
 * @access public
 * @return void
*/
function my_parse_str($query, &$params) {
    if (empty($query)) {
        return;
    }

    $decode = function ($str) {
        return rawurldecode(str_replace('~', '%7E', $str));
    };

    $data = explode('&', $query);
    $params = array();
    foreach ($data as $value) {
        list ($key, $val) = explode('=', $value, 2);
        if (isset($params[$key])) {
            if (!is_array($params[$key])) {
                $params[$key] = array($params[$key]);
            }
            $params[$key][] = $val;
        } else {
            $params[$key] = $decode($val);
        }
    }
}

/**
 * my_urlencode
 *
 * @param mixed $str
 * @access public
 * @return void
*/
function my_urlencode($str) {
    return str_replace('%7E', '~', rawurlencode($str));
}

/**
 * my_build_query
 *
 * @param mixed $params
 * @access public
 * @return void
*/
function my_build_query($parameters) {
    $params = array();
    foreach ($parameters as $var => $value) {
        if (is_array($value)) {
            foreach ($value as $v) {
                $params[] = $var.'='.my_urlencode($v);
            }
        } else {
            $params[] = $var.'='.my_urlencode($value);
        }
    }

    sort($params, SORT_STRING);
    return implode('&', $params);
}

/**
 * my_headers
 *
 * @param mixed $headers
 * @access public
 * @return void
*/
function my_headers() {
    $date = gmdate('D, d M Y H:i:s e');
    $sig = base64_encode(hash_hmac('sha256', $date, SES_SECRET, true));

    $headers = array();
    $headers[] = 'Date: ' . $date;
    $headers[] = 'Host: ' . SES_HOST;

    $auth = 'AWS3-HTTPS AWSAccessKeyId=' . SES_KEY;
    $auth .= ',Algorithm=HmacSHA256,Signature=' . $sig;

    $headers[] = 'X-Amzn-Authorization: ' . $auth;
    $headers[] = 'Content-Type: application/x-www-form-urlencoded';

    return $headers;
}

/**
 * parse_data
 *
 * @param mixed $data
 * @access public
 * @return void
*/
function parse_data(&$data) {
    my_parse_str($data, $params);

    if (!empty($params)) {
        $redis = new Redis();
        $redis->connect(REDIS_HOST, REDIS_PORT);

        // 多个发送地址
        if (isset($params['Destination.ToAddresses.member.2'])) {
            $address = array();
            $mKey = uniqid();

            $i = 2;
            while (isset($params['Destination.ToAddresses.member.' . $i])) {
                $aKey = uniqid();
                $key = 'Destination.ToAddresses.member.' . $i;
                $address[$aKey] = $params[$key];
                unset($params[$key]);

                $i ++;
            }

            $data = my_build_query($params);

            unset($params['Destination.ToAddresses.member.1']);
            $redis->set('m:' . $mKey, my_build_query($params));
            foreach ($address as $k => $a) {
                $redis->hSet('a:' . $mKey, $k, $a);
                $redis->lPush('mail', $k . '|' . $mKey);
            }
        }
    }
}

You can see that the parse_data function starts from the second recipient, assembles them into separate emails, and puts them in the redis queue for other independent processes to read and send.

Why not start with the first recipient?

Because it needs to be compatible with the original protocol, when the client sends an email request, you always have to return something to it. I am too lazy to forge it, so the email request of the first recipient is sent directly. Without entering the queue, I can get a real SES server receipt and return it to the client, and the client code can handle this return without any modification.

What should I do if SES emails require a signature?

Yes, all SES emails require a signature. So after you unpack it, the email data changes, so the signature must change too. The my_build_query function does this, it re-signs the request parameters.

The following is the last component of this proxy system, the mail sending queue implementation, which is also a php file. You can use the nohup php command to start several php processes in the background according to your own quota size to achieve concurrent mail sending. Its structure is also very simple, it is to read the emails in the queue and then use curl to send the request

include __DIR__ . '/includes.php';

$redis = new Redis();
$redis->connect(REDIS_HOST, REDIS_PORT);

do {
 $pop = $redis->brPop('mail', 10);
 if (empty($pop)) {
 continue;
 }

 list ($k, $id) = $pop;
 list($aKey, $mKey) = explode('|', $id);

 $address = $redis->hGet('a:' . $mKey, $aKey);
 if (empty($address)) {
 continue ;
 }

 $data = $redis->get('m:' . $mKey);
 if (empty($data)) {
 continue;
 }

 my_parse_str($data, $params);
 $params['Destination.ToAddresses.member.1'] = $address;
 $data = my_build_query($params);
 $headers = my_headers();
 $url = 'https://' . SES_HOST . ' /';

 $ch = curl_init();
 curl_setopt($ch, CURLOPT_USERAGENT, 'SimpleEmailService/php');
 curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false);
 curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);

 curl_setopt($ch, CURLOPT_CUSTOMREQUEST, 'POST');
 curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
 curl_setopt($ch, CURLOPT_POSTFIELDS, $data);
 curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
 curl_setopt ($ch, CURLOPT_URL, $url);
 curl_setopt($ch, CURLOPT_TIMEOUT, 10);

 curl_exec($ch);
 curl_close($ch);

 unset($ch);
 unset($data) ;

} while (true);

The above is my entire idea of ​​​​writing the SES mail proxy server. Everyone is welcome to discuss it together.

Proxy server, PHP, Amazon