Here we are again! Another tutorial in the series of “hardcores or… masochists!”. I find it interesting into digging into the protocols that we use many times a day and we have no clue what is going on behind the scenes. And here i come with the next most used thing after the HTTP, the mailing system. I have not talked about any specific protocol yet because it’s not only one, it’s three. It is POP (Post Office Protocol) and IMAP (Internet Message Access Protocol) for reading/receiving your email and SMTP (Simple Mail Transfer Protocol) for sending.
You will, once more, be surprised how simple and easy are these protocols. All three work similarly to HTTP, headers are sent both sides and then the message is retreived. The only difference is that the connection between the server and the client is open until the client closes it, or it has timed out or an error occurs. This is important for the programs/scripts anyone will attempt to write (i will clarify later).
So, let’s begin our safari in the jungle of protocols. First of the POP protocol. Very well know, heard all the time. The dafault port of the POP server is 110. The sequence is this. You open a connection with the server and you get a welcome message. Then, you send a username and get the message which is an ok message or an error message. In case of OK, you need to send the password and again get OK or ERR. If everything has gone well by now you are logged in and ready to issue commands. Basicaly there are the following commands you will need:
- USER <username> -> send the username
- PASS <password> -> send the password
- STAT -> the reply will be like this “+OK #msgs #bytes”, in other words the total messages on the server and the total bytes they all sum up to.
- LIST -> the reply will be a list of all the messages, one per line, and the bytes for each one.
- RETR <msgno> -> the reply will be the whole message
- DELE <msgno> -> flags the message for deletion. It will not be deleted until you issue a quit. If the connection to the server is lost the message will not be deleted.
- QUIT-> closing the connection with the server.
There are some more commands, that are not standard. But, using those ones, you can do the job very easy. An example of a session with a POP server is listed below:
> telnet myserver.com 110
Trying IP...
Connected to myserver.com.
Escape character is '^]'.
+OK Hello there.
USER myuser
+OK Password required.
PASS mypass
+OK logged in.
STAT
+OK 13 53370
LIST
+OK POP3 clients that break here, they violate STD53.
1 3853
2 5272
3 3529
4 2383
5 4848
6 7640
7 6549
8 8490
9 2764
10 4174
11 1525
12 770
13 1573
.
RETR 12
+OK 770 octets follow.
Return-path:
Envelope-to:
Delivery-date: Sun, 09 Mar 2008 00:54:45 +0200
Received: from [smtp_ip_here] (helo=[from_ip_here])
by with esmtpsa (TLSv1:AES256-SHA:256)
(Exim 4.68)
(envelope-from )
id 1JY7wD-0006cn-8A; Sun, 09 Mar 2008 00:54:45 +0200
Message-ID: <47D3191C.7010301@mailserver_here>
Date: Sun, 09 Mar 2008 00:54:20 +0200
From: from_name_here
User-Agent: Thunderbird 2.0.0.12 (Windows/20080213)
MIME-Version: 1.0
To:
Subject: last test i hope
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
last test
.
quit
+OK Bye-bye.
As you can see i did what i explained above. I opened the connection, sent username and password, issued a stat command to see the total messages and bytes, issued a list to see details for each message, issued a retr to get the 12th email and finaly closed connection with the server with a quit. Let’s focus on the message. As you can see, it’s very very similar with the HTTP protocol. Headers sent with info about the message, delimited with CRLF’s (\r\n) and then two CRLF’s seperate the headers with the message.
Here comes the question though. What about attachments?? Well can you imagine of the solution? Just like HTTP. The message headers are the same with the simple message. The difference comes in the one with the content type. With the simple message above it was plain/text this time it’s Content-Type: multipart/mixed; As you can imagine, the boundary seperates the various parts of the message.
boundary="------------070003020608000203050002"
Before giving you the dump of the message i would like to point out one more thing. In the message before you can see a header “MIME-Version: 1.0”. MIME is short for Multipurpose Internet Mail Extensions. This protocol defines the way that we send the data. You can remember the parts one after another delimited with the boundaries. This is what MIME is all about. All the parts of the message are sent sequntialy delimited with the boundary defined.
So i sent a message containing a body saying “this is the body of the message” and i attached a file, with the filename “foo.txt” and the content “bar”. Let’s take a peek.
This is a multi-part message in MIME format.
--------------070003020608000203050002
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
this is the body of the message.
--------------070003020608000203050002
Content-Type: text/plain;
name="foo.txt"
Content-Transfer-Encoding: base64
Content-Disposition: inline;
filename="foo.txt"
YmFy
--------------070003020608000203050002--
.
As you can see we have two parts in the message, the message body, sent first, and the attachment sent second. But hey! Where is the content of the file! We were expecting “bar”, instead we got “YmFy”. What happened? Huh. Notice the header of the part that says Content-Transfer-Encoding: base64? This means that the encoding is different than the one that the console i used supports. All we need to do is the conversion from the base64 to the 7-bit we got on the plain message on the previous part.
This is an overview of the POP protocol and a small howto for it. If you want full description and details over the protocol then you can always see at the RFC-1939.
The POP protocol is the easiest way of accessing your mail anywhere. The other server used for such cause is IMAP. This one is more complicated. It has much more many options and parameters. I will try to briefly show you how things work but i would stringly suggest that if you just want to make a small script or just play around go with POP. If you want to make a serious program/script then first of all you are in the wrong place and secondly you should be reading the IMAP rfc, RFC-3501.
If you are still here then let’s take a small overview of the protocol. Once more it’s a simple server with the difference that the command set is much more complicated. This is because the IMAP protocol gives advanced control over you email boxes and emails. This means that it supports multiple folders and message flaging. Under the folder inbox can be the folders Drafts, Sent etc etc. Also a message can have statuses, meaning flags, such as Answered, Flaged, Drafts etc.
The basic procedure is similar to POP extending the capabilities. You open a connection, log in to the server, retrieve a list of the messages over the various folders and then retrieve the messages over the varies flags. Let’s see a dump of a small session i made with the IMAP server retrieving the same message as before with the POP.
> telnet myserver.com 143
Trying IP...
Connected to myserver.com.
Escape character is '^]'.
* OK [CAPABILITY IMAP4rev1 UIDPLUS CHILDREN NAMESPACE THREAD=ORDEREDSUBJECT THREAD=REFERENCES SORT QUOTA IDLE ACL ACL2=UNION STARTTLS] Courier-IMAP ready. Copyright 1998-2005 Double Precision, Inc. See COPYING for distribution information.
. login username pass
. OK LOGIN Ok.
. list "" "*"
* LIST (\HasNoChildren) "." "INBOX.mailboxlist"
* LIST (\HasNoChildren) "." "INBOX.Sent"
* LIST (\HasNoChildren) "." "INBOX.Trash"
* LIST (\Marked \HasChildren) "." "INBOX"
. OK LIST completed
. status INBOX (messages)
* STATUS "INBOX" (MESSAGES 14)
. OK STATUS Completed.
. select INBOX
* FLAGS (\Draft \Answered \Flagged \Deleted \Seen \Recent)
* OK [PERMANENTFLAGS (\* \Draft \Answered \Flagged \Deleted \Seen)] Limited
* 14 EXISTS
* 14 RECENT
* OK [UIDVALIDITY 1179162644] Ok
* OK [MYRIGHTS "acdilrsw"] ACL
. OK [READ-WRITE] Ok
. fetch 12 rfc822
* 12 FETCH (RFC822 {770}
Return-path: from_mail_here
Envelope-to: to_mail_here
Delivery-date: Sun, 09 Mar 2008 00:54:45 +0200
Received: from [from_ip] (helo=[client_ip])
by sender_smtp with esmtpsa (TLSv1:AES256-SHA:256)
(Exim 4.68)
(envelope-from from_mail)
id 1JY7wD-0006cn-8A; Sun, 09 Mar 2008 00:54:45 +0200
Message-ID: 47D3191C.7010301@myserver.com
Date: Sun, 09 Mar 2008 00:54:20 +0200
From: name mail
User-Agent: Thunderbird 2.0.0.12 (Windows/20080213)
MIME-Version: 1.0
To: to_emails
Subject: last test i hope
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
last test
)
. OK FETCH completed.
. logout
* BYE Courier-IMAP server shutting down
. OK LOGOUT completed
Connection closed by foreign host.
As you can see the result was the same but the process was much more complicated. I connected, logged in, listed my mailboxes, selected the inbox, fetched the 12th message according to the RFC-822 and finaly closed the connection. Two pointers here. First of all the RFC-822 is the protocol “STANDARD FOR THE FORMAT OF ARPA INTERNET TEXT MESSAGES”. It actualy describes how the messages will be sent using mail services. Secondly, you can see that when fetching the mail i said which protocol i use. I could have retreived only the headers of the message by issuing . fetch 12 rfc822.header instead of the simple . fetch 12 rfc822. There are much more options and things to do. Also, listing the mails has many options according to the flags given to each message.
In general, the main difference between IMAP and POP is that with the second one you just retreive your messages and all the processing on them is done client side where with IMAP the process / categorizing / flaging / organizing etc is done on the server. In other words, a mail client like Outlook, Thundebird, Eudora, The Bat! or even Pine and The Mutt is pointless to use IMAP where a webmail, like Horde and Squirel, ought to use it.
This does it for receiving your email. But what about sending? Well here comes the SMTP servers. The process is once more very simple and similar to all the above. The difference is that we do it the other way round. The server is here to listen for data instead of sending the data. A brief description of the process is that we open the connection to the server, enter the from address, the to address, the message and then close the connection. Here is a dump of a session i made.
> telnet mailserver.com 25
Trying IP...
Connected to mailserver.com.
Escape character is '^]'.
220-mailserver.com ESMTP Exim 4.68 #1 Sun, 09 Mar 2008 18:58:31 +0200
220-We do not authorize the use of this system to transport unsolicited,
220 and/or bulk e-mail.
HELO mydomainname.com
250 mailserver.com Hello mydomainname.com [my domain ip]
MAIL FROM: your_email
250 OK
RCPT TO: target_email
250 Accepted
DATA
354 Enter message, ending with "." on a line by itself
Subject: test send through SMTP
This is a test body from SMTP
.
250 OK id=1JYOs6-0006uT-GE
quit
221 mailserver.cm=om closing connection
Connection closed by foreign host.
A couple of pointers. First of all, when you do a HELO mydomainname.com you actualy tell the SMTP who you are. Then the rest of the process is trivial. But there is a tricky spot here. What if the SMTP requires login? Well the process changes a little bit. What you have to do is summarized in the next few lines:
telnet myserver.com 25
Trying IP...
Connected to myserver.com
220 smtp106.prem.mail.ac4.yahoo.com ESMTP
HELO (just say hi)
250 smtp106.prem.mail.ac4.yahoo.com
AUTH LOGIN (start the login process)
334 VXNlcm5hbWU6 (a number given by the server)
USERNAME in base64 encoding
334 UGFzc3dvcmQ6 (response by the server)
PASSWORD id base64 encoding
either ok response or 535 authorization failed (#5.7.0)
If the response is ok the proceed as above with your message. If not then… bummer! You have to try again. If you want to convert your string on a base64 encoding you can do it here.
All in all, i think i pretty much covered the subject, as a small walkthrough that is. Your comments are not only welcomed but also a must so this tutorial can get even better. Hope i helped!
PS: I know there is a hole in all this because i did not cover the HTTPS protocol over the http and mail. Don’t worry, i will pretty soon 😉