Hello, World!, again!

When asked for advice on what to do as a fresh Ph.D. student a few months ago, I said confidently: “Create a blog and publish regularly.” ¹ So… two and a half years, a move to Hamburg, an excursion as a cryptography engineer, and a few new hobbies later, here’s the next post!

Joking aside, the last blog post was crucial to me as it paved the way toward a grant from the NLnet Foundation! NLnet heavily contributes to an open information society and helps to fix the Internet. I recommend looking at what NLnet can do for you and thinking about what you can do for NLnet!

What motivated this blog post is that I’m just about to conclude the imap-codec project. And what is the best way to make people look at it? Exactly: A blog post about IMAP!

Today, I’ll tell you everything I think is helpful to get up to speed with IMAP. This blog post will be a “gentle” introduction. The next one will be about everything that makes IMAP hard to tame.

Introduction

State of Mind

IMAP is a lot. It’s a stateful protocol that doesn’t lend itself to a simple implementation and can be intimidating to someone who is used to HTTP. The syntax is branched, subtle, and has various issues. It doesn’t help either that there are more than 70 extensions, with any of them potentially breaking your understanding of how IMAP operates. Take your time.

It may seem pretentious to use complicated phrases for simple concepts – I sometimes also joke about the term “string interpolation.” But phrases like “command continuation request,” “command completion result,” “non-synchronizing literal,” etc. provide clarity.

Lastly (and I know this may sound sarcastic): Try to accept things as they are. Many things are/were there for a reason. And those that aren’t/weren’t can’t be changed anyway.

Piece of Mind

But there is good news! When using imap-codec, you don’t need to care about syntax. imap-codec is based on imap-types, and it promises that everything you can create is valid. Try to create something in imap-types! If it works, it’s allowed.

Thus, let’s annotate all messages with the corresponding Rust Debug-print. As we will see, this makes it obvious what is supported and what is not.

Maybe the most significant change since the last blog post is that we now have imap-types that position itself as a candidate for a “standard library” for IMAP in Rust. The similarity to http is on purpose :-)

Speaking IMAP

The most efficient, most secure, and recommended way to use IMAP is through TLS on port 993. Non-encrypted connections are traditionally made on port 143 with the option to do an inline transition to TLS. This is called STARTTLS and should be avoided.

We will use a non-encrypted connection on port 1143 to make our life easier because most operating systems require special permissions when handling port numbers below 1024.

To follow along, you can start imap-codec’s tokio-server demo using …

cd imap-codec
cargo run -p tokio-server -- 127.0.0.1:1143

… and connect to it using …

nc -C 127.0.0.1 1143

We use Netcat with the -C parameter so that all newlines (\n) are encoded as “Internet newlines” (\r\n) as required by IMAP.

Warning: Don’t use Netcat to connect to anything remotely important, as nothing is encrypted. If you really want to experiment with a real server, you could use …

openssl s_client -verify_return_error -crlf -brief -connect <host>:<port>

… instead and connect through TLS.

Greeting

In contrast to, e.g., HTTP, the server sends the first message in IMAP. Thus, you should already see something like …

* OK [CAPABILITY IMAP4REV1 AUTH=PLAIN] IMAP server ready

Let’s use a comment for “bytes on the wire,” denote the role, e.g., the client (C: ) or server (S: ), and add a Debug-print to show the type instance:

// S: * OK [CAPABILITY IMAP4REV1 AUTH=PLAIN] IMAP server ready
Greeting {
    kind: Ok,
    code: Some(Capability([Imap4Rev1, Auth(AuthMechanism(Plain))]+)),
    text: Text("IMAP server ready"),
}

The type already tells us a bit about Greetings. Generally, when you encounter a * , this means “untagged.” We have yet to talk about tags, but greetings are never tagged. Thus, there is no tag field.

The information we can extract from the greeting is a kind, (possibly) a code, and a human-readable text. The greeting kind is usually Ok and signals that the server will serve our connection. But it could as well be… well… look into GreetingKind :-)

The Code is optional and may carry additional data. Here, it already tells us what capabilities the server supports.

Note the + at the end of the capabilities vector. This signals we have a NonEmptyVec.

If you are like me, you might have wondered how applications should parse IMAP server ready. They don’t. It’s just a human-readable Text that could be shown to a user, e.g., in case of an error. Let’s always use ... for brevity.

Note: You can toy around with your messages using …

cargo run --example=parse_{greeting,command,response}

… that will parse and Debug-print your input.

Capability agreement (again?)

If the server doesn’t send its capabilities – or the client doesn’t understand “codes in greetings” – it needs to ask for them using the CAPABILITY command. Let’s pretend we didn’t already know the capabilities and ask again:

// C: A1 CAPABILITY
Command { tag: Tag("A1"), body: Capability }
// S: * CAPABILITY IMAP4REV1 AUTH=PLAIN
Data(Capability([Imap4Rev1, Auth(AuthMechanism(Plain))]+))
// S: A1 OK ...
Status(
Ok {
    tag: Some(Tag("A1")),
        code: None,
        text: Text("..."),
    },
)

Now is a good time to talk about Tags. Unlike SMTP and POP3, an IMAP client can send multiple commands immediately, and the server can answer them out of order. Thus, we need a mechanism to match the command/response pairs.

In the above trace, the client used A1 as the command tag, and the server responded with two responses, Data and a Status. Here, the second response reflects the command tag.

Remember what we said about terminology?

Looking closely, the status tag is optional. In fact, status responses are “command completion results” and say something about a command’s result when tagged. Untagged status responses serve as an alternative form of server data and don’t say anything about a command.

Let’s also stress again that the server can return multiple responses. It’s even more complicated: No response is really “bound” to a command and can be emitted by the server for unrelated reasons. The server can also emit responses despite no “in-flight” command.

The following trace could be perfectly valid:

Command { .. } // Tagged with A1  // Command
Data(_)                           // Server data
Data(_)                           // Server data 
Data(_)                           // Server data
Status(_)      // Untagged        // Server data
Data(_)                           // Server data
Status(_)      // Tagged with A1  // Command completion result

// No in-flight command, but we get more server data.

Data(_)                           // Server data
Status(_)      // Untagged        // Server data

It’s helpful to think of an IMAP server as something that can send (almost) any response at any time. You always need to be prepared to receive a response. If you are interested in something particular, e.g., a specific email, you can ask the server for more responses using a command. But again: Asking doesn’t mean you will immediately get it. The server may tell you about other emails before handing you the requested information (so to say).

Note: The IMAP standard describes what server data the client can expect to receive before the command completion result. It may receive more than that but should also receive the “REQUIRED untagged response(s).”

Of course, the server MUST NOT send made-up tags. A command completion result without a command to conclude doesn’t make sense (except for interesting attacks :-))

Authentication

Now, let’s look at a login sequence. I hope it has become easier to read by now.

// C: A1 LOGIN alice password
Command {
    tag: Tag("A1"),
    body: Login {
        username: Atom(AtomExt("alice")),
        password: Atom(AtomExt("password")),
    },
}
// S: A1 OK ...
Status(
    Ok {
        tag: Some(Tag("A1")),
        code: None,
        text: Text("..."),
    },
)

IMAP has multiple ways to authenticate. There is a LOGIN and AUTHENTICATE command with similar purposes. The AUTHENTICATE command makes IMAP usable with authentication mechanisms defined by “SASL.” We can almost ignore SASL here. However, the base IMAP protocol requires you to implement the SASL AUTH=PLAIN mechanisms which would look like this …

// C: A1 AUTHENTICATE PLAIN
Command {
    tag: Tag("A1"),
    body: Authenticate { mechanism: AuthMechanism(Plain) },
}

Where are the username and password, you ask? Let’s figure it out in the next blog post :-)

Note: AUTH=LOGIN is a more verbose, less efficient, non-standardized mechanism invented for unknown reasons.

Note: If you want to connect to Gmail through IMAP, you need AUTH=XOAUTH2 or AUTH=OAUTH2. This requires registering an App with Google to create the necessary tokens. It’s not trivial and took me an hour to do using a Python script I downloaded from the Internet.

Unfortunately, you can’t speak IMAP with Google by any other means. It’s good for security but complicates testing.

Update: As pointed out by @csb6 and @sir, it’s still possible to authenticate with a password. First, make sure to activate 2FA. (You can use a security key to keep your phone number private.) Then, in the 2FA settings, scroll down to the bottom and generate an “app password” for email. You can use this as a usual password for your gmail account.

Folder selection

We want to read some emails now, but not so fast! We first need to SELECT a folder …

// C: A1 SELECT INBOX
Command { tag: Tag("A1"), body: Select { mailbox: Inbox } }
// S: * 3 EXISTS
Data(Exists(3))
// S: * 3 RECENT
Data(Recent(3))
// S: * OK [UNSEEN 1] ...
Status(Ok { tag: None, code: Some(Unseen(1)), text: Text("...") })
// S: * OK [UIDNEXT 8] ...
Status(Ok { tag: None, code: Some(UidNext(8)), text: Text("...") })
// S: A1 OK ...
Status(Ok { tag: Some(Tag("A1")), code: None, text: Text("...") })

Great, we got a gazillion responses and a positive command completion result. This means we are in the INBOX now. And we already know how many messages the INBOX holds due to the Exists response.

Fetching of email data

Let’s now FETCH the first email:

// C: A1 FETCH 1 (BODY[])
Command {
    tag: Tag("A1"),
    body: Fetch {
        sequence_set: SequenceSet([Single(Value(1))]+),
        macro_or_item_names: MessageDataItemNames(
            [
                BodyExt {
                    section: None,
                    partial: None,
                    peek: false,
                },
            ],
        ),
        uid: false,
    },
}

Why does one line result in such a complicated type, you ask?

Back answer: Did you know that an IMAP server needs to parse and understand the complete structure of every email it holds to serve partial information? Do you know GraphQL? IMAP already had it!

Besides having surprising consequences on end-to-end encrypted email, partial fetching allows, e.g., to fetch only all subjects in the inbox. This is really important to write good clients. You don’t want your email client to download your whole INBOX (and all attachments) before being able to show you a list of “from,” and “subject.”

You can request subjects only. Or specific headers. Or a particular MIME part.

Or …

• 1337 bytes
• starting at index 42
• of the header
• of the second MIME part
• of the first MIME part
• of the first email plus emails from sequence number 2 to the end
• without marking them as read

… like …

//    Tag
//    |    Fetch by UID
//    |    |         Which emails?
//    |    |         |      What exactly?
//    |    |         |      |
//    /--\ /-------\ /---\  /----------------------------\
// C: ABC1 UID FETCH 1,2:* (BODY.PEEK[1.2.HEADER]<42.1337>)
//                          \-------/ \--------/  \/ \--/
//                          |         |           |  |
//                          |         |           |  Count
//                          |         |           Start
//                          |         Part
//                          Body (w/o marking as read)
Command {
    tag: Tag("ABC1"),
    body: Fetch {
        sequence_set: SequenceSet(
            [
                Single(Value(1)),
                Range(Value(2), Asterisk),
            ]+,
        ),
        macro_or_item_names: MessageDataItemNames(
            [
                BodyExt {
                    section: Some(Header(Some(Part([1, 2, ]+)))),
                    partial: Some((42, 1337)),
                    peek: true,
                },
            ],
        ),
        uid: true,
    },
}                      

You get the idea! Hopefully, you are convinced you don’t want to write this by hand or parse it yourself. Have I already told you about imap-c… just kidding :-)

It doesn’t end here …

Well, yes, this blog post does end here. But I will release another one very soon™. In the next one, I will show you everything I avoided mentioning here. We will cover literals, authentication, and why designing a sound IMAP library is hard.

See you soon!