Six Apart, Ltd.

548 4th Street San Francisco CA 94107 USA aaron@serendipity.palo-alto.ca.us

Applications memcache memcached cache This memo explains the memcache binary protocol for informational purposes. Memcache is a high performance key-value cache. It is intentionally a dumb cache, optimized for speed only. Applications using memcache do not rely on it for data -- a persistent database with guaranteed reliability is strongly recommended -- but applications can run much faster when cached data is available in memcache.

Memcache is a high performance key-value cache. It is intentionally a dumb cache, optimized for speed only. Applications using memcache do not rely on it for data -- a persistent database with guaranteed reliability is strongly recommended -- but applications can run much faster when cached data is available in memcache. Memcache was originally written to make LiveJournal go faster. It now powers all of the fastest web sites that you love.

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in .

General format of a packet: Byte/ 0 | 1 | 2 | 3 | / | | | | |0 1 2 3 4 5 6 7|0 1 2 3 4 5 6 7|0 1 2 3 4 5 6 7|0 1 2 3 4 5 6 7| +---------------+---------------+---------------+---------------+ 0/ HEADER / / / / / / / +---------------+---------------+---------------+---------------+ 16/ COMMAND-SPECIFIC EXTRAS (as needed) / +/ (note length in th extras length header field) / +---------------+---------------+---------------+---------------+ m/ Key (as needed) / +/ (note length in key length header field) / +---------------+---------------+---------------+---------------+ n/ Value (as needed) / +/ (note length is total body length header field, minus / +/ sum of the extras and key length body fields) / +---------------+---------------+---------------+---------------+ Total 16 bytes

Request header: Byte/ 0 | 1 | 2 | 3 | / | | | | |0 1 2 3 4 5 6 7|0 1 2 3 4 5 6 7|0 1 2 3 4 5 6 7|0 1 2 3 4 5 6 7| +---------------+---------------+---------------+---------------+ 0| Magic | Opcode | Key length | +---------------+---------------+---------------+---------------+ 4| Extras length | Data type | Reserved | +---------------+---------------+---------------+---------------+ 8| Total body length | +---------------+---------------+---------------+---------------+ 12| Message ID | +---------------+---------------+---------------+---------------+ Total 16 bytes

Response header: Byte/ 0 | 1 | 2 | 3 | / | | | | |0 1 2 3 4 5 6 7|0 1 2 3 4 5 6 7|0 1 2 3 4 5 6 7|0 1 2 3 4 5 6 7| +---------------+---------------+---------------+---------------+ 0| Magic | Opcode | Status | +---------------+---------------+---------------+---------------+ 4| Extras length | Data type | Reserved | +---------------+---------------+---------------+---------------+ 8| Total body length | +---------------+---------------+---------------+---------------+ 12| Message ID | +---------------+---------------+---------------+---------------+ Total 16 bytes

Header fields: Magic number. Command code. Length in bytes of the text key that follows the command extras. Status of the response (non-zero on error). Length in bytes of the command extras. Reserved for future use (Sean is using this soon). Really reserved for future use (up for grabs). Length in bytes of extra + key + value. Will be copied back to you in the response. FIXME: Can this be used to organize packets?

Request packet for this protocol version Response packet for this protocol version Magic byte / version. For each version of the protocol, we'll use a different request/reponse value pair. This is useful for protocol analyzers to know what a packet is in isolation from which direction it is moving. Note that it is common to run a memcached instance on a host that also runs an application server. Such a host will both send and receive memcache packets. The version should hopefully correspond only to different meanings of the command byte. In an ideal world, we will not change the header format. As reserved bytes are given defined meaning, the protocol version / magic byte values should be incremented. Traffic analysis tools are encouraged to identify memcache packets and provide detailed interpretation if the magic bytes are recognized and otherwise to provide a generic breakdown of the packet. Note that the key and value positions can always be identified even if the magic byte or command opcode are not recognized.

Possible values of this two-byte field: No error Unknown command Key not found Key exists

Possible values of the one-byte field: Get Set Add Replace Delete Increment Decrement Quit Flush GetQ No-op Version

Possible values of the one-byte field: Raw bytes

MUST have extras. MUST have key. MUST NOT have value. 4 byte flags 8 byte data version check

Extra data for get/getq: Byte/ 0 | 1 | 2 | 3 | / | | | | |0 1 2 3 4 5 6 7|0 1 2 3 4 5 6 7|0 1 2 3 4 5 6 7|0 1 2 3 4 5 6 7| +---------------+---------------+---------------+---------------+ 0| Data version check | | | +---------------+---------------+---------------+---------------+ 8| Flags | +---------------+---------------+---------------+---------------+ Total 12 bytes

The get command gets a single key. The getq command is both mum on cache miss and quiet, holding its response until a non-quiet command is issued. You're not guaranteed a response to a getq cache hit until you send a non-getq command later, which uncorks the server which bundles up IOs to send to the client in one go. Clients should implement multi-get (still important for reducing network roundtrips!) as n pipelined requests, the first n-1 being getq, the last being a regular get. that way you're guaranteed to get a response, and you know when the server's done. you can also do the naive thing and send n pipelined gets, but then you could potentially get back a lot of "NOT_FOUND!" error code packets. alternatively, you can send 'n' getqs, followed by an 'echo' or 'noop' command.

MAY have extras (FIXME: Is it OK to issue a delete without extras?). MUST have key. MUST NOT have value. 4 byte expiration time

Extra data for delete: Byte/ 0 | 1 | 2 | 3 | / | | | | |0 1 2 3 4 5 6 7|0 1 2 3 4 5 6 7|0 1 2 3 4 5 6 7|0 1 2 3 4 5 6 7| +---------------+---------------+---------------+---------------+ 0| Expiration | +---------------+---------------+---------------+---------------+ Total 4 bytes

When allows you to 'reserve' a key. When 'when' is set for, say, ten seconds in the future, the 'add' and 'replace' operations will fail for that key until ten seconds from now. The 'set' operation will succeed regardless of any reserved deletes. FIXME: Is the reservation also cancelled? Say there's a delete with a 10 second hold. Two seconds later, an 'add' is received. It fails. Two second later, a 'set' is received. Is succeeds unconditionally. What if another 'add' is received two more seconds later (a total of six seconds since the original 10 second delete-hold, thus still within its purview).

MUST have extras. MUST have key. MUST have value. 4 byte flags 4 byte expiration time 8 byte data version check

Extra data for set/add/replace: Byte/ 0 | 1 | 2 | 3 | / | | | | |0 1 2 3 4 5 6 7|0 1 2 3 4 5 6 7|0 1 2 3 4 5 6 7|0 1 2 3 4 5 6 7| +---------------+---------------+---------------+---------------+ 0| Data version check | | | +---------------+---------------+---------------+---------------+ 8| Flags | +---------------+---------------+---------------+---------------+ 12| Expiration | +---------------+---------------+---------------+---------------+ Total 16 bytes

If the Data Version Check is present and nonzero, the set MUST succeed if the key exists and has a version identifier identical to the provided value, and MUST NOT succeed if the key does not exist or has a different version identifier. The set response packet will include the same values in all three fields. If the Data Version Check is zero, the set MUST succeed unconditionally. The set response packet will include idential values for flags and expiration, and a new value for Data Version Check, which the client SHOULD keep track of. The key MAY be reserved according to , causing the set to fail.

MUST NOT have extras. MUST NOT have key. MUST NOT have value. Used as a keep alive. Flushes outstanding getq's.

MUST have extras. MUST have key. MUST NOT have value. 8 byte value to add / subtract (FIXME: Is this unsigned?) 8 byte initial value (unsigned) 4 byte expiration time

Extra data for incr/decr: Byte/ 0 | 1 | 2 | 3 | / | | | | |0 1 2 3 4 5 6 7|0 1 2 3 4 5 6 7|0 1 2 3 4 5 6 7|0 1 2 3 4 5 6 7| +---------------+---------------+---------------+---------------+ 0| Amount to add | | | +---------------+---------------+---------------+---------------+ 8| Initial value | | | +---------------+---------------+---------------+---------------+ 16| Expiration | +---------------+---------------+---------------+---------------+ Total 20 bytes

incr/decr response body: Byte/ 0 | 1 | 2 | 3 | / | | | | |0 1 2 3 4 5 6 7|0 1 2 3 4 5 6 7|0 1 2 3 4 5 6 7|0 1 2 3 4 5 6 7| +---------------+---------------+---------------+---------------+ 0| 64-bit unsigned response. | | | +---------------+---------------+---------------+---------------+ Total 8 bytes

These commands will either add or remove the specified amount to the requested counter. If the counter does not exist, one of two things may happen: If the expiration value is all one-bits (0xffffffff), the operation will fail with NOT_FOUND. For all other expiration values, the operation will succeed by seeding the value for this key with the provided initial value to expire with the provided expiration time. Note that in the creation case, flags will be set to zero (FIXME: Should they be provided here as well?)

We start up our application, and it asks for the value associated with the 'Hello' key.

Get request: Byte/ 0 | 1 | 2 | 3 | / | | | | |0 1 2 3 4 5 6 7|0 1 2 3 4 5 6 7|0 1 2 3 4 5 6 7|0 1 2 3 4 5 6 7| +---------------+---------------+---------------+---------------+ 0| 0x80 | 0x00 | 5 in big endian (BE) | +---------------+---------------+---------------+---------------+ | 12 in BE | 0x00 | | +---------------+---------------+---------------+---------------+ | 17 in BE | +---------------+---------------+---------------+---------------+ | 0xDEADBEEF | +---------------+---------------+---------------+---------------+ 16| 0x00000000 | +---------------+---------------+---------------+---------------+ 24| 0xDECAF 0x15 0xBAD 0xC0FFEE | | | +---------------+---------------+---------------+---------------+ 28| 'H' 'e' 'l' 'l' | | 'o' | +---------------+ Total 33 bytes (16 header + 12 get-extras + 5 key)

Since nobody has set this key, it returns not found.

Get response: Byte/ 0 | 1 | 2 | 3 | / | | | | |0 1 2 3 4 5 6 7|0 1 2 3 4 5 6 7|0 1 2 3 4 5 6 7|0 1 2 3 4 5 6 7| +---------------+---------------+---------------+---------------+ 0| 0x81 | 0x00 | 0x0001 | +---------------+---------------+---------------+---------------+ | 0 in BE | 0x00 | | +---------------+---------------+---------------+---------------+ | 0 in BE | +---------------+---------------+---------------+---------------+ | 0xDEADBEEF | +---------------+---------------+---------------+---------------+ Total 16 bytes

Well, looks like we need to set the key! Let's set it to expire on December 15, 2007 at 9:51:09 PM.

Set request: Byte/ 0 | 1 | 2 | 3 | / | | | | |0 1 2 3 4 5 6 7|0 1 2 3 4 5 6 7|0 1 2 3 4 5 6 7|0 1 2 3 4 5 6 7| +---------------+---------------+---------------+---------------+ 0| 0x80 | 0x01 | 5 in BE | +---------------+---------------+---------------+---------------+ | 16 in BE | 0x00 | | +---------------+---------------+---------------+---------------+ | 26 in BE | +---------------+---------------+---------------+---------------+ | 0xDA7ABA5E | +---------------+---------------+---------------+---------------+ 16| 0x00000000 | +---------------+---------------+---------------+---------------+ 20| 0xDCCB4674 | +---------------+---------------+---------------+---------------+ 24| 0xDECAF 0x15 0xBAD 0xC0FFEE | | | +---------------+---------------+---------------+---------------+ 32| 'H' 'e' 'l' 'l' | | 'o' | 'W' 'o' 'r' | | 'l' 'd' | +---------------+---------------+ Total 42 bytes (16 header + 16 set-extras + 5 key + 5 value)

The set succeeds.

Set response: Byte/ 0 | 1 | 2 | 3 | / | | | | |0 1 2 3 4 5 6 7|0 1 2 3 4 5 6 7|0 1 2 3 4 5 6 7|0 1 2 3 4 5 6 7| +---------------+---------------+---------------+---------------+ 0| 0x81 | 0x01 | 0x0000 | +---------------+---------------+---------------+---------------+ | 16 in BE | 0x00 | | +---------------+---------------+---------------+---------------+ | 16 in BE | +---------------+---------------+---------------+---------------+ | 0xDA7ABA5E | +---------------+---------------+---------------+---------------+ 16| 0x00000000 | +---------------+---------------+---------------+---------------+ 20| 0xDCCB4674 | +---------------+---------------+---------------+---------------+ 24| 0xDECAF 0x15 0xBAD 0xC0FFEE | | | +---------------+---------------+---------------+---------------+ Total 32 bytes (16 header + 16 set-extras)

If the original get request is sent again, the key would be found.

Get response: Byte/ 0 | 1 | 2 | 3 | / | | | | |0 1 2 3 4 5 6 7|0 1 2 3 4 5 6 7|0 1 2 3 4 5 6 7|0 1 2 3 4 5 6 7| +---------------+---------------+---------------+---------------+ 0| 0x81 | 0x00 | 0x00 | | +---------------+---------------+---------------+---------------+ | 12 in BE | 0x00 | | +---------------+---------------+---------------+---------------+ | 17 in BE | +---------------+---------------+---------------+---------------+ | 0xDEADBEEF | +---------------+---------------+---------------+---------------+ 16| 0xDCCB4674 | +---------------+---------------+---------------+---------------+ 24| 0xDECAF 0x15 0xBAD 0xC0FFEE | | | +---------------+---------------+---------------+---------------+ 28| 'W' 'o' 'r' 'l' | | 'd' | +---------------+ Total 33 bytes (16 header + 12 get-extras + 5 value)

Memcache has no authentication or security layers whatsoever. It is RECOMMENDED that memcache be deployed strictly on closed, protected, back-end networks within a single data center, within a single cluster of servers, or even on a single host, providing shared caching for multiple applications. Memcache MUST NOT be made available on a public network.

Danga Interactive http://www.livejournal.com/

Thanks to Brad Fitzpatrick, Anatoly Vorobey, Steven Grimm, and Dustin Sallings, for their work on the memcached server. Thanks to Sean Chittenden, Jonathan Steinert, Brian Aker, Evan Martin, Nathan Neulinger, Eric Hodel, Michael Johnson, Paul Querna, Jamie McCarthy, Philip Neustrom, Andrew O'Brien, Josh Rotenberg, Robin H. Johnson, Tim Yardley, Paolo Borelli, Eli Bingham, Jean-Francois Bustarret, Paul G, Paul Lindner, Alan Kasindorf, Chris Goffinet, Tomash Brechko, and others for their work reporting bugs and maintaining memcached client libraries and bindings in many languages.