Memcache

Summary
memcached

First, you start up the memcached daemon on as many spare machines as you have. The daemon has no configuration file, just a few command line options, only 3 or 4 of which you'll likely use:

./memcached -d -m 2048 -l 10.0.0.40 -p 11211

This starts memcached up as a daemon, using 2GB of memory, and listening on IP 10.0.0.40, port 11211. Because a 32-bit process can only address 4GB of virtual memory (usually significantly less, depending on your operating system), if you have a 32-bit server with 4-64GB of memory using PAE you can just run multiple processes on the machine, each using 2 or 3GB of memory.


 * memcached FAQ
 * Fastest languages for memcached


 * ACID
 * Create, read, update and delete

Memcache Protocol

What are some limits in memcached I might hit?: The simple limits you will probably see with memcache are the key and item size limits. Keys are restricted to 250 characters. Stored data cannot exceed 1 megabyte in size, since that is the largest typical slab size.

Yum Method
RPMForge Method: (version 1.4.5-1.el5.rf as of 2010.10.25)
 * See RPMforge for RPMforge installation
 * Yum method info provided here: http://serverfault.com/questions/342826/problems-installing-memcached

yum install memcached --enablerepo=rpmforge --enablerepo=rpmforge-extras
 * 1) x64 only - to fix x64 issue with SSLeay:
 * 2) wget http://pkgs.repoforge.org/perl-Net-SSLeay/perl-Net-SSLeay-1.36-1.el5.rfx.x86_64.rpm
 * 3) rpm -Uvh perl-Net-SSLeay-1.36-1.el5.rfx.x86_64.rpm


 * 1) to fix: Starting memcached: chown: cannot access `/var/run/memcached': No such file or directory
 * 2) touch /var/run/memcached

/etc/sysconfig/memcached (original): PORT="11211" USER="nobody" MAXCONN="1024" CACHESIZE="64" OPTIONS=""

/etc/sysconfig/memcached (modified, with cachesize in megabytes): PORT="11211" USER="nobody" MAXCONN="1024" CACHESIZE="128" OPTIONS="-l 127.0.0.1"
 * 1) in megabytes:
 * 1) additional options, specify address to listen on

Start service: chkconfig memcached on service memcached start


 * 1) ?eh? yum install php-eaccelerator

Manual Method
Installing memcached

Download Libevent: mkdir -p ~/.src ; cd ~/.src wget http://www.monkey.org/~provos/libevent-1.3e.tar.gz tar -zvxf libevent-1.3e.tar.gz cd libevent-1.3e ./configure; make; make install;

ln -s /usr/local/lib/libevent-1.3e.so.1 /lib/libevent-1.3e.so.1 ln -s /usr/local/lib/libevent-1.3e.so.1 /lib64/libevent-1.3e.so.1
 * 1) 32bit:
 * 1) 64bit:

Download memcached: mkdir -p ~/.src ; cd ~/.src wget http://www.danga.com/memcached/dist/memcached-1.2.2.tar.gz tar -zxvf memcached-1.2.2.tar.gz cd memcached-1.2.2 ./configure; make; make install;

Start memcached: memcached -d -u nobody -m 512 127.0.0.1 -p 11211

---

Often libevent.so cannot be found when executing memcache. A useful command LD_DEBUG, is very helpful to determine where libraries are being loaded from. LD_DEBUG=help memcached -v LD_DEBUG=libs memcached -v 2>&1 > /dev/null | less 18990: find library=libevent-1.3b.so.1 [0]; searching ... 18990: trying file=/usr/lib/libevent-1.3b.so.1 18990: memcached: error while loading shared libraries: libevent-1.3b.so.1: cannot open shared object file: No such file or directory
 * 1) help

Simply place the library where memcached will find it and execute memcached: ln -s /usr/local/lib/libevent-1.3e.so.1 /lib/libevent-1.3e.so.1 ln -s /usr/local/lib/libevent-1.3e.so.1 /lib64/libevent-1.3e.so.1
 * 1) 64bit:

Testing memcache
How to install memcache on Debian Etch

Quick stats check: echo -e "stats\nquit" | nc 127.0.0.1 11211

Testing the memcached daemon with telnet: telnet 1.2.3.4 11211

The commands look like this: set   \r\n

Example: set test2 1 0 2 ab STORED

get test2 VALUE test2 1 2 ab END

set test4 1 0 2 abcde CLIENT_ERROR bad data chunk ERROR

To see statistics: stats

To exit: quit

The flag is an arbitrary number that you can use in your client logic. It is intended to be metadata that you can assign to each cached object.

I set the exptime to zero (never expire)

the bytes to the number of characters I want to store. If the bytes and the number of characters don't match, an error occurs.

For other commands see protocol.txt

An example of how to get the stats using a bash script

Stats Check
Dump all stats: echo -e "stats\nquit" | nc 127.0.0.1 11211

Get important items: echo -e "stats\nquit" | nc 127.0.0.1 11211 | grep -e "\(uptime\|get_hits\|get_misses\|bytes \|limit_maxbytes\)"
 * 1) uptime - Number of seconds this server has been running
 * 2) get_hits - request that are in cache
 * 3) get_misses - requests that weren't in cache
 * 4) limit_maxbytes - total max bytes can store
 * 5) bytes - current bytes in cache

Stats Protocol
Name             Type     Meaning -- pid              32u      Process id of this server process uptime           32u      Number of seconds this server has been running time             32u      current UNIX time according to the server version          string   Version string of this server pointer_size     32       Default size of pointers on the host OS                           (generally 32 or 64) rusage_user      32u:32u  Accumulated user time for this process (seconds:microseconds) rusage_system    32u:32u  Accumulated system time for this process (seconds:microseconds) curr_items       32u      Current number of items stored by the server total_items      32u      Total number of items stored by this server ever since it started bytes            64u      Current number of bytes used by this server to store items curr_connections 32u      Number of open connections total_connections 32u     Total number of connections opened since the server started running connection_structures 32u Number of connection structures allocated by the server cmd_get          64u      Cumulative number of retrieval requests cmd_set          64u      Cumulative number of storage requests get_hits         64u      Number of keys that have been requested and found present get_misses       64u      Number of items that have been requested and not found evictions        64u      Number of valid items removed from cache to free memory for new items bytes_read       64u      Total number of bytes read by this server from network bytes_written    64u      Total number of bytes sent by this server to                            network limit_maxbytes   32u      Number of bytes this server is allowed to                           use for storage. threads          32u      Number of worker threads requested. (see doc/threads.txt)

Source: Protocol - http://code.sixapart.com/svn/memcached/trunk/server/doc/protocol.txt

Slab Size
What are some limits in memcached I might hit?:
 * "The simple limits you will probably see with memcache are the key and item size limits. Keys are restricted to 250 characters. Stored data cannot exceed 1 megabyte in size, since that is the largest typical slab size."

[http://www.socialtext.net/memcached/index.cgi?faq#how_does_memcached_s_memory_allocation_work_why_not_use_malloc_free_why_the_hell_does_it_use_slabs How does memcached's memory allocation work? Why not use malloc/free!? Why the hell does it use slabs!?]:
 * "Actually, it's a compile time option. The default is to use the internal slab allocator. You really really want to use the built-in slab allocator. At first memcached did just use malloc/free for everything. However this does not play very well with OS memory managers. You get fragmentation, and your OS ends up spending more time trying to find contiguous blocks of memory to feed malloc than it does running the memcached process. If you disagree, of course you're free to try malloc! just don't complain on the lists ;)
 * The slab allocator was built to work around this. Memory is allocated in chunks internally and constantly reused. Since memory is broken into different size slabs, you do waste memory if your items do not fit perfectly into the slab the server chooses to put it in. This has enjoyed considerable efficiency improvements by Steven Grimm.
 * Some older posts about the slab changes (power of n vs power of 2), and some tradeoffs are on the mailing list:
 * http://lists.danga.com/pipermail/memcached/2006-May/002163.html
 * http://lists.danga.com/pipermail/memcached/2007-March/003753.html
 * And if you'd like to attempt to use malloc/free and see how it works, you may define 'USE_SYSTEM_MALLOC' in build process. It might not be tested very well, so getting developer support for it is unlikely."
 * http://lists.danga.com/pipermail/memcached/2007-March/003753.html
 * And if you'd like to attempt to use malloc/free and see how it works, you may define 'USE_SYSTEM_MALLOC' in build process. It might not be tested very well, so getting developer support for it is unlikely."
 * And if you'd like to attempt to use malloc/free and see how it works, you may define 'USE_SYSTEM_MALLOC' in build process. It might not be tested very well, so getting developer support for it is unlikely."

References:
 * single object size limit
 * "The maximum object size is a few bytes under 1MB. You're trying to set about 250k more than it can take."
 * "You can recompile memcached with a higher POWER_LARGEST if you need more than 1MB. It's in slabs.c at the top.  Make it 21 to get 2MB max, or 22 for 4MB max, etc."


 * SERVER_ERROR object too large for cache
 * "The maximum size is 1 megabyte. It can be increased; I believe there was a patch (against the old 1.1.12 version) to do it. The trick is that memcached's memory manager allocates memory in 1MB chunks, so increasing the maximum is not just a matter of changing a "maximum object size" setting somewhere. But it is certainly doable."


 * new memcached memory layout
 * "I propose all objects are stored as a linked list of 64 byte chunks."


 * Memory management
 * "We've talked internally about eventually moving away from the slab-class-based system. It should be possible to run a job in the background (especially in the multithreaded version, but even in the single-threaded one) to shuffle objects around in memory to pack them optimally. If you separate the fixed-size headers from the item data, you can even do away with byte-alignment-based waste. That would give you 100% memory efficiency for any arbitrary mix of item sizes, and true LRU behavior across the whole cache, at the cost of greater CPU consumption."


 * Patch - Configurable slab size and large object support
 * memcached overrides memory limit
 * Patch - Increased memory efficiency

Startup Scripts
Daemon scripts for memcached:

touch /etc/memcached.conf cat > /etc/memcached.conf

-m 16 -p 11211 -u nobody -l 127.0.0.1
 * 1) Memory a usar
 * 1) default port
 * 1) user to run daemon nobody/apache/www-data
 * 1) only listen locally

touch /etc/init.d/memcached chmod +x /etc/init.d/memcached cat > /etc/init.d/memcached

. /etc/rc.d/init.d/functions PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin DAEMON=/usr/local/bin/memcached DAEMONBOOTSTRAP=/usr/local/bin/start-memcached DAEMONCONF=/etc/memcached.conf NAME=memcached DESC=memcached PIDFILE=/var/run/$NAME.pid [ -x $DAEMON ] || exit 0 [ -x $DAEMONBOOTSTRAP ] || exit 0 RETVAL=0 start { echo -n $"Starting $DESC: " daemon $DAEMONBOOTSTRAP $DAEMONCONF RETVAL=$? [ $RETVAL -eq 0 ] && touch $PIDFILE echo return $RETVAL } stop { echo -n $"Shutting down $DESC: " killproc $NAME RETVAL=$? echo [ $RETVAL -eq 0 ] && rm -f $PIDFILE return $RETVAL } case "$1" in start) start  ;; stop) stop ;; restart|reload) stop  start  RETVAL=$?  ;; status) status $prog RETVAL=$? ;; *) echo $"Usage: $0 {start|stop|restart|status}"  exit 1 esac exit $RETVAL
 * 1) !/bin/bash
 * 2) memcached    This shell script takes care of starting and stopping
 * 3)              standalone memcached.
 * 4) chkconfig: - 80 12
 * 5) description: memcached is a high-performance, distributed memory
 * 6)              object caching system, generic in nature, but
 * 7)              intended for use in speeding up dynamic web
 * 8)              applications by alleviating database load.
 * 9) processname: memcached
 * 10) config: /etc/memcached.conf
 * 11) Source function library.
 * 1) config: /etc/memcached.conf
 * 2) Source function library.
 * 1) See how we were called.

touch /usr/local/bin/start-memcached chmod +x /usr/local/bin/start-memcached cat > /usr/local/bin/start-memcached

use strict; if ($> != 0 and $< != 0) { print STDERR "Only root wants to run start-memcached.\n"; exit; } my $etcfile = shift || "/etc/memcached.conf"; my $params = []; my $etchandle; my $memcached = "/usr/local/bin/memcached"; my $pidfile = "/var/run/memcached.pid"; my $fd_reopened = "/dev/null"; sub handle_logfile { my ($logfile) = @_; $fd_reopened = $logfile; } sub reopen_logfile { my ($logfile) = @_; open *STDERR, ">>$logfile"; open *STDOUT, ">>$logfile"; open *STDIN, ">>/dev/null"; $fd_reopened = $logfile; } my $conf_directives = { "logfile" => \&handle_logfile }; if (open $etchandle, $etcfile) { foreach my $line (<$etchandle>) { $line =~ s/\#.*//go; $line = join ' ', split ' ', $line; next unless $line; next if $line =~ /^\-[dh]/o; if ($line =~ /^[^\-]/o) { my ($directive, $arg) = $line =~ /^(.*?)\s+(.*)/; $conf_directives->{$directive}->($arg); next; } push @$params, $line; } } unshift @$params, "-u root" unless (grep $_ eq '-u', @$params); $params = join " ", @$params; if (-e $pidfile) { open PIDHANDLE, "$pidfile"; my $localpid = ; close PIDHANDLE; chomp $localpid; if (-d "/proc/$localpid") { print STDERR "memcached is already running.\n"; exit; } else { `rm -f $localpid`; } } my $pid = fork; if ($pid == 0) { reopen_logfile($fd_reopened); exec "$memcached $params"; exit(0); } elsif (open PIDHANDLE,">$pidfile") { print PIDHANDLE $pid; close PIDHANDLE; } else { print STDERR "Can't write pidfile to $pidfile.\n"; }
 * 1) !/usr/bin/perl -w
 * 2) start-memcached
 * 3) 2003/2004 - Jay Bonci 
 * 4) This script handles the parsing of the /etc/memcached.conf file
 * 5) and was originally created for the Debian distribution.
 * 6) Anyone may use this little script under the same terms as
 * 7) memcached itself.
 * 1) This script assumes that memcached is located at /usr/bin/memcached, and
 * 2) that the pidfile is writable at /var/run/memcached.pid
 * 1) If we don't get a valid logfile parameter in the /etc/memcached.conf file,
 * 2) we'll just throw away all of our in-daemon output. We need to re-tie it so
 * 3) that non-bash shells will not hang on logout. Thanks to Michael Renner for
 * 4) the tip
 * 1) This is set up in place here to support other non -[a-z] directives

service memcached restart chkconfig memcached on

An alternative startup script:
 * Memcached startup files for Red Hat (RHEL)

Bash Stat Command
Bash Stat Command:


 * 1) !/bin/bash
 * 2) Make a 'stat' command on a memcache daemon server
 * 3) Outputs to a pipe for later using with syslog-ng
 * 4) alexandre.abreu at gmail.com
 * 1) alexandre.abreu at gmail.com
 * 1) alexandre.abreu at gmail.com

PATH=/bin:/usr/bin:/sbin:/usr/sbin SERVER=localhost PORT=11111 FIFO=/export/logs/memcache.fifo BUFF="memcache: "

[ -p $FIFO ] || mkfifo $FIFO || exit 1

exec 666<>/dev/tcp/$SERVER/$PORT || exit 1

append { BUFF="$BUFF $@" [ "$1" == "uptime" ] && RUNTIME=`echo "scale=3; $2/60" | bc`

[ "$1" == "get_hits" ] && { rate=`echo "scale=3; $2/$RUNTIME" | bc` rate=`printf "%.3f" $rate` append hitrate $rate }   [ "$1" == "get_misses" ] && { rate=`echo "scale=3; $2/$RUNTIME" | bc` rate=`printf "%.3f" $rate` append missrate $rate } }

echo "stats" >&666

while read line; do   line=`echo $line | sed 's|cntrl:||g'` set - $line append $2 $3 [ "$2" == "limit_maxbytes" ] && break done <&666

echo "$BUFF" > $FIFO echo "quit" >&666 exit 0

My Version:
 * 1) !/bin/bash
 * 2) Make a 'stat' command on a memcache daemon server
 * 3) Outputs to a pipe for later using with syslog-ng
 * 1) Outputs to a pipe for later using with syslog-ng

PATH=/bin:/usr/bin:/sbin:/usr/sbin SERVER=10.20.30.40 PORT=11211 FIFO=/tmp/memcache.fifo

[ -p $FIFO ] || mkfifo $FIFO || exit 1

exec 666<>/dev/tcp/$SERVER/$PORT || exit 1

echo "stats" >&666

while read line; do   echo $line; FLAG=`echo $line | cut -b 1-3` [ "$FLAG" == "END" ] && break done <&666

echo "quit" >&666 exit 0