Memcache

From Omnia
(Redirected from Memcached)
Jump to navigation Jump to search

Summary

memcached

First, you start up the memcached daemon on as many spare machines as you have. The daemon has no configuration file, just a few command line options, only 3 or 4 of which you'll likely use:

./memcached -d -m 2048 -l 10.0.0.40 -p 11211

This starts memcached up as a daemon, using 2GB of memory, and listening on IP 10.0.0.40, port 11211. Because a 32-bit process can only address 4GB of virtual memory (usually significantly less, depending on your operating system), if you have a 32-bit server with 4-64GB of memory using PAE you can just run multiple processes on the machine, each using 2 or 3GB of memory.



Memcache Protocol


What are some limits in memcached I might hit?: [1]

The simple limits you will probably see with memcache are the key and item size limits. Keys are
restricted to 250 characters. Stored data cannot exceed 1 megabyte in size, since that is the
largest typical slab size.

Installation

Yum Method

RPMForge Method: (version 1.4.5-1.el5.rf as of 2010.10.25)

yum install memcached --enablerepo=rpmforge --enablerepo=rpmforge-extras

# x64 only - to fix x64 issue with SSLeay:
# wget http://pkgs.repoforge.org/perl-Net-SSLeay/perl-Net-SSLeay-1.36-1.el5.rfx.x86_64.rpm
# rpm -Uvh perl-Net-SSLeay-1.36-1.el5.rfx.x86_64.rpm
# to fix: Starting memcached: chown: cannot access `/var/run/memcached': No such file or directory
# touch /var/run/memcached

/etc/sysconfig/memcached (original):

PORT="11211"
USER="nobody"
MAXCONN="1024"
CACHESIZE="64"
OPTIONS=""

/etc/sysconfig/memcached (modified, with cachesize in megabytes):

PORT="11211"
USER="nobody"
MAXCONN="1024"
# in megabytes:
CACHESIZE="128"
# additional options, specify address to listen on
OPTIONS="-l 127.0.0.1"

Start service:

chkconfig memcached on
service memcached start
  1. ?eh? yum install php-eaccelerator

Manual Method

Installing memcached

Download Libevent:

mkdir -p ~/.src ; cd ~/.src
wget http://www.monkey.org/~provos/libevent-1.3e.tar.gz

tar -zvxf libevent-1.3e.tar.gz
cd libevent-1.3e
./configure; make; make install;
#32bit:
ln -s /usr/local/lib/libevent-1.3e.so.1 /lib/libevent-1.3e.so.1
#64bit:
ln -s /usr/local/lib/libevent-1.3e.so.1 /lib64/libevent-1.3e.so.1

Download memcached:

mkdir -p ~/.src ; cd ~/.src
wget http://www.danga.com/memcached/dist/memcached-1.2.2.tar.gz

tar -zxvf memcached-1.2.2.tar.gz
cd memcached-1.2.2
./configure; make; make install;

Start memcached:

memcached -d -u nobody -m 512 127.0.0.1 -p 11211

---

Often libevent.so cannot be found when executing memcache. A useful command LD_DEBUG, is very helpful to determine where libraries are being loaded from.

# help
LD_DEBUG=help memcached -v
 
LD_DEBUG=libs memcached -v 2>&1 > /dev/null | less
18990: find library=libevent-1.3b.so.1 [0]; searching
...
18990: trying file=/usr/lib/libevent-1.3b.so.1
18990:
memcached: error while loading shared libraries: libevent-1.3b.so.1: cannot open shared object file: No such file or directory

Simply place the library where memcached will find it and execute memcached:

ln -s /usr/local/lib/libevent-1.3e.so.1 /lib/libevent-1.3e.so.1
#64bit:
ln -s /usr/local/lib/libevent-1.3e.so.1 /lib64/libevent-1.3e.so.1

Testing memcache

How to install memcache on Debian Etch

Quick stats check:

echo -e "stats\nquit" | nc 127.0.0.1 11211

Testing the memcached daemon with telnet:

telnet 1.2.3.4 11211

The commands look like this:

set <key> <flag> <exptime> <bytes>\r\n

Example:

set test2 1 0 2
ab
STORED
get test2
VALUE test2 1 2
ab
END
set test4 1 0 2
abcde
CLIENT_ERROR bad data chunk
ERROR

To see statistics:

stats

To exit:

quit

The flag is an arbitrary number that you can use in your client logic. It is intended to be metadata that you can assign to each cached object.

I set the exptime to zero (never expire)

the bytes to the number of characters I want to store. If the bytes and the number of characters don't match, an error occurs.

For other commands see protocol.txt

An example of how to get the stats using a bash script

Stats Check

Dump all stats:

echo -e "stats\nquit" | nc 127.0.0.1 11211

Get important items:

echo -e "stats\nquit" | nc 127.0.0.1 11211 | grep -e "\(uptime\|get_hits\|get_misses\|bytes \|limit_maxbytes\)"
# uptime - Number of seconds this server has been running
# get_hits - request that are in cache
# get_misses - requests that weren't in cache
# limit_maxbytes - total max bytes can store
# bytes - current bytes in cache

Stats Protocol

Name              Type     Meaning
----------------------------------
pid               32u      Process id of this server process
uptime            32u      Number of seconds this server has been running
time              32u      current UNIX time according to the server
version           string   Version string of this server
pointer_size      32       Default size of pointers on the host OS
                           (generally 32 or 64)
rusage_user       32u:32u  Accumulated user time for this process 
                           (seconds:microseconds)
rusage_system     32u:32u  Accumulated system time for this process 
                           (seconds:microseconds)
curr_items        32u      Current number of items stored by the server
total_items       32u      Total number of items stored by this server 
                           ever since it started
bytes             64u      Current number of bytes used by this server 
                           to store items
curr_connections  32u      Number of open connections
total_connections 32u      Total number of connections opened since 
                           the server started running
connection_structures 32u  Number of connection structures allocated 
                           by the server
cmd_get           64u      Cumulative number of retrieval requests
cmd_set           64u      Cumulative number of storage requests
get_hits          64u      Number of keys that have been requested and 
                           found present
get_misses        64u      Number of items that have been requested 
                           and not found
evictions         64u      Number of valid items removed from cache
                           to free memory for new items
bytes_read        64u      Total number of bytes read by this server 
                           from network
bytes_written     64u      Total number of bytes sent by this server to 
                           network
limit_maxbytes    32u      Number of bytes this server is allowed to
                           use for storage. 
threads           32u      Number of worker threads requested.
                           (see doc/threads.txt)

Source: Protocol - http://code.sixapart.com/svn/memcached/trunk/server/doc/protocol.txt

Slab Size

What are some limits in memcached I might hit?:

"The simple limits you will probably see with memcache are the key and item size limits. Keys are restricted to 250 characters. Stored data cannot exceed 1 megabyte in size, since that is the largest typical slab size."

How does memcached's memory allocation work? Why not use malloc/free!? Why the hell does it use slabs!?:

"Actually, it's a compile time option. The default is to use the internal slab allocator. You really really want to use the built-in slab allocator. At first memcached did just use malloc/free for everything. However this does not play very well with OS memory managers. You get fragmentation, and your OS ends up spending more time trying to find contiguous blocks of memory to feed malloc() than it does running the memcached process. If you disagree, of course you're free to try malloc! just don't complain on the lists ;)
 
The slab allocator was built to work around this. Memory is allocated in chunks internally and constantly reused. Since memory is broken into different size slabs, you do waste memory if your items do not fit perfectly into the slab the server chooses to put it in. This has enjoyed considerable efficiency improvements by Steven Grimm.
 
Some older posts about the slab changes (power of n vs power of 2), and some tradeoffs are on the mailing list:
http://lists.danga.com/pipermail/memcached/2006-May/002163.html
http://lists.danga.com/pipermail/memcached/2007-March/003753.html
 
And if you'd like to attempt to use malloc/free and see how it works, you may define 'USE_SYSTEM_MALLOC' in build process. It might not be tested very well, so getting developer support for it is unlikely."

References:

"The maximum object size is a few bytes under 1MB. You're trying to set about 250k more than it can take."
"You can recompile memcached with a higher POWER_LARGEST if you need more than 1MB. It's in slabs.c at the top. Make it 21 to get 2MB max, or 22 for 4MB max, etc."
"The maximum size is 1 megabyte. It can be increased; I believe there was a patch (against the old 1.1.12 version) to do it. The trick is that memcached's memory manager allocates memory in 1MB chunks, so increasing the maximum is not just a matter of changing a "maximum object size" setting somewhere. But it is certainly doable."
"I propose all objects are stored as a linked list of 64 byte chunks."
"We've talked internally about eventually moving away from the slab-class-based system. It should be possible to run a job in the background (especially in the multithreaded version, but even in the single-threaded one) to shuffle objects around in memory to pack them optimally. If you separate the fixed-size headers from the item data, you can even do away with byte-alignment-based waste. That would give you 100% memory efficiency for any arbitrary mix of item sizes, and true LRU behavior across the whole cache, at the cost of greater CPU consumption."

Startup Scripts

Daemon scripts for memcached:

touch /etc/memcached.conf
cat > /etc/memcached.conf
#Memory a usar
-m 16
# default port
-p 11211
# user to run daemon nobody/apache/www-data
-u nobody
# only listen locally
-l 127.0.0.1
touch /etc/init.d/memcached
chmod +x /etc/init.d/memcached
cat > /etc/init.d/memcached
#!/bin/bash
#
# memcached    This shell script takes care of starting and stopping
#              standalone memcached.
#
# chkconfig: - 80 12
# description: memcached is a high-performance, distributed memory
#              object caching system, generic in nature, but
#              intended for use in speeding up dynamic web
#              applications by alleviating database load.
# processname: memcached
# config: /etc/memcached.conf
# Source function library.
. /etc/rc.d/init.d/functions
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
DAEMON=/usr/local/bin/memcached
DAEMONBOOTSTRAP=/usr/local/bin/start-memcached
DAEMONCONF=/etc/memcached.conf
NAME=memcached
DESC=memcached
PIDFILE=/var/run/$NAME.pid
[ -x $DAEMON ] || exit 0
[ -x $DAEMONBOOTSTRAP ] || exit 0
RETVAL=0
start() {
 echo -n $"Starting $DESC: "
 daemon $DAEMONBOOTSTRAP $DAEMONCONF
 RETVAL=$?
 [ $RETVAL -eq 0 ] && touch $PIDFILE
 echo
 return $RETVAL
}
stop() {
 echo -n $"Shutting down $DESC: "
 killproc $NAME
 RETVAL=$?
 echo
 [ $RETVAL -eq 0 ] && rm -f $PIDFILE
 return $RETVAL
}
# See how we were called.
case "$1" in
 start)
  start
  ;;
 stop)
  stop
  ;;
 restart|reload)
  stop
  start
  RETVAL=$?
  ;;
 status)
  status $prog
  RETVAL=$?
  ;;
 *)
  echo $"Usage: $0 {start|stop|restart|status}"
  exit 1
esac
exit $RETVAL

touch /usr/local/bin/start-memcached
chmod +x  /usr/local/bin/start-memcached
cat > /usr/local/bin/start-memcached
#!/usr/bin/perl -w
# start-memcached
# 2003/2004 - Jay Bonci <jaybonci@debian.org>
# This script handles the parsing of the /etc/memcached.conf file
# and was originally created for the Debian distribution.
# Anyone may use this little script under the same terms as
# memcached itself.
use strict;
if ($> != 0 and $< != 0) {
 print STDERR "Only root wants to run start-memcached.\n";
 exit;
}
my $etcfile = shift || "/etc/memcached.conf";
my $params = [];
my $etchandle; 
# This script assumes that memcached is located at /usr/bin/memcached, and
# that the pidfile is writable at /var/run/memcached.pid
my $memcached = "/usr/local/bin/memcached";
my $pidfile = "/var/run/memcached.pid";
# If we don't get a valid logfile parameter in the /etc/memcached.conf file,
# we'll just throw away all of our in-daemon output. We need to re-tie it so
# that non-bash shells will not hang on logout. Thanks to Michael Renner for 
# the tip
my $fd_reopened = "/dev/null";
sub handle_logfile {
 my ($logfile) = @_;
 $fd_reopened = $logfile;
}
sub reopen_logfile {
 my ($logfile) = @_;
 open *STDERR, ">>$logfile";
 open *STDOUT, ">>$logfile";
 open *STDIN, ">>/dev/null";
 $fd_reopened = $logfile;
}
# This is set up in place here to support other non -[a-z] directives
my $conf_directives = {
 "logfile" => \&handle_logfile
};
if (open $etchandle, $etcfile) {
 foreach my $line (<$etchandle>) {
  $line =~ s/\#.*//go;
  $line = join ' ', split ' ', $line;
  next unless $line;
  next if $line =~ /^\-[dh]/o;
  if ($line =~ /^[^\-]/o) {
   my ($directive, $arg) = $line =~ /^(.*?)\s+(.*)/; 
   $conf_directives->{$directive}->($arg);
   next;
  }
  push @$params, $line;
 }
}
unshift @$params, "-u root" unless (grep $_ eq '-u', @$params);
$params = join " ", @$params;
if (-e $pidfile) {
 open PIDHANDLE, "$pidfile";
 my $localpid = <PIDHANDLE>;
 close PIDHANDLE;
 chomp $localpid;
 if (-d "/proc/$localpid") {
  print STDERR "memcached is already running.\n"; 
  exit;
 } else {
  `rm -f $localpid`;
 }
}
my $pid = fork();
if ($pid == 0) {
 reopen_logfile($fd_reopened);
 exec "$memcached $params";
 exit(0);
} elsif (open PIDHANDLE,">$pidfile") {
 print PIDHANDLE $pid;
 close PIDHANDLE;
} else {
 print STDERR "Can't write pidfile to $pidfile.\n";
}

service memcached restart
chkconfig memcached on

An alternative startup script:

Bash Stat Command

Bash Stat Command:

#!/bin/bash
#
# Make a 'stat' command on a memcache daemon server
# Outputs to a pipe for later using with syslog-ng
#
# alexandre.abreu at gmail.com

PATH=/bin:/usr/bin:/sbin:/usr/sbin
SERVER=localhost
PORT=11111
FIFO=/export/logs/memcache.fifo
BUFF="memcache: "

[ -p $FIFO ] || mkfifo $FIFO || exit 1

exec 666<>/dev/tcp/$SERVER/$PORT || exit 1

append() {
    BUFF="$BUFF $@"
    [ "$1" == "uptime" ] && RUNTIME=`echo "scale=3; $2/60" | bc`

    [ "$1" == "get_hits" ] && {
        rate=`echo "scale=3; $2/$RUNTIME" | bc`
        rate=`printf "%.3f" $rate`
        append hitrate $rate
    }
    [ "$1" == "get_misses" ] && {
        rate=`echo "scale=3; $2/$RUNTIME" | bc`
        rate=`printf "%.3f" $rate`
        append missrate $rate
    }
}

echo "stats" >&666

while read line; do
    line=`echo $line | sed 's|[[:cntrl:]]||g'`
    set - $line
    append $2 $3
    [ "$2" == "limit_maxbytes" ] && break
done <&666

echo "$BUFF" > $FIFO
echo "quit" >&666
exit 0

My Version:

#!/bin/bash
#
# Make a 'stat' command on a memcache daemon server
# Outputs to a pipe for later using with syslog-ng
#

PATH=/bin:/usr/bin:/sbin:/usr/sbin
SERVER=10.20.30.40
PORT=11211
FIFO=/tmp/memcache.fifo

[ -p $FIFO ] || mkfifo $FIFO || exit 1

exec 666<>/dev/tcp/$SERVER/$PORT || exit 1

echo "stats" >&666

while read line; do
    echo $line;
    FLAG=`echo $line | cut -b 1-3`
    [ "$FLAG" == "END" ] && break
done <&666

echo "quit" >&666
exit 0