
or how do I prevent the timemachine from dropping packets? 

$Id: TUNING 231 2008-08-06 12:14:25Z gregor $

This document only covers tuning for FreeBSD 6 and Linux. 

We reccomand that one uses FreeBSD to run the timemachine, because
	- FreeBSD outpferforms Linux when capturing packets 
	  (see http://www.net.t-labs.tu-berlin.de/research/bpcs/)
	- FreeBSD's thread scheduling can be tuned, which is an essential
	  factor when using indexes.


The timemachine is a multithreaded application. Esp. when using indexes you
may experience spikes in CPU usage that might cause the TM to drop packets.
To prevent this, you should: 


MEMORY ALLOCATOR (FreeBSD Users: Read this!).

If you are using FreeBSD, you should really use a different memory allocator,
since FreeBSD's default allocator has really poor performance. 
We are using pmtalloc2 (http://www.malloc.de/en/), which is also shipped with
the timemachine. When building the timemachine on FreeBSD, then ptmalloc
will be used  as memory allocator. (see also ./configure --help)
DO NOT USE ptmalloc3 (yet). We got segaults using it. 

ptmalloc is used in glibc, so if you are using Linux, you are fine!


THREAD SCHEDULING 

NOTE: While doing further testing with thread scheduling we found out that
some FreeBSD releases and hardware platforms do not adhere to the infrmation
below. SO YOU WILL HAVE TO EXPERIMENT WITH THESE SETTINGS TO DETERMINE THE
BEST SOLUTION FOR YOU.

Understanding and tuning thread scheduling can have huge impact on the
TM's performance. The TM's capture thread is the most crucial part. It must
read packets from the capture interface, otherwise the packets are lost. 
Therefore the idea is give the capture thread preference over all the other
threads. Unfortunaly every operating system and libpthread 
implementation has quite different semantics for thread scheduling.
For more info on pthread scheduling in general see: 
http://www.net.t-labs.tu-berlin.de/~gregor/tools/pthread-scheduling.html

The 'tweak_capture_thread' config option is used to tune thread 
scheduling parameters. If you have other (CPU demanding) processes 
running, you can use 'nice' to adjust the nice value. I.e., give the 
TM a lower nice value or give the other process a higher one. 

In general the best thing to do is to set 'tweak_capture_thread' to
'priority'. BUT: read on, since this option might not work per 
default!!

Linux:
On Linux thread scheduling cannot be tuned. Well to be excact: It 
could be done, but with severe side effects. The TM doesn't support
thread tuning on Linux therefore. 

FreeBSD 6: 
You must use the libpthread thread implementation, if you want to 
tweak the capture thread. The libpthread implementation is the default
(check the output of ldd tm). Otherwise, libmap.conf(5) to select the
thread implementation at runtime. 

FreeBSD 7:
You must use the libkse thread implementation, if you want to 
tweak the capture thread. Unfortunaly it isn't the default on FreeBSD 7.
Use libmap.conf(5) to change the implementation. Example:
$ cat /etc/libmap.conf
libthr.so.3   libkse.so.3
libthr.so     libkse.so

DO NOT USE libpthread ON FreeBSD 7!! libpthread is now a symlink to 
libthr!! And with libthr one cannot tune thread scheduling parameters.


Explanation what the tweak_capture_thread does:

On FreeBSD the timemachine process contents for the
CPU(s) with all other processes. Every thread in the timemachine
will get an equal share of CPU time. Since the capture thread
is time critical, the priority of the capture thread
can be increased, so that the capture thread will get a CPU
whenever it is runnable and the timemachine as a whole has
the CPU(s). Note that with the higher priority, the capture thread will 
get a CPU *whenever* it is runnable, i.e. on a single CPU/Core system it may 
starve other timemachine threads. 

If the 'tweak_capture_thread' option in the config file is set to 
'pirority' then the capture thread will be assigned a higher priority. 
On multi core / multi CPU system this option should be enabled.

On signle CPU / core system, or systems with other (large) processes
running, setting this option might stare the index threads (i.e. the
indexes are not maintained any more). Check the tm.log file for ERRORs
and/or WARNINGs (esp. for idx_queue related message).

On such system it might be better to set the 'tweak_capture_thread'
option to 'scope'. When this is done, the capture thread will be
scheduled as if it were a process. I.e the capture thread, the
group off all other timemachine processes and the other processes
will content for the CPU. 

One should try to set 'tweak_capture_thread' option to 'pirority' and
if the results are not as desired, it should be changed to 
'scope' or disabled altogether. Or one can try to use nice, to give
the TM a higher CPU share before setting 'tweak_capture_thread' to
'scope'.

As mentioned before these options are only available on FreeBSD!


NUMBER OF CPUs 
For high speed links you should use a Dual CPU systems. A Dual core CPU
will probably work as well. Esp. using multiple indexes puts quite
some load to the CPUs. If you have only one core and you experience 
packet loss, reducing the number of indexes might help.


MEMORY USAGE 

The timemachine will use much more memory than the memory used for
the memory packet fifos. As a rule of thumb, the timemachine will
use 2 to 5 times as much memory as it needs for the memory fifos.
Of course, this figure wil change with the number of indexes. 
So if you lose packet, check if the timemachine is swapping


HARDDISK I/O

For faster links we stronly suggest to use a (hardware) RAID 0
You might conisder storing the indexes and the packet data on 
different partitions.


CAPTURING THE PACKETS

Another vital part is the capturing interface. The timemachine
uses libpacp to capture the packets. 

For best performance, you should get specializes hardware, like
Endace DAG network monitoring cards.
http://www.endace.com/

If you are using a normal NIC have a look at
http://www.net.t-labs.tu-berlin.de/research/bpcs/ for a packet 
capture tuning howto. 

In short: 

Linux:
	echo 33554432 > /proc/sys/net/core/rmem_default
	echo 33554432 > /proc/sys/net/core/rmem_max
	echo 10000 > /proc/sys/net/core/netdev_max_backlog
and/or use Phil Wood's mmap patch: http://public.lanl.gov/cpw/

FreeBSD >=6:
	sysctl -w net.bpf.bufsize=10485760
	sysctl -w net.bpf.maxbufsize=10485760

FreeBSD <6:
	sysctl -w debug.bpf_bufsize=10485760
	sysctl -w debug.maxbpf_bufsize=10485760

If you still experiance loss, you might increase the buffers to 
20MB: 
	sysctl -w net.bpf.bufsize=20971520
	sysctl -w net.bpf.maxbufsize=20971520
