1# F-Stack Development Guide
2
3With the rapid development of NIC, the poor performance of data packets processing with Linux kernel has become the bottleneck. However the rapid development of the Internet needs high performance of network processing, kernel bypass has caught more and more attention. There are various similar technologies appear, such as DPDK, NETMAP and PF_RING. The main idea of kernel bypass is that Linux is only used to deal with control flow, all data streams are processed in user space. Therefore kernel bypass can avoid performance bottlenecks caused by kernel packet copy, thread scheduling, system calls and interrupt. Further more, kernel bypass can achieve higher performance with multi optimizing methods.  Within various techniques, DPDK has been widely used because of its more thorough isolation from kernel scheduling and active community support.
4
5F-Stack is an open source network framework with high performance based on DPDK. With follow characteristics
6
71. Ultra high network performance which can achieve network card under full load, 10 million concurrent, five million RPS, 1 million CPS.
82. Transplant FreeBSD 11.01 user space stack, provides a complete stack function, cut a great amount of irrelevant features. Therefore greatly enhance the performance.
93. Support Nginx, Redis and other mature applications, service can easily use F-Stack
104. With Multi-process architecture, easy to extend
115. Provide micro thread interface. Various applications with long time consuming can easily use F-Stack to get high performance without processing complex asynchronous logic.
126. Provide Epoll/kqueue interface that allow many kinds of applications easily use F-Stack
13
14## Structure of F-Stack code
15
16    ├── app  -- Nginx(1.11.10)/Redis(3.2.8)/Microthread framework
17    ├── config.ini
18    ├── doc
19    ├── dpdk -- Intel DPDK(16.07) directory
20    ├── example -- DEMO
21    ├── freebsd -- FreeBSD(11.0) Network Stack directory
22    ├── lib -- F-Stack lib directory
23    ├── mk
24    └── start.sh
25
26
27## DPDK initialization
28
29### PORT & SOCKET
30
31F-Stack simplify the initialization of the standard DPDK. By setting the NIC port and CPU core mask, you can set binding relationship of the port and CPU and lcore on different socket node. If there is no binding relationship set, port0 and socket node 0 will be set by default.
32
33### KNI related
34
35If the server does not have dedicated port, or all port used for service process, you need to open the KNI in the configuration file, and set the related protocol and port number to decide which packets need to be processed by the F-Stack, remaining packets will be forwarded to kernel by KNI, to support SSH management functions.
36
37## Revise of FreeBSD Network Stack and DPDK based
38
39Since DPDK is open source, there are various open source network stacks based on DPDK to support the higher level application in the market. Some are will be packaging Linux network stack into a library, some are porting FreeBSD network stack.
40
41At the beginning of this work, F-Stack used a simple TCP/IP stack that developed by ourselves. However, with the growth of various services, this stack couldn't meet the needs of these services while continue to develop and maintain a complete network stack will cost high. So the FreeBSD network stack was ported into F-Stack. The FreeBSD network stack provides complete features and can follow up the improvement from the community. Thanks to [libplebnet](https://gitorious.org/freebsd/kmm-sandbox/commit/fa8a11970bc0ed092692736f175925766bebf6af?p=freebsd:kmm-sandbox.git;a=tree;f=lib/libplebnet;h=ae446dba0b4f8593b69b339ea667e12d5b709cfb;hb=refs/heads/work/svn_trunk_libplebnet) and [libuinet](https://github.com/pkelsey/libuinet), this work becomes a lot easier.
42
43In order to minimize the impact of resource sharing and kernel system (such as scheduling, locks, etc.) on the performance, F-Stack uses a multi-process architecture. Following are the changes to the FreeBSD network stack.
44
45### Scheduling
46
47Cut kernel thread, interrupt thread, timer thread, sched, wakeup, sleep, etc of FreeBSD Network Stack
48
49### Lock
50
51Cut lock operations of FreeBSD Network Stack, including mtx、rw、rm、sx、cond, etc.
52
53### Memory related
54
55Using phymem, uma\_page\_slab\_hash, uma initialization, kmem_malloc malloc
56
57### Global variables
58
59pcpu curthread proc0 thread0, initialization
60
61### Environment variable
62
63setenv getenv
64
65### SYS_INIT
66
67mi_startup
68
69### Clock
70
71timecounter, ticks, hz, timer
72
73### Other
74
75Linux and freebsd errno conversion, glue code, Remove unnecessary modules
76
77## Applications use F-Stack
78
79F-Stack provides ff API (See  *F-Stack\_API\_Reference*) to support applications. F-Stack also integrates third-party application such as Nginx, Redis, etc and. Micro thread interface is also provided to help original application easily use F-Stack.
80
81### Web application
82
83HTTP web application can use F-Stack with Nginx.
84
85### key-value application
86
87key-value db application can use F-Stack with redis, and can start multi Redis instance.
88
89### Stateful(High latency) applications
90
91Applications with stateful(high latency) use F-Stack , state need to be stored for a long time, can directly use the F-Stack micro threading framework. Applications only need to focus on with the service logic. And with synchronous programming, high performance asynchronous service server can be achieved.
92
93## F-Stack configure file reference
94
95  DPDK related parameters, including coremask adn NIC ports num
96
97	[dpdk]
98    lcore_mask=3
99    ## Port mask, enable and disable ports.
100    ## Default: all ports are enabled.
101    #port_mask=1
102    channel=4
103    nb_ports=1
104    promiscuous=1
105    numa_on=1
106
107    [port0]
108    addr=192.168.1.2
109    netmask=255.255.255.0
110    broadcast=192.168.1.255
111    gateway=192.168.1.1
112
113    ## Packet capture path, this will hurt performance
114    #pcap=./a.pcap
115
116    ## Kni config: if enabled and method=reject,
117    ## all packets that do not belong to the following tcp_port and udp_port
118    ## will transmit to kernel; if method=accept, all packets that belong to
119    ## the following tcp_port and udp_port will transmit to kernel.
120    #[kni]
121    #enable=1
122    #method=reject
123    #tcp_port=80
124    #udp_port=53
125
126    # log is invalid
127    [log]
128    level=1
129    dir=/var/log
130
131    ## FreeBSD network performance tuning configurations.
132    ## Most native FreeBSD configurations are supported.
133    [freebsd.boot]
134    hz=100
135
136    kern.ipc.maxsockets=262144
137
138    net.inet.tcp.syncache.hashsize=4096
139    net.inet.tcp.syncache.bucketlimit=100
140
141    net.inet.tcp.tcbhashsize=65536
142
143    [freebsd.sysctl]
144    kern.ipc.somaxconn=32768
145    kern.ipc.maxsockbuf=16777216
146
147    net.inet.tcp.fast_finwait2_recycle=1
148    net.inet.tcp.sendspace=16384
149    net.inet.tcp.recvspace=8192
150    net.inet.tcp.nolocaltimewait=1
151    net.inet.tcp.cc.algorithm=htcp
152    net.inet.tcp.sendbuf_max=16777216
153    net.inet.tcp.recvbuf_max=16777216
154    net.inet.tcp.sendbuf_auto=1
155    net.inet.tcp.recvbuf_auto=1
156    net.inet.tcp.sendbuf_inc=16384
157    net.inet.tcp.recvbuf_inc=524288
158    net.inet.tcp.inflight.enable=0
159    net.inet.tcp.sack=1
160    net.inet.tcp.blackhole=1
161    net.inet.tcp.msl=2000
162    net.inet.tcp.delayed_ack=0
163
164    net.inet.udp.blackhole=1
165    net.inet.ip.redirect=0
166
167## F-Stack Application Start
168
169F-Stack use a multi process architecture to remove resource sharing. There are some attentions for start of application dock with F-Stack. We take the example of start.sh under F-Stack root directory.
170
171    #!/bin/bash
172
173    function usage() {
174        echo "F-Stack app start tool"
175        echo "Options:"
176        echo " -c [conf]                Path of config file"
177        echo " -b [N]                   Path of binary"
178        echo " -h                       show this help"
179        exit
180    }
181
182    conf=config.ini
183    bin=./helloword
184
185    while getopts "c:b:h" args
186    do
187        case $args in
188             c)
189                conf=$OPTARG
190                ;;
191             b)
192                bin=$OPTARG
193                ;;
194             h)
195                usage
196                exit 0
197                ;;
198        esac
199    done
200
201    allcmask0x=`cat ${conf}|grep lcore_mask|awk -F '=' '{print $2}'`
202    ((allcmask=16#$allcmask0x))
203
204    # match coremask actual number of CPU core, and calculate the specified startup parameters of all processes, including
205    #	-c coremask，The coremask parameters and the actual number of CPU core match, and calculate the specific startup parameters of all processes, including
206    #	--proc-type=primary/secondary
207    #	--num-procs = number of process
208    #	--proc-id = current process ID, increase from 0
209    num_procs=0
210    PROCESSOR=$(grep 'processor' /proc/cpuinfo |sort |uniq |wc -l)
211    for((i=0;i<${PROCESSOR};++i))
212    do
213        mask=`echo "2^$i"|bc`
214        ((result=${allcmask} & ${mask}))
215        if [ ${result} != 0 ]
216        then
217            ((num_procs++));
218            cpuinfo[$i]=1
219        else
220            cpuinfo[$i]=0
221        fi
222    done
223    proc_id=0
224    for((i=0;i<${PROCESSOR};++i))
225    do
226        if ((cpuinfo[$i] == 1))
227        then
228            cmask=`echo "2^$i"|bc`
229            cmask=`echo "obase=16;${cmask}"|bc`
230            if ((proc_id == 0))
231            then
232                #echo "${bin} config.ini -c $cmask  --proc-type=primary --num-procs=${num_procs} --proc-id=${proc_id}"
233                ${bin} config.ini -c ${cmask}  --proc-type=primary --num-procs=${num_procs} --proc-id=${proc_id} &
234                sleep 5
235            else
236                #echo "${bin} config.ini -c $cmask --proc-type=secondary --num-procs=${num_procs} --proc-id=${proc_id}"
237                ${bin} config.ini -c $cmask --proc-type=secondary --num-procs=${num_procs} --proc-id=${proc_id} &
238            fi
239            ((proc_id++))
240        fi
241    done
242