xref: /f-stack/README.md (revision 6b8a3e40) 
1[![Build Status](https://travis-ci.org/F-Stack/f-stack.svg?branch=master)](https://travis-ci.org/F-Stack/f-stack)
2
3# F-Stack
4![](F-Stack.png)
5
6## Introduction
7
8With the rapid development of Network Interface Cards the poor performance of data packet processing with the Linux kernel has become the bottleneck in modern network systems. Yet, the increasing demands of the Internet's growth demand a higher performant network processing solution. Kernel bypass has emerged to catch more and more attention. There are various similar technologies such as: DPDK, NETMAP and PF_RING. The main idea of kernel bypass is that Linux is only used to deal with control flow; all data streams are processed in user space. Therefore, kernel bypass can avoid performance bottlenecks caused by kernel packet copying, thread scheduling, system calls, and interrupts. Furthermore, kernel bypass can achieve higher performance with multi-optimizing methods. Within various techniques, DPDK has been widely used because of it's more thorough isolation from kernel scheduling and active community support.
9
10[F-Stack](http://www.f-stack.org/?from=github) is an open source high performant network framework based on DPDK with the following characteristics:
11
121. Ultra high network performance which the network card can achieve under full load: 10 million concurrent connections, 5 million RPS, 1 million CPS.
132. Transplant FreeBSD 11.01 user space stack, which provides a complete stack function, and cut a great amount of irrelevant features. This greatly enhances network performance.
143. Support Nginx, Redis, and other mature applications. Services can easily use F-Stack.
154. Easy to extend with multi-process architecture.
165. Provides micro thread interface. Various applications with stateful applications can easily use F-Stack to get high performance without processing complex asynchronous logic.
176. Provide an Epoll/Kqueue interface that allow many kinds of applications to easily use F-Stack.
18
19## History
20
21To deal with the increasingly severe DDoS attacks the authorized DNS server of Tencent Cloud DNSPod switched from Gigabit Ethernet to 10-Gigabit at the end of 2012. We faced several options: one is to continue to use the original network stack in the Linux kernel, another is to use kernel bypass techniques. After several rounds of investigation; we finally chose to develop our next generation of DNS server based on DPDK. The reason is DPDK provides ultra-high performance and can be seamlessly extended to 40G, or even 100G NIC, in the future.
22
23After several months of development and testing, DKDNS, high-performance DNS server based on DPDK officially released in October 2013. It's capable of achieving up to 11 million QPS with a single 10GE port and 18.2 million QPS with two 10GE ports. And then we developed a user-space TCP/IP stack called F-Stack that can process 0.6 million RPS with a single 10GE port.
24
25With the fast growth of Tencent Cloud more and more of our services needed higher network access performance. Meanwhile, F-Stack was continuing to improve, being driven by our business growth, and, ultimately developed into a general network access framework. But our initial TCP/IP stack couldn't meet the needs of these services. Continuing to develop and maintain a complete high performance network stack would have been too expensive. After evaluating several plans; we finally determined to port FreeBSD's (11.0 stable) TCP/IP stack into F-Stack. Not only does this allow us to stop reinventing the wheel, we can take advantage of the the improvements the FreeBSD community will bring in the future. Thanks to [libplebnet](https://gitorious.org/freebsd/kmm-sandbox/commit/fa8a11970bc0ed092692736f175925766bebf6af?p=freebsd:kmm-sandbox.git;a=tree;f=lib/libplebnet;h=ae446dba0b4f8593b69b339ea667e12d5b709cfb;hb=refs/heads/work/svn_trunk_libplebnet) and [libuinet](https://github.com/pkelsey/libuinet) this work became a lot easier.
26
27With the rapid development of all kinds of applications, in order to help different APPs quick and easily use F-Stack, F-Stack has integrated Nginx, Redis and other commonly used APPs, and a micro thread framework, and provides a standard Epoll/Kqueue interface.
28
29Currently, besides authorized DNS server of DNSPod, there are various products in Tencent Cloud has used the F-Stack, such as HttpDNS (D+), COS access module, CDN access module, etc..
30
31## Quick Start
32
33    # clone F-Stack
34    mkdir -p /data/f-stack
35    git clone https://github.com/F-Stack/f-stack.git /data/f-stack
36
37    # Install libnuma-dev
38    yum install numactl-devel          # on Centos
39    #sudo apt-get install libnuma-dev  # on Ubuntu
40
41    cd f-stack
42    # Compile DPDK
43    cd dpdk/
44    meson -Denable_kmods=true build
45    ninja -C build
46    ninja -C build install
47
48    # Set hugepage
49    # single-node system
50    echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
51
52    # or NUMA
53    echo 1024 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages
54    echo 1024 > /sys/devices/system/node/node1/hugepages/hugepages-2048kB/nr_hugepages
55
56    # Using Hugepage with the DPDK
57    mkdir /mnt/huge
58    mount -t hugetlbfs nodev /mnt/huge
59
60    # Close ASLR; it is necessary in multiple process
61    echo 0 > /proc/sys/kernel/randomize_va_space
62
63    # Install python for running DPDK python scripts
64    sudo apt install python # On ubuntu
65
66    # Offload NIC
67    modprobe uio
68    insmod /data/f-stack/dpdk/build/kernel/linux/igb_uio/igb_uio.ko
69    insmod /data/f-stack/dpdk/build/kernel/linux/kni/rte_kni.ko carrier=on # carrier=on is necessary, otherwise need to be up `veth0` via `echo 1 > /sys/class/net/veth0/carrier`
70    python dpdk-devbind.py --status
71    ifconfig eth0 down
72    python dpdk-devbind.py --bind=igb_uio eth0 # assuming that use 10GE NIC and eth0
73
74    # On Ubuntu, use gawk instead of the default mawk.
75    #sudo apt-get install gawk  # or execute `sudo update-alternatives --config awk` to choose gawk.
76
77    # Install dependencies for F-Stack
78    sudo apt install gcc make libssl-dev # On ubuntu
79
80    # Upgrade pkg-config while version < 0.28
81    #cd /data
82    #wget https://pkg-config.freedesktop.org/releases/pkg-config-0.29.2.tar.gz
83    #tar xzvf pkg-config-0.29.2.tar.gz
84    #cd pkg-config-0.29.2
85    #./configure --with-internal-glib
86    #make
87    #make install
88    #mv /usr/bin/pkg-config /usr/bin/pkg-config.bak
89    #ln -s /usr/local/bin/pkg-config /usr/bin/pkg-config
90
91    # Compile F-Stack
92    export FF_PATH=/data/f-stack
93    export PKG_CONFIG_PATH=/usr/lib64/pkgconfig:/usr/local/lib64/pkgconfig:/usr/lib/pkgconfig
94    cd /data/f-stack/lib/
95    make
96
97    # Install F-STACK
98    # libfstack.a will be installed to /usr/local/lib
99    # ff_*.h will be installed to /usr/local/include
100    # start.sh will be installed to /usr/local/bin/ff_start
101    # config.ini will be installed to /etc/f-stack.conf
102    make install
103
104#### Nginx
105
106    cd app/nginx-1.16.1
107    bash ./configure --prefix=/usr/local/nginx_fstack --with-ff_module
108    make
109    make install
110    cd ../..
111    /usr/local/nginx_fstack/sbin/nginx
112
113for more details, see [nginx guide](https://github.com/F-Stack/f-stack/blob/master/doc/F-Stack_Nginx_APP_Guide.md).
114
115#### Redis
116
117    cd app/redis-5.0.5/
118    make
119    make install
120
121  If KNI is enabled in the configuration file, you should create a virtual NIC after F-Stack started, and set the ipaddr, netmask, mac addr, route table, etc. These addrs must be same with F-Stack.
122
123  If you don't have another management port, you should execute a script like this.
124
125    /usr/local/nginx_fstack/sbin/nginx
126    sleep 10
127    ifconfig veth0 <ipaddr>  netmask <netmask>  broadcast <broadcast> hw ether <mac addr>
128    route add -net 0.0.0.0 gw <gateway> dev veth0
129    echo 1 > /sys/class/net/veth0/carrier # if `carrier=on` not set while `insmod rte_kni.ko`
130    # route add -net ...  # other route rules
131
132## Nginx Testing Result
133
134Test environment
135
136    NIC:Intel Corporation Ethernet Controller XL710 for 40GbE QSFP+
137    CPU:Intel(R) Xeon(R) CPU E5-2670 v3 @ 2.30GHz(NUMA)
138    Memory：128G
139    OS:CentOS Linux release 7.2 (Final)
140    Kernel：3.10.104-1-tlinux2-0041.tl2
141
142Nginx uses linux kernel's default config, all soft interrupts are working in the first CPU core.
143
144Nginx si means modify the smp_affinity of every IRQ, so that the decision to service an interrupt with a particular CPU is made at the hardware level, with no intervention from the kernel.
145
146Nginx Reuseport means enable "reuseport" in `nginx.conf`.
147
148Nginx_FStack's 600 cache bytes' body was returned directly in nginx.conf.
149
150All of these test cases use CPUs' physical cores.
151
152
153CPS (Connection:close, Small data packet)  test result
154![](CPS.png)
155
156CPS_Reuseport (Connection:close, Small data packet)  test result, This test case runs in a different test environment
157![](CPS_Reuseport.png)
158
159RPS (Connection:Keep-Alive, Small data packet) test data
160![](RPS.png)
161
162Bandwidth (Connection:Keep-Alive, 3.7k bytes data packet) test data
163![](Bandwidth.png)
164
165## Licenses
166See [LICENSE](LICENSE)
167
168## Join us
169
170Tencent Cloud F-Stack team developed F-Stack which is a general network framework based on DPDK and provides ultra high network performance. We are here looking for more and more talented people with great passion on technology to join us. You would have the chance to work with brightest minds on this planet and help Tencent cloud and F-stack continuously evolve. Send us your resume or refer your friend to us if you are interested in joining us.
171
172Open Positions: Software engineer(C/C++), Web developer, IOS/Android developer, Product Manager, Operating Manager, etc.
173Contact: Please send your resume to [us](mailto:[email protected])
174