1# F-Stack 2 3 4## Introduction 5With the rapid development of NIC, the poor performance of data packets processing with Linux kernel has become the bottleneck. However, the rapid development of the Internet needs high performance of network processing, kernel bypass has caught more and more attention. There are various similar technologies appear, such as DPDK, NETMAP and PF_RING. The main idea of kernel bypass is that Linux is only used to deal with control flow, all data streams are processed in user space. Therefore, kernel bypass can avoid performance bottlenecks caused by kernel packet copy, thread scheduling, system calls and interrupt. Furthermore, kernel bypass can achieve higher performance with multi optimizing methods. Within various techniques, DPDK has been widely used because of its more thorough isolation from kernel scheduling and active community support. 6 7[F-Stack](http://www.f-stack.org/?from=github) is an open source network framework with high performance based on DPDK. With follow characteristics 8 91. Ultra high network performance which can achieve network card under full load, 10 million concurrent connection, 5 million RPS, 1 million CPS. 102. Transplant FreeBSD 11.01 user space stack, provides a complete stack function, cut a great amount of irrelevant features. Therefore greatly enhance the performance. 113. Support Nginx, Redis and other mature applications, service can easily use F-Stack 124. With Multi-process architecture, easy to extend 135. Provide micro thread interface. Various applications with stateful app can easily use F-Stack to get high performance without processing complex asynchronous logic. 146. Provide Epoll/Kqueue interface that allow many kinds of applications easily use F-Stack 15 16## History 17 18In order to deal with the increasingly severe DDoS attacks, authorized DNS server of Tencent Cloud DNSPod switched from Gigabit Ethernet to 10-Gigabit at the end of 2012. We faced several options, one is to continue to use the original model another is to use kernel bypass technology. After several rounds of investigation, we finally chose to develop our next generation of DNS server based on DPDK. The reason is DPDK provides ultra-high performance and can be seamlessly extended to 40G, or even 100G NIC in the future. 19 20After several months of development and testing, DKDNS, high-performance DNS server based on DPDK officially released in October 2013. It's capable of achieving up to 11 million QPS with a single 10GE port and 18.2 million QPS with two 10GE ports. And then we developed a user-space TCP/IP stack called F-Stack that can process 0.6 million RPS with a single 10GE port. 21 22With the fast growth of Tencent Cloud, more and more services need higher network access performance. Meanwhile, F-Stack was continuous improving driven by the business growth, and ultimately developed into a general network access framework. But this TCP/IP stack couldn't meet the needs of these services while continue to develop and maintain a complete network stack will cost high, we've tried several plans and finally determined to port FreeBSD(11.0 stable) TCP/IP stack into F-Stack. Thus, we can reduce the cost of maintenance and follow up the improvement from community quickly.Thanks to [libplebnet](https://gitorious.org/freebsd/kmm-sandbox/commit/fa8a11970bc0ed092692736f175925766bebf6af?p=freebsd:kmm-sandbox.git;a=tree;f=lib/libplebnet;h=ae446dba0b4f8593b69b339ea667e12d5b709cfb;hb=refs/heads/work/svn_trunk_libplebnet) and [libuinet](https://github.com/pkelsey/libuinet), this work becomes a lot easier. 23 24With the rapid development of all kinds of application, in order to help different APPs quick and easily use F-Stack, F-Stack has integrated Nginx, Redis and other commonly used APPs, and a micro thread framework, and provides a standard Epoll/Kqueue interface. 25 26Currently, besides authorized DNS server of DNSPod, there are various products in Tencent Cloud has used the F-Stack, such as HttpDNS (D+), COS access module, CDN access module, etc.. 27 28## Quick Start 29 30 #clone F-Stack 31 mkdir /data/f-stack 32 git clone https://github.com/F-Stack/f-stack.git /data/f-stack 33 34 cd f-stack 35 # compile DPDK 36 cd dpdk/tools 37 ./dpdk-setup.sh # compile with x86_64-native-linuxapp-gcc 38 39 # Set hugepage 40 # single-node system 41 echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages 42 43 # or NUMA 44 echo 1024 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages 45 echo 1024 > /sys/devices/system/node/node1/hugepages/hugepages-2048kB/nr_hugepages 46 47 # Using Hugepage with the DPDK 48 mkdir /mnt/huge 49 mount -t hugetlbfs nodev /mnt/huge 50 51 # close ASLR; it is necessary in multiple porcess 52 echo 0 > /proc/sys/kernel/randomize_va_space 53 54 # offload NIC 55 modprobe uio 56 insmod /data/f-stack/dpdk/x86_64-native-linuxapp-gcc/build/kmod/igb_uio.ko 57 insmod /data/f-stack/dpdk/x86_64-native-linuxapp-gcc/build/kmod/rte_kni.ko 58 python dpdk-devbind.py --status 59 ifconfig eth0 down 60 python dpdk-devbind.py --bind=igb_uio eth0 # assuming that use 10GE NIC and eth0 61 62 # Compile F-Stack 63 cd ../../lib/ 64 make 65 export FF_PATH=/data/f-stack 66 export FF_DPDK=/data/f-stack/dpdk/x86_64-native-linuxapp-gcc 67 68#### Nginx 69 70 cd app/nginx-1.11.10 71 ./configure --prefix=/usr/local/nginx_fstack --with-ff_module 72 make 73 make install 74 cd ../.. 75 ./start.sh -b /usr/local/nginx_fstack/sbin/nginx -c config.ini 76 77#### Redis 78 79 cd app/redis-3.2.8/ 80 make 81 make install 82 83 84## Nginx Testing Result 85 86Test environment 87 88 NIC:Intel Corporation Ethernet Controller XL710 for 40GbE QSFP+ 89 CPU:Intel(R) Xeon(R) CPU E5-2670 v3 @ 2.30GHz 90 Memory：128G 91 OS:CentOS Linux release 7.2 (Final) 92 Kernel：3.10.104-1-tlinux2-0041.tl2 93 94Nginx uses linux kernel's default config, all soft interrupts are working in the first CPU core. 95 96Nginx si means modify the smp_affinity of every IRQ, so that the decision to service an interrupt with a particular CPU is made at the hardware level, with no intervention from the kernel. 97 98 99CPS (Connection:close, Small data packet) test result 100 101 102RPS (Connection:Keep-Alive, Small data packet) test data 103 104 105Bandwidth (Connection:Keep-Alive, 3.7k bytes data packet) test data 106 107 108## Licenses 109See [LICENSE](LICENSE) 110
){kind=link}
){kind=link}
){kind=link}