1[](https://travis-ci.org/F-Stack/f-stack) 2 3# F-Stack 4 5 6## Introduction 7With the rapid development of NIC, the poor performance of data packets processing with Linux kernel has become the bottleneck. However, the rapid development of the Internet needs high performance of network processing, kernel bypass has caught more and more attentions. There are various similar technologies appear, such as DPDK, NETMAP and PF_RING. The main idea of kernel bypass is that Linux is only used to deal with control flow, all data streams are processed in user space. Therefore, kernel bypass can avoid performance bottlenecks caused by kernel packet copying, thread scheduling, system calls and interrupts. Furthermore, kernel bypass can achieve higher performance with multi optimizing methods. Within various techniques, DPDK has been widely used because of its more thorough isolation from kernel scheduling and active community support. 8 9[F-Stack](http://www.f-stack.org/?from=github) is an open source network framework with high performance based on DPDK. With following characteristics 10 111. Ultra high network performance which can achieve network card under full load, 10 million concurrent connections, 5 million RPS, 1 million CPS. 122. Transplant FreeBSD 11.01 user space stack, provides a complete stack function, cut a great amount of irrelevant features. Therefore greatly enhance the performance. 133. Support Nginx, Redis and other mature applications, service can easily use F-Stack 144. With Multi-process architecture, easy to extend 155. Provide micro thread interface. Various applications with stateful app can easily use F-Stack to get high performance without processing complex asynchronous logic. 166. Provide Epoll/Kqueue interface that allow many kinds of applications easily use F-Stack 17 18## History 19 20In order to deal with the increasingly severe DDoS attacks, authorized DNS server of Tencent Cloud DNSPod switched from Gigabit Ethernet to 10-Gigabit at the end of 2012. We faced several options, one is to continue to use the original model another is to use kernel bypass technology. After several rounds of investigation, we finally chose to develop our next generation of DNS server based on DPDK. The reason is DPDK provides ultra-high performance and can be seamlessly extended to 40G, or even 100G NIC in the future. 21 22After several months of development and testing, DKDNS, high-performance DNS server based on DPDK officially released in October 2013. It's capable of achieving up to 11 million QPS with a single 10GE port and 18.2 million QPS with two 10GE ports. And then we developed a user-space TCP/IP stack called F-Stack that can process 0.6 million RPS with a single 10GE port. 23 24With the fast growth of Tencent Cloud, more and more services need higher network access performance. Meanwhile, F-Stack was continuous improving driven by the business growth, and ultimately developed into a general network access framework. But this TCP/IP stack couldn't meet the needs of these services while continue to develop and maintain a complete network stack will cost high, we've tried several plans and finally determined to port FreeBSD(11.0 stable) TCP/IP stack into F-Stack. Thus, we can reduce the cost of maintenance and follow up the improvement from community quickly.Thanks to [libplebnet](https://gitorious.org/freebsd/kmm-sandbox/commit/fa8a11970bc0ed092692736f175925766bebf6af?p=freebsd:kmm-sandbox.git;a=tree;f=lib/libplebnet;h=ae446dba0b4f8593b69b339ea667e12d5b709cfb;hb=refs/heads/work/svn_trunk_libplebnet) and [libuinet](https://github.com/pkelsey/libuinet), this work becomes a lot easier. 25 26With the rapid development of all kinds of application, in order to help different APPs quick and easily use F-Stack, F-Stack has integrated Nginx, Redis and other commonly used APPs, and a micro thread framework, and provides a standard Epoll/Kqueue interface. 27 28Currently, besides authorized DNS server of DNSPod, there are various products in Tencent Cloud has used the F-Stack, such as HttpDNS (D+), COS access module, CDN access module, etc.. 29 30## Quick Start 31 32 #clone F-Stack 33 mkdir /data/f-stack 34 git clone https://github.com/F-Stack/f-stack.git /data/f-stack 35 36 cd f-stack 37 # compile DPDK 38 cd dpdk/tools 39 ./dpdk-setup.sh # compile with x86_64-native-linuxapp-gcc 40 41 # Set hugepage 42 # single-node system 43 echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages 44 45 # or NUMA 46 echo 1024 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages 47 echo 1024 > /sys/devices/system/node/node1/hugepages/hugepages-2048kB/nr_hugepages 48 49 # Using Hugepage with the DPDK 50 mkdir /mnt/huge 51 mount -t hugetlbfs nodev /mnt/huge 52 53 # close ASLR; it is necessary in multiple process 54 echo 0 > /proc/sys/kernel/randomize_va_space 55 56 # offload NIC 57 modprobe uio 58 insmod /data/f-stack/dpdk/x86_64-native-linuxapp-gcc/kmod/igb_uio.ko 59 insmod /data/f-stack/dpdk/x86_64-native-linuxapp-gcc/kmod/rte_kni.ko 60 python dpdk-devbind.py --status 61 ifconfig eth0 down 62 python dpdk-devbind.py --bind=igb_uio eth0 # assuming that use 10GE NIC and eth0 63 64 # On Ubuntu, use gawk instead of the default mawk. 65 #sudo apt-get install gawk # or execute `sudo update-alternatives --config awk` to choose gawk. 66 67 # Compile F-Stack 68 export FF_PATH=/data/f-stack 69 export FF_DPDK=/data/f-stack/dpdk/x86_64-native-linuxapp-gcc 70 cd ../../lib/ 71 make 72 73#### Nginx 74 75 cd app/nginx-1.11.10 76 bash ./configure --prefix=/usr/local/nginx_fstack --with-ff_module 77 make 78 make install 79 cd ../.. 80 /usr/local/nginx_fstack/sbin/nginx 81 82for more details, see [nginx guide](https://github.com/F-Stack/f-stack/blob/master/doc/F-Stack_Nginx_APP_Guide.md). 83 84#### Redis 85 86 cd app/redis-3.2.8/ 87 make 88 make install 89 90 If KNI is enabled in the configuration file, you should create a virtual NIC after F-Stack started, and set the ipaddr, netmask, mac addr, route table, etc. These addrs must be same with F-Stack. 91 92 If you don't have another management port, you should execute a script like this. 93 94 /usr/local/nginx_fstack/sbin/nginx 95 sleep 10 96 ifconfig veth0 <ipaddr> netmask <netmask> broadcast <broadcast> hw ether <mac addr> 97 route add -net 0.0.0.0 gw <gateway> dev veth0 98 # route add -net ... # other route rules 99 100## Nginx Testing Result 101 102Test environment 103 104 NIC:Intel Corporation Ethernet Controller XL710 for 40GbE QSFP+ 105 CPU:Intel(R) Xeon(R) CPU E5-2670 v3 @ 2.30GHz(NUMA) 106 Memory：128G 107 OS:CentOS Linux release 7.2 (Final) 108 Kernel：3.10.104-1-tlinux2-0041.tl2 109 110Nginx uses linux kernel's default config, all soft interrupts are working in the first CPU core. 111 112Nginx si means modify the smp_affinity of every IRQ, so that the decision to service an interrupt with a particular CPU is made at the hardware level, with no intervention from the kernel. 113 114Nginx_FStack's 600 cache bytes' body was returned directly in nginx.conf. 115 116All of these test cases use CPUs' physical cores. 117 118 119CPS (Connection:close, Small data packet) test result 120 121 122RPS (Connection:Keep-Alive, Small data packet) test data 123 124 125Bandwidth (Connection:Keep-Alive, 3.7k bytes data packet) test data 126 127 128## Licenses 129See [LICENSE](LICENSE) 130 131## Join us 132 133Tencent Cloud F-Stack team developed F-Stack which is a general network framework based on DPDK and provides ultra high network performance. We are here looking for more and more talented people with great passion on technology to join us. You would have the chance to work with brightest minds on this planet and help Tencent cloud and F-stack continuously evolve. Send us your resume or refer your friend to us if you are interested in joining us. 134 135Open Positions: Software engineer(C/C++), Web developer, IOS/Android developer, Product Manager, Operating Manager, etc. 136Contact: Please send your resume to [us](mailto:[email protected]) 137
){kind=link}
){kind=link}
){kind=link}
){kind=link}