1# F-Stack Development Guide
2
3With the rapid development of Network Interface Cards the poor performance of data packet processing with the Linux kernel has become the bottleneck in modern network systems. Yet, the increasing demands of the Internet's growth demand a higher performant network processing solution. Kernel bypass has emerged to catch more and more attention. There are various similar technologies such as: DPDK, NETMAP and PF_RING. The main idea of kernel bypass is that Linux is only used to deal with control flow; all data streams are processed in user space. Therefore, kernel bypass can avoid performance bottlenecks caused by kernel packet copying, thread scheduling, system calls, and interrupts. Furthermore, kernel bypass can achieve higher performance with multi-optimizing methods. Within various techniques, DPDK has been widely used because of it's more thorough isolation from kernel scheduling and active community support.
4
5F-Stack is an open source high performant network framework based on DPDK with the following characteristics:
6
71. Ultra high network performance which the network card can achieve under full load: 10 million concurrent connections, 5 million RPS, 1 million CPS.
82. Transplant FreeBSD 11.01 user space stack, which provides a complete stack function, and cut a great amount of irrelevant features. This greatly enhances network performance.
93. Support Nginx, Redis, and other mature applications. Services can easily use F-Stack.
104. Easy to extend with multi-process architecture.
115. Provides micro thread interface. Various applications with stateful applications can easily use F-Stack to get high performance without processing complex asynchronous logic.
126. Provide an Epoll/Kqueue interface that allow many kinds of applications to easily use F-Stack.
13
14## Structure of F-Stack code
15
16    ├── app  -- Nginx(1.11.10)/Redis(3.2.8)/Microthread framework
17    ├── config.ini
18    ├── doc
19    ├── dpdk -- Intel DPDK(16.07) directory
20    ├── example -- DEMO
21    ├── freebsd -- FreeBSD(11.0) Network Stack directory
22    ├── lib -- F-Stack lib directory
23    ├── mk
24    └── start.sh
25
26
27## DPDK initialization
28
29### PORT & SOCKET
30
31F-Stack simplify the initialization of the standard DPDK. By setting the NIC port and CPU core mask, you can set binding relationship of the port and CPU and lcore on different socket node. If there is no binding relationship set, port0 and socket node 0 will be set by default.
32
33### KNI related
34
35If the server does not have dedicated port, or all port used for service process, you need to open the KNI in the configuration file, and set the related protocol and port number to decide which packets need to be processed by the F-Stack, remaining packets will be forwarded to kernel by KNI, to support SSH management functions.
36
37## Revise of FreeBSD Network Stack and DPDK based
38
39Since DPDK is open source, there are various open source network stacks based on DPDK to support the higher level application in the market. Some are will be packaging Linux network stack into a library, some are porting FreeBSD network stack.
40
41At the beginning of this work, F-Stack used a simple TCP/IP stack that developed by ourselves. However, with the growth of various services, this stack couldn't meet the needs of these services while continue to develop and maintain a complete network stack will cost high. So the FreeBSD network stack was ported into F-Stack. The FreeBSD network stack provides complete features and can follow up the improvement from the community. Thanks to [libplebnet](https://gitorious.org/freebsd/kmm-sandbox/commit/fa8a11970bc0ed092692736f175925766bebf6af?p=freebsd:kmm-sandbox.git;a=tree;f=lib/libplebnet;h=ae446dba0b4f8593b69b339ea667e12d5b709cfb;hb=refs/heads/work/svn_trunk_libplebnet) and [libuinet](https://github.com/pkelsey/libuinet), this work becomes a lot easier.
42
43In order to minimize the impact of resource sharing and kernel system (such as scheduling, locks, etc.) on the performance, F-Stack uses a multi-process architecture. Following are the changes to the FreeBSD network stack.
44
45### Scheduling
46
47Cut kernel thread, interrupt thread, timer thread, sched, wakeup, sleep, etc of FreeBSD Network Stack
48
49### Lock
50
51Cut lock operations of FreeBSD Network Stack, including mtx、rw、rm、sx、cond, etc.
52
53### Memory related
54
55Using phymem, uma\_page\_slab\_hash, uma initialization, kmem_malloc malloc
56
57### Global variables
58
59pcpu curthread proc0 thread0, initialization
60
61### Environment variable
62
63setenv getenv
64
65### SYS_INIT
66
67mi_startup
68
69### Clock
70
71timecounter, ticks, hz, timer
72
73### Other
74
75Linux and freebsd errno conversion, glue code, Remove unnecessary modules
76
77## Applications use F-Stack
78
79F-Stack provides ff API (See  *F-Stack\_API\_Reference*) to support applications. F-Stack also integrates third-party application such as Nginx, Redis, etc and. Micro thread interface is also provided to help original application easily use F-Stack.
80
81### Web application
82
83HTTP web application can use F-Stack with Nginx.
84
85### key-value application
86
87key-value db application can use F-Stack with redis, and can start multi Redis instance.
88
89### Stateful(High latency) applications
90
91Applications with stateful(high latency) use F-Stack , state need to be stored for a long time, can directly use the F-Stack micro threading framework. Applications only need to focus on with the service logic. And with synchronous programming, high performance asynchronous service server can be achieved.
92
93## F-Stack configure file reference
94
95  DPDK related parameters, including coremask adn NIC ports num.
96  FreeBSD related parameters, similar with original FreeBSD's /boot.config and /etc/sysctl.conf.
97
98## Start a F-Stack application
99
100Since F-Stack is multi-process architecture, every F-Stack application process should call `ff_init(argc, argv)` to initialize the environments.
101For example, if `lcore_mask=f` in config.ini, you can start your app like this:
102
103    ${bin} --conf config.ini --proc-type=primary --proc-id=0
104    ${bin} --conf config.ini --proc-type=secondary --proc-id=1
105    ${bin} --conf config.ini --proc-type=secondary --proc-id=2
106    ${bin} --conf config.ini --proc-type=secondary --proc-id=3
107
108Or you can just use `start.sh` under F-Stack root directory.
109