zl程序教程

您现在的位置是:首页 >  Javascript

当前栏目

十,iperf3源代码分析:iperf3 -P多连接并行测试时的TCP多流的创建管理使用释放流程

2023-04-18 14:23:17 时间


通过 iPerf3 -P参数详细图文分析我们知道,在以下二种情况下,我们有可能会使用到-P参数来优化测试参数以达到最准确的网络测试结果。

  1. 测试:“客户端和服务端之间多连接”的测试场景
  2. TCP接收窗口和发送窗口太小,以至于无法充分利用网络带宽

第一种场景,是我们使用-P参数的标准场景,而第二种场景,我们通常需要通过-W参数来调整TCP接收窗口和发送窗口的大小,来充分利用网络带宽。本文描述的是iperf3如何在客户端和接收端管理与处理“iperf3 -P多连接流并行测试”场景下的多个连接的收发包测试过程 。

从以下文章中,我们已经分析了状态机转换和代码调用的主要过程。

1、总体代码结构和业务流程

总体来说:

1.1、对于服务端:

  1. 创建过程:
    在进入CREATE_STREAM状态后,iperf3服务端在收到客户段发过来的测试TCP连接建立的请求后,会先调用iperf_accept()—>iperf_tcp_accept()来建立测试链接test stream,然后调用iperf_new_stream()—>iperf_init_stream()—>iperf_add_stream()创建并添加当前测试连接test stream到test实例下,注意这个测试链接test stream依赖于-P参数的设置,可以有一条或者多条

  2. 管理与使用过程:
    在进入TEST_RUNNING状态后,iperf3服务端会用select轮询所有测试连接test stream,并通过所有有收到测试数据的连接的est stream的socket上调用iperf_recv()—>iperf_tcp_recv()来收取测试数据,并做测试结果统计。直到收到客户端发过来的TEST_END指令,进入TEST_END状态为止。

  3. 释放过程:
    在进入IPERF_DONE状态后,iperf3服务端会调用iperf_reset_test()—>iperf_free_stream() 遍历所有测试连接test stream,并挨个释放测试连接test stream,直到释放完全部test stream。

1.2、对于客户端:

  1. 创建过程:
    在收到服务端发过来的的CREATE_STREAM指令后,进入CREATE_STREAM状态,iperf3客户端会主动发起TCP连接建立的请求,并调用iperf_new_stream—iperf_init_stream—iperf_add_stream直到所有测试连接test stream建立成功,并挂到到test实例下,注意这个测试链接test stream依赖于-P参数的设置,可以有一条或者多条

  2. 管理与使用过程:
    在收到服务端发过来的的TEST_RUNNING指令后,在进入TEST_RUNNING状态后,iperf3客户端会轮询所有测试连接test stream,并调用iperf_send—iperf_tcp_send来发送每个stream上的测试数据,并做测试结果统计。直到完成本次测试为止。

  3. 释放过程:
    在进入IPERF_DONE状态后,iperf3客户端会调用iperf_free_test—iperf_free_stream 遍历所有测试连接test stream,并挨个释放测试连接test stream,直到释放完全部test stream。

2、数据结构分析:

test steam是test这个结构下的了一个指针,这个指针指向一个单向列表,管理当前test实例下的一个或者多个test stream, 每个test stream 对应一个TCP测试连接。


struct iperf_test
{
    char      role;                             /* 'c' lient or 's' erver */
    enum iperf_mode mode;
    ......										//此处省略很多代码
    SLIST_HEAD(slisthead, iperf_stream) streams;//这里通过SLIST_HEAD这个宏定义了指向test->streams的单列表指针
    SLIST_HEAD(plisthead, protocol) protocols;
    ......										//此处省略很多代码
};

这时通过SLIST_HEAD这个宏展开后struct slisthead{struct iperf_stream *slh_first;} streams;就是定义了一个类型为struct slisthead的变量streams, 这个变量的成员只有一个,就是类型为struct iperf_stream的指针slh_first。
可以通过streams.slh_first来进行访问。

3、重要的几个管理用的函数及期功能分析:

3.1、对于服务端:

  1. 创建过程:
    iperf_run_server()—>iperf_tcp_accept()
    iperf_run_server()—>iperf_new_stream()—>iperf_init_stream()—>iperf_add_stream()
    接受客户端发起的TCP连接,创始并保存每个TCP连接实例。

  2. 管理与使用过程:
    iperf_run_server()—>iperf_recv()—>iperf_tcp_recv(),查询每一条TCP连接,有数据接收的则调用iperf_tcp_recv()接收该连接上的所有数据:

  3. 释放过程:
    run()—>iperf_reset_test()—>iperf_free_stream() ,reset测试对象,释放所有测试连接,等待下一次测试开始

3.2、对于客户端:

  1. 创建过程:
    iperf_create_streams()—>create_socket()—>iperf_new_stream()—>iperf_init_stream()—>iperf_add_stream()
    调用iperf_create_streams后,循环调用create_socket向服务端发起TCPI测试连接,并创建TCP测试连接实例,每个TCP测试连接对应一个实例,并且以单列表的方式保存在test->streams.slh_first中。

  2. 管理与使用过程:
    iperf_run_client()—>iperf_send()—>iperf_tcp_send(),循环发送每一个TCP测试连接里的数据,直到本次测试结束。

  3. 释放过程:
    main()—>iperf_free_test()—>iperf_free_stream(), 完成本次测试后,返回到main函数,开始释放。

4、使用到的管理列表用的宏:

列表管理的宏都放在queue.h这个文件里,主要是对test->streams.slh_first这个单位列表进行,初始化,增加,删除,释放,查找,遍历等操作。以下摘录一部分代码做了一些注释,可以供阅读代码参考使用。

/*
 * Singly-linked List definitions.
 */
 // 定义一个单列表变量指针
#define SLIST_HEAD(name, type)						
struct name {								
	struct type *slh_first;	/* first element */			
}

#define	SLIST_HEAD_INITIALIZER(head)					
	{ NULL }

#define SLIST_ENTRY(type)						
struct {								
	struct type *sle_next;	/* next element */			
}
/*
 * Singly-linked List access methods.
 */
 //查找单列表头结点
#define	SLIST_FIRST(head)	((head)->slh_first)
#define	SLIST_END(head)		NULL
#define	SLIST_EMPTY(head)	(SLIST_FIRST(head) == SLIST_END(head))
#define	SLIST_NEXT(elm, field)	((elm)->field.sle_next)

//遍历单列表
#define	SLIST_FOREACH(var, head, field)					
	for((var) = SLIST_FIRST(head);					
	    (var) != SLIST_END(head);					
	    (var) = SLIST_NEXT(var, field))

#define	SLIST_FOREACH_PREVPTR(var, varp, head, field)			
	for ((varp) = &SLIST_FIRST((head));				
	    ((var) = *(varp)) != SLIST_END(head);			
	    (varp) = &SLIST_NEXT((var), field))

/*
 * Singly-linked List functions.
 */
 //初始化一个列表
#define	SLIST_INIT(head) {						
	SLIST_FIRST(head) = SLIST_END(head);				
}

5、运行过程分析

5.1、对于服务端:

  1. 创建过程:
    iperf_run_server()—>iperf_tcp_accept()
    iperf_run_server()—>iperf_new_stream()—>iperf_init_stream()—>iperf_add_stream()
    接受客户端发起的TCP连接,创始并保存每个TCP连接实例。调用Log如下所示:
iperf3 -s
......
debug out: func = iperf_run_server         ,line =  625, file = iperf_server_api.c
debug out: func = iperf_tcp_accept         ,line =  123, file = iperf_tcp.c
debug out: func = iperf_run_server         ,line =  727, file = iperf_server_api.c
debug out: func = iperf_new_stream         ,line = 4247, file = iperf_api.c
debug out: func = iperf_init_stream        ,line = 4398, file = iperf_api.c
debug out: func = iperf_add_stream         ,line = 4461, file = iperf_api.c
debug out: add stream 1
[  5] local 127.0.0.1 port 5201 connected to 127.0.0.1 port 56288
---------------------------------------------------------------------------------------
debug out: func = iperf_run_server         ,line =  625, file = iperf_server_api.c
debug out: func = iperf_tcp_accept         ,line =  123, file = iperf_tcp.c
debug out: func = iperf_run_server         ,line =  727, file = iperf_server_api.c
debug out: func = iperf_new_stream         ,line = 4247, file = iperf_api.c
debug out: func = iperf_init_stream        ,line = 4398, file = iperf_api.c
debug out: func = iperf_add_stream         ,line = 4461, file = iperf_api.c
debug out: search stream 1, i = 3
debug out: add stream 3, i = 3
[  8] local 127.0.0.1 port 5201 connected to 127.0.0.1 port 56292
---------------------------------------------------------------------------------------
debug out: func = iperf_run_server         ,line =  625, file = iperf_server_api.c
debug out: func = iperf_tcp_accept         ,line =  123, file = iperf_tcp.c
debug out: func = iperf_run_server         ,line =  727, file = iperf_server_api.c
debug out: func = iperf_new_stream         ,line = 4247, file = iperf_api.c
debug out: func = iperf_init_stream        ,line = 4398, file = iperf_api.c
debug out: func = iperf_add_stream         ,line = 4461, file = iperf_api.c
debug out: search stream 1, i = 3
debug out: search stream 3, i = 4
debug out: add stream 4, i = 4
[ 10] local 127.0.0.1 port 5201 connected to 127.0.0.1 port 56304
---------------------------------------------------------------------------------------
debug out: func = iperf_run_server         ,line =  625, file = iperf_server_api.c
debug out: func = iperf_tcp_accept         ,line =  123, file = iperf_tcp.c
debug out: func = iperf_run_server         ,line =  727, file = iperf_server_api.c
debug out: func = iperf_new_stream         ,line = 4247, file = iperf_api.c
debug out: func = iperf_init_stream        ,line = 4398, file = iperf_api.c
debug out: func = iperf_add_stream         ,line = 4461, file = iperf_api.c
debug out: search stream 1, i = 3
debug out: search stream 3, i = 4
debug out: search stream 4, i = 5
debug out: add stream 5, i = 5
[ 12] local 127.0.0.1 port 5201 connected to 127.0.0.1 port 56314
---------------------------------------------------------------------------------------


  1. 管理与使用过程:
    iperf_run_server()—>iperf_recv()—>iperf_tcp_recv()

调用Log如下所示,查询每一条TCP连接,有数据接收的则调用iperf_tcp_recv()接收该连接上的所有数据:

iperf3 -s
......
debug out: func = iperf_run_server         ,line =  848, file = iperf_server_api.c
debug out: func = iperf_recv               ,line = 1951, file = iperf_api.c
debug out: current sp->id =1 
debug out: current sp->id =3 
debug out: current sp->id =4 
debug out: current sp->id =5 
---------------------------------------------------------------------------------------
debug out: func = iperf_run_server         ,line =  848, file = iperf_server_api.c
debug out: func = iperf_recv               ,line = 1951, file = iperf_api.c
debug out: current sp->id =1 
debug out: receiving at sp->id =1 
debug out: func = iperf_tcp_recv           ,line =   59, file = iperf_tcp.c
debug out: current sp->id =3 
debug out: current sp->id =4 
debug out: current sp->id =5 
---------------------------------------------------------------------------------------
debug out: func = iperf_run_server         ,line =  848, file = iperf_server_api.c
debug out: func = iperf_recv               ,line = 1951, file = iperf_api.c
debug out: current sp->id =1 
debug out: current sp->id =3 
debug out: receiving at sp->id =3 
debug out: func = iperf_tcp_recv           ,line =   59, file = iperf_tcp.c
debug out: current sp->id =4 
debug out: receiving at sp->id =4 
debug out: func = iperf_tcp_recv           ,line =   59, file = iperf_tcp.c
debug out: current sp->id =5 
---------------------------------------------------------------------------------------
debug out: func = iperf_run_server         ,line =  848, file = iperf_server_api.c
debug out: func = iperf_recv               ,line = 1951, file = iperf_api.c
debug out: current sp->id =1 
debug out: receiving at sp->id =1 
debug out: func = iperf_tcp_recv           ,line =   59, file = iperf_tcp.c
debug out: current sp->id =3 
debug out: receiving at sp->id =3 
debug out: func = iperf_tcp_recv           ,line =   59, file = iperf_tcp.c
debug out: current sp->id =4 
debug out: current sp->id =5 
debug out: receiving at sp->id =5 
debug out: func = iperf_tcp_recv           ,line =   59, file = iperf_tcp.c
---------------------------------------------------------------------------------------
debug out: func = iperf_run_server         ,line =  848, file = iperf_server_api.c
debug out: func = iperf_recv               ,line = 1951, file = iperf_api.c
debug out: current sp->id =1 
debug out: current sp->id =3 
debug out: current sp->id =4 
debug out: receiving at sp->id =4 
debug out: func = iperf_tcp_recv           ,line =   59, file = iperf_tcp.c
debug out: current sp->id =5 
debug out: receiving at sp->id =5 
debug out: func = iperf_tcp_recv           ,line =   59, file = iperf_tcp.c
---------------------------------------------------------------------------------------
  1. 释放过程:
    run()—>iperf_reset_test()—>iperf_free_stream()
debug out: func = run                      ,line =  179, file = main.c
debug out: func = iperf_reset_test         ,line = 3072, file = iperf_api.c
debug out: func = iperf_free_stream        ,line = 4222, file = iperf_api.c
debug out: free stream 1
debug out: func = iperf_free_stream        ,line = 4222, file = iperf_api.c
debug out: free stream 3
debug out: func = iperf_free_stream        ,line = 4222, file = iperf_api.c
debug out: free stream 4
debug out: func = iperf_free_stream        ,line = 4222, file = iperf_api.c
debug out: free stream 5

5.2、对于客户端:

  1. 创建过程:
    iperf_create_streams()—>create_socket()—>iperf_new_stream()—>iperf_init_stream()—>iperf_add_stream()
    调用iperf_create_streams后,循环调用create_socket向服务端发起TCPI测试连接,并创建TCP测试连接实例,每个TCP测试连接对应一个实例,并且以单列表的方式保存在test->streams.slh_first中。
debug out: func = iperf_create_streams     ,line =   69, file = iperf_client_api.c
debug out: func = create_socket            ,line =  129, file = net.c
debug out: func = iperf_new_stream         ,line = 4247, file = iperf_api.c
debug out: func = iperf_init_stream        ,line = 4398, file = iperf_api.c
debug out: func = iperf_add_stream         ,line = 4461, file = iperf_api.c
debug out: add stream 1
[  5] local 127.0.0.1 port 56288 connected to 127.0.0.1 port 5201
debug out: func = create_socket            ,line =  129, file = net.c
debug out: func = iperf_new_stream         ,line = 4247, file = iperf_api.c
debug out: func = iperf_init_stream        ,line = 4398, file = iperf_api.c
debug out: func = iperf_add_stream         ,line = 4461, file = iperf_api.c
debug out: search stream 1, i = 3
debug out: add stream 3, i = 3
[  7] local 127.0.0.1 port 56292 connected to 127.0.0.1 port 5201
debug out: func = create_socket            ,line =  129, file = net.c
debug out: func = iperf_new_stream         ,line = 4247, file = iperf_api.c
debug out: func = iperf_init_stream        ,line = 4398, file = iperf_api.c
debug out: func = iperf_add_stream         ,line = 4461, file = iperf_api.c
debug out: search stream 1, i = 3
debug out: search stream 3, i = 4
debug out: add stream 4, i = 4
[  9] local 127.0.0.1 port 56304 connected to 127.0.0.1 port 5201
debug out: func = create_socket            ,line =  129, file = net.c
debug out: func = iperf_new_stream         ,line = 4247, file = iperf_api.c
debug out: func = iperf_init_stream        ,line = 4398, file = iperf_api.c
debug out: func = iperf_add_stream         ,line = 4461, file = iperf_api.c
debug out: search stream 1, i = 3
debug out: search stream 3, i = 4
debug out: search stream 4, i = 5
debug out: add stream 5, i = 5
[ 11] local 127.0.0.1 port 56314 connected to 127.0.0.1 port 5201

  1. 管理与使用过程:
    iperf_run_client()—>iperf_send()—>iperf_tcp_send()
    循环发送每一个TCP测试连接里的数据,直到本次测试结束。
debug out: func = iperf_run_client         ,line =  633, file = iperf_client_api.c
debug out: func = iperf_send               ,line = 1890, file = iperf_api.c
debug out: func = iperf_send               ,line = 1907, file = iperf_api.c
debug out: func = iperf_tcp_send           ,line =   87, file = iperf_tcp.c
debug out: sending at sp->id = 1
debug out: func = iperf_send               ,line = 1907, file = iperf_api.c
debug out: func = iperf_tcp_send           ,line =   87, file = iperf_tcp.c
debug out: sending at sp->id = 3
debug out: func = iperf_send               ,line = 1907, file = iperf_api.c
debug out: func = iperf_tcp_send           ,line =   87, file = iperf_tcp.c
debug out: sending at sp->id = 4
debug out: func = iperf_send               ,line = 1907, file = iperf_api.c
debug out: func = iperf_tcp_send           ,line =   87, file = iperf_tcp.c
debug out: sending at sp->id = 5
debug out: func = iperf_send               ,line = 1907, file = iperf_api.c
debug out: func = iperf_tcp_send           ,line =   87, file = iperf_tcp.c
debug out: sending at sp->id = 1
debug out: func = iperf_send               ,line = 1907, file = iperf_api.c
debug out: func = iperf_tcp_send           ,line =   87, file = iperf_tcp.c
debug out: sending at sp->id = 3
debug out: func = iperf_send               ,line = 1907, file = iperf_api.c
debug out: func = iperf_tcp_send           ,line =   87, file = iperf_tcp.c
debug out: sending at sp->id = 4
debug out: func = iperf_send               ,line = 1907, file = iperf_api.c
debug out: func = iperf_tcp_send           ,line =   87, file = iperf_tcp.c
debug out: sending at sp->id = 5

  1. 释放过程:
    main()—>iperf_free_test()—>iperf_free_stream(), 完成本次测试后,返回到main函数,开始释放。
iperf Done.
debug out: func = main                     ,line = 121,  file = main.c
debug out: func = iperf_free_test          ,line = 2940, file = iperf_api.c
debug out: func = iperf_free_stream        ,line = 4222, file = iperf_api.c
debug out: free stream 1
debug out: func = iperf_free_stream        ,line = 4222, file = iperf_api.c
debug out: free stream 3
debug out: func = iperf_free_stream        ,line = 4222, file = iperf_api.c
debug out: free stream 4
debug out: func = iperf_free_stream        ,line = 4222, file = iperf_api.c
debug out: free stream 5