-----============= acceptance-small: sanity-lnet ============----- Wed Sep 9 11:56:54 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky Stopping clients: oleg324-client.virtnet /mnt/lustre (opts:) Stopping client oleg324-client.virtnet /mnt/lustre opts: Stopping clients: oleg324-client.virtnet /mnt/lustre2 (opts:) Stopping /mnt/lustre-mds1 (opts:-f) on oleg324-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg324-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg324-server unloading modules via unload_modules_local on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing unload_modules_local oleg324-server: modules unloaded. Writer error: failed to resolve Netlink family id Stopping clients: oleg324-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg324-client.virtnet /mnt/lustre2 (opts:-f) LNET unconfigure error 22: Invalid argument unloading modules via unload_modules_local on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing unload_modules_local oleg324-server: oleg324-server: LNET unconfigure error 22: Invalid argument modules unloaded. /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.203.24@tcp status: up interfaces: 0: ens2 1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: ens2: mtu 9000 qdisc fq_codel state UP group default qlen 1000 link/ether 56:be:5a:26:b8:c8 brd ff:ff:ff:ff:ff:ff altname enp0s2 inet 192.168.203.24/24 brd 192.168.203.255 scope global dynamic ens2 valid_lft 59701sec preferred_lft 59701sec inet6 fe80::54be:5aff:fe26:b8c8/64 scope link valid_lft forever preferred_lft forever oleg324-server: /home/green/git/lustre-release/lustre/tests/except/sanity-lnet.ex - see CLIENT_VERSION >= v2_16_51-17-g22ba71dccb8 (34683216 >= 34616081) for LU-18129, go 230 - see CLIENT_VERSION >= v2_16_57-110-ged70ccb471 (34683216 >= 34617710) for LU-15135, go 226 excepting tests: 253 254 410 Cleaning up LNet Writer error: failed to resolve Netlink family id unloading modules via unload_modules_local on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing unload_modules_local oleg324-server: oleg324-server: LNET unconfigure error 22: Invalid argument modules unloaded. == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 11:58:28 (1788969508) Loading LNet and configuring DLC /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import Expected: global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 PASS 0 (10s) == sanity-lnet test 1: Add peer with single nid (tcp) ==== 11:58:38 (1788969518) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp Expected: peer: - primary nid: 1.1.1.1@tcp Multi-Rail: True peer ni: - nid: 1.1.1.1@tcp global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: peer: - primary nid: 1.1.1.1@tcp Multi-Rail: true peer ni: - nid: 1.1.1.1@tcp global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 PASS 1 (8s) == sanity-lnet test 2: Add peer with single nid (o2ib) === 11:58:46 (1788969526) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib Expected: peer: - primary nid: 2.2.2.2@o2ib Multi-Rail: True peer ni: - nid: 2.2.2.2@o2ib global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: peer: - primary nid: 2.2.2.2@o2ib Multi-Rail: true peer ni: - nid: 2.2.2.2@o2ib global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 PASS 2 (8s) == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 11:58:55 (1788969535) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib Expected: peer: - primary nid: 3.3.3.3@tcp Multi-Rail: True peer ni: - nid: 3.3.3.3@tcp - nid: 3.3.3.3@o2ib global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: peer: - primary nid: 3.3.3.3@tcp Multi-Rail: true peer ni: - nid: 3.3.3.3@tcp - nid: 3.3.3.3@o2ib global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 PASS 3 (8s) == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 11:59:03 (1788969543) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure Add peer with nidrange (tcp) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp Expected: peer: - primary nid: 4.4.4.4@tcp Multi-Rail: True peer ni: - nid: 4.4.4.4@tcp - nid: 4.4.4.1@tcp - nid: 4.4.4.2@tcp - nid: 4.4.4.3@tcp global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: peer: - primary nid: 4.4.4.4@tcp Multi-Rail: true peer ni: - nid: 4.4.4.4@tcp - nid: 4.4.4.1@tcp - nid: 4.4.4.2@tcp - nid: 4.4.4.3@tcp global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Add peer with nidrange that overlaps primary nid (tcp) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp Expected: peer: - primary nid: 4.4.4.4@tcp Multi-Rail: True peer ni: - nid: 4.4.4.4@tcp - nid: 4.4.4.1@tcp - nid: 4.4.4.2@tcp - nid: 4.4.4.3@tcp global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: peer: - primary nid: 4.4.4.4@tcp Multi-Rail: true peer ni: - nid: 4.4.4.4@tcp - nid: 4.4.4.1@tcp - nid: 4.4.4.2@tcp - nid: 4.4.4.3@tcp global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 PASS 4 (9s) == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 11:59:12 (1788969552) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure Add peer with nidrange (o2ib) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib Expected: peer: - primary nid: 5.5.5.5@o2ib Multi-Rail: True peer ni: - nid: 5.5.5.5@o2ib - nid: 5.5.5.1@o2ib - nid: 5.5.5.2@o2ib - nid: 5.5.5.3@o2ib - nid: 5.5.5.4@o2ib global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: peer: - primary nid: 5.5.5.5@o2ib Multi-Rail: true peer ni: - nid: 5.5.5.5@o2ib - nid: 5.5.5.1@o2ib - nid: 5.5.5.2@o2ib - nid: 5.5.5.3@o2ib - nid: 5.5.5.4@o2ib global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Add peer with nidranage that overlaps primary nid (o2ib) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib Expected: peer: - primary nid: 5.5.5.5@o2ib Multi-Rail: True peer ni: - nid: 5.5.5.5@o2ib - nid: 5.5.5.1@o2ib - nid: 5.5.5.2@o2ib - nid: 5.5.5.3@o2ib - nid: 5.5.5.4@o2ib global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: peer: - primary nid: 5.5.5.5@o2ib Multi-Rail: true peer ni: - nid: 5.5.5.5@o2ib - nid: 5.5.5.1@o2ib - nid: 5.5.5.2@o2ib - nid: 5.5.5.3@o2ib - nid: 5.5.5.4@o2ib global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Add peer with nidranage that contain , plus primary nid (o2ib) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib Expected: peer: - primary nid: 5.5.5.5@o2ib Multi-Rail: True peer ni: - nid: 5.5.5.5@o2ib - nid: 5.5.5.1@o2ib - nid: 5.5.5.2@o2ib - nid: 5.5.5.3@o2ib - nid: 5.5.5.4@o2ib global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: peer: - primary nid: 5.5.5.5@o2ib Multi-Rail: true peer ni: - nid: 5.5.5.5@o2ib - nid: 5.5.5.1@o2ib - nid: 5.5.5.2@o2ib - nid: 5.5.5.3@o2ib - nid: 5.5.5.4@o2ib global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 PASS 5 (8s) == sanity-lnet test 6: Add peer with multiple nidranges == 11:59:20 (1788969560) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi,[6-12/4]@bxi3f Expected: peer: - primary nid: 6.6.6.6@tcp Multi-Rail: True peer ni: - nid: 6.6.6.6@tcp - nid: 6.6.6.0@tcp - nid: 6.6.6.2@tcp - nid: 6.6.6.4@tcp - nid: 6.6.7.0@tcp - nid: 6.6.7.2@tcp - nid: 6.6.7.4@tcp - nid: 6.6.1.0@o2ib - nid: 6.6.1.3@o2ib - nid: 6.6.1.6@o2ib - nid: 6.6.3.0@o2ib - nid: 6.6.3.3@o2ib - nid: 6.6.3.6@o2ib - nid: 6@gni - nid: 10@gni - nid: 6@kfi - nid: 10@kfi - nid: 6@bxi3f - nid: 10@bxi3f global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: peer: - primary nid: 6.6.6.6@tcp Multi-Rail: true peer ni: - nid: 6.6.6.6@tcp - nid: 6.6.6.0@tcp - nid: 6.6.6.2@tcp - nid: 6.6.6.4@tcp - nid: 6.6.7.0@tcp - nid: 6.6.7.2@tcp - nid: 6.6.7.4@tcp - nid: 6.6.1.0@o2ib - nid: 6.6.1.3@o2ib - nid: 6.6.1.6@o2ib - nid: 6.6.3.0@o2ib - nid: 6.6.3.3@o2ib - nid: 6.6.3.6@o2ib - nid: 6@gni - nid: 10@gni - nid: 6@kfi - nid: 10@kfi - nid: 6@bxi3f - nid: 10@bxi3f global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 PASS 6 (7s) == sanity-lnet test 7: Various peer delete tests ========= 11:59:27 (1788969567) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure Delete peer with single nid (tcp) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp Expected: global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Delete peer with single nid (o2ib) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib Expected: global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Delete peer that has multiple nids (tcp) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp Expected: global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Delete peer that has multiple nids (o2ib) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib Expected: global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Delete peer that has both tcp and o2ib nids /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp Expected: global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Delete peer with single nid (gni) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni Expected: global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Delete peer that has multiple nids (gni) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni Expected: global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Delete peer with single nid (kfi) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi Expected: global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Delete peer that has multiple nids (kfi) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi Expected: global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Delete peer with single nid (bxi3f) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f Expected: global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Delete peer that has multiple nids (bxi3f) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@bxi3f --nid [8-12]@bxi3f /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@bxi3f Expected: global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Delete peer that has tcp, o2ib, gni, kfi and bxi3f nids /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi,[3-7]@bxi3f /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni Expected: global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Delete peer that contains nid equal to primary nid /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp Expected: global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 PASS 7 (14s) == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 11:59:42 (1788969582) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp Expected: peer: - primary nid: 8.8.8.8@tcp Multi-Rail: True peer ni: - nid: 8.8.8.8@tcp - nid: 8.8.8.10@tcp - nid: 8.8.8.11@tcp - nid: 8.8.8.12@tcp - nid: 8.8.8.14@tcp - nid: 8.8.8.15@tcp global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: peer: - primary nid: 8.8.8.8@tcp Multi-Rail: true peer ni: - nid: 8.8.8.8@tcp - nid: 8.8.8.10@tcp - nid: 8.8.8.11@tcp - nid: 8.8.8.12@tcp - nid: 8.8.8.14@tcp - nid: 8.8.8.15@tcp global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 PASS 8 (8s) == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 11:59:50 (1788969590) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp Expected: peer: - primary nid: 9.9.9.9@tcp Multi-Rail: True peer ni: - nid: 9.9.9.9@tcp global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: peer: - primary nid: 9.9.9.9@tcp Multi-Rail: true peer ni: - nid: 9.9.9.9@tcp global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 PASS 9 (8s) == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 11:59:57 (1788969597) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp Expected: peer: - primary nid: 10.10.10.10@tcp Multi-Rail: True peer ni: - nid: 10.10.10.10@tcp - nid: 10.10.10.12@tcp - nid: 10.10.10.13@tcp - nid: 10.10.10.15@tcp - nid: 10.10.10.16@tcp global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: peer: - primary nid: 10.10.10.10@tcp Multi-Rail: true peer ni: - nid: 10.10.10.10@tcp - nid: 10.10.10.12@tcp - nid: 10.10.10.13@tcp - nid: 10.10.10.15@tcp - nid: 10.10.10.16@tcp global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 PASS 10 (8s) == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 12:00:05 (1788969605) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp Expected: peer: - primary nid: 11.11.11.11@tcp Multi-Rail: True peer ni: - nid: 11.11.11.11@tcp global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: peer: - primary nid: 11.11.11.11@tcp Multi-Rail: true peer ni: - nid: 11.11.11.11@tcp global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 PASS 11 (9s) == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 12:00:14 (1788969614) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp Expected: peer: - primary nid: 12.12.12.12@o2ib Multi-Rail: True peer ni: - nid: 12.12.12.12@o2ib - nid: 13.13.13.13@o2ib - nid: 14.13.13.13@o2ib - nid: 14.15.13.13@o2ib - nid: 15.17.1.5@tcp - nid: 15.17.1.10@tcp - nid: 15.17.1.20@tcp global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: peer: - primary nid: 12.12.12.12@o2ib Multi-Rail: true peer ni: - nid: 12.12.12.12@o2ib - nid: 13.13.13.13@o2ib - nid: 14.13.13.13@o2ib - nid: 14.15.13.13@o2ib - nid: 15.17.1.5@tcp - nid: 15.17.1.10@tcp - nid: 15.17.1.20@tcp global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 PASS 12 (8s) == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 12:00:23 (1788969623) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib Expected: peer: - primary nid: 13.13.13.13@o2ib Multi-Rail: True peer ni: - nid: 13.13.13.13@o2ib global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: peer: - primary nid: 13.13.13.13@o2ib Multi-Rail: true peer ni: - nid: 13.13.13.13@o2ib global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 PASS 13 (7s) == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 12:00:30 (1788969630) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure Create single peer, single nid, using import Generating peer yaml for 1 peers with 0 secondary nids /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import Expected: peer: - primary nid: 1.1.1.1@tcp Multi-Rail: True peer ni: - nid: 1.1.1.1@tcp global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: peer: - primary nid: 1.1.1.1@tcp Multi-Rail: true peer ni: - nid: 1.1.1.1@tcp global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Delete single peer using import --del /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del Expected: global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 PASS 14 (7s) == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 12:00:37 (1788969637) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure Create multiple peers, single nid per peer, using import Generating peer yaml for 5 peers with 0 secondary nids /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import Delete multiple peers, single nid per peer, using import --del /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del Expected: global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 PASS 15 (7s) == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 12:00:45 (1788969645) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure Create single peer, multiple nids, using import Generating peer yaml for 1 peers with 5 secondary nids /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import Delete single peer, multiple nids, using import --del /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del Expected: global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 PASS 16 (8s) == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 12:00:52 (1788969652) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure Create multiple peers, multiple nids per peer, using import Generating peer yaml for 5 peers with 7 secondary nids /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import Delete multiple peers, multiple nids per peer, using import --del /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del Expected: global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 PASS 17 (8s) == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 12:01:01 (1788969661) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure Import peer with 5 nids peer: - primary nid: 1.1.1.1@tcp Multi-Rail: True peer ni: - nid: 1.1.1.1@tcp - nid: 2.2.2.2@tcp - nid: 4.4.4.4@tcp - nid: 3.3.3.3@o2ib - nid: 5@gni /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import Delete three of the nids peer: - primary nid: 1.1.1.1@tcp Multi-Rail: True peer ni: - nid: 2.2.2.2@tcp - nid: 3.3.3.3@o2ib - nid: 5@gni /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del Check peer has expected nids remaining Expected: peer: - primary nid: 1.1.1.1@tcp Multi-Rail: True peer ni: - nid: 1.1.1.1@tcp - nid: 4.4.4.4@tcp global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: peer: - primary nid: 1.1.1.1@tcp Multi-Rail: true peer ni: - nid: 1.1.1.1@tcp - nid: 4.4.4.4@tcp global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 PASS 18a (9s) == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 12:01:09 (1788969669) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure Import two peers with 5 nids each peer: - primary nid: 1.1.1.1@tcp Multi-Rail: True peer ni: - nid: 1.1.1.1@tcp - nid: 2.2.2.2@tcp - nid: 4.4.4.4@tcp - nid: 3.3.3.3@o2ib - nid: 5@gni - primary nid: 6.6.6.6@o2ib Multi-Rail: True peer ni: - nid: 6.6.6.6@o2ib - nid: 7.7.7.7@tcp - nid: 8.8.8.8@tcp - nid: 9.9.9.9@tcp - nid: 10@gni /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import Delete three of the nids from each peer peer: - primary nid: 1.1.1.1@tcp Multi-Rail: True peer ni: - nid: 2.2.2.2@tcp - nid: 3.3.3.3@o2ib - nid: 5@gni - primary nid: 6.6.6.6@o2ib Multi-Rail: True peer ni: - nid: 7.7.7.7@tcp - nid: 8.8.8.8@tcp - nid: 10@gni /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del Check peers have expected nids remaining Expected: peer: - primary nid: 6.6.6.6@o2ib Multi-Rail: True peer ni: - nid: 6.6.6.6@o2ib - nid: 7.7.7.7@tcp - primary nid: 1.1.1.1@tcp Multi-Rail: True peer ni: - nid: 1.1.1.1@tcp - nid: 4.4.4.4@tcp global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: peer: - primary nid: 1.1.1.1@tcp Multi-Rail: true peer ni: - nid: 1.1.1.1@tcp - nid: 4.4.4.4@tcp - primary nid: 6.6.6.6@o2ib Multi-Rail: true peer ni: - nid: 6.6.6.6@o2ib - nid: 9.9.9.9@tcp global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 PASS 18b (8s) == sanity-lnet test 19: Add peer with single nid (gni) === 12:01:17 (1788969677) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni Expected: peer: - primary nid: 19@gni Multi-Rail: True peer ni: - nid: 19@gni global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: peer: - primary nid: 19@gni Multi-Rail: true peer ni: - nid: 19@gni global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 PASS 19 (8s) == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 12:01:25 (1788969685) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib Expected: peer: - primary nid: 20@gni Multi-Rail: True peer ni: - nid: 20@gni - nid: 20.20.20.20@tcp - nid: 20.20.20.20@o2ib global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: peer: - primary nid: 20@gni Multi-Rail: true peer ni: - nid: 20@gni - nid: 20.20.20.20@tcp - nid: 20.20.20.20@o2ib global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 PASS 20 (8s) == sanity-lnet test 21: Add peer with nidrange (gni) ===== 12:01:33 (1788969693) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure Add peer with nidrange (gni) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni Expected: peer: - primary nid: 21@gni Multi-Rail: True peer ni: - nid: 21@gni - nid: 22@gni - nid: 23@gni - nid: 24@gni - nid: 25@gni global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: peer: - primary nid: 21@gni Multi-Rail: true peer ni: - nid: 21@gni - nid: 22@gni - nid: 23@gni - nid: 24@gni - nid: 25@gni global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Add peer with nidrange that overlaps primary nid (gni) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni Expected: peer: - primary nid: 21@gni Multi-Rail: True peer ni: - nid: 21@gni - nid: 22@gni - nid: 23@gni - nid: 24@gni - nid: 25@gni global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: peer: - primary nid: 21@gni Multi-Rail: true peer ni: - nid: 21@gni - nid: 22@gni - nid: 23@gni - nid: 24@gni - nid: 25@gni global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 PASS 21 (9s) == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 12:01:42 (1788969702) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni Expected: peer: - primary nid: 22@gni Multi-Rail: True peer ni: - nid: 22@gni - nid: 24@gni - nid: 25@gni - nid: 27@gni - nid: 28@gni - nid: 29@gni global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: peer: - primary nid: 22@gni Multi-Rail: true peer ni: - nid: 22@gni - nid: 24@gni - nid: 25@gni - nid: 27@gni - nid: 28@gni - nid: 29@gni global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 PASS 22 (9s) == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 12:01:51 (1788969711) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni Expected: peer: - primary nid: 23@gni Multi-Rail: True peer ni: - nid: 23@gni global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: peer: - primary nid: 23@gni Multi-Rail: true peer ni: - nid: 23@gni global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 PASS 23 (8s) == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 12:02:00 (1788969720) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp Expected: peer: - primary nid: 24@gni Multi-Rail: True peer ni: - nid: 24@gni - nid: 11@gni - nid: 13.13.13.13@o2ib - nid: 14.13.13.13@o2ib - nid: 14.15.13.13@o2ib - nid: 15.17.1.5@tcp - nid: 15.17.1.10@tcp - nid: 15.17.1.20@tcp global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: peer: - primary nid: 24@gni Multi-Rail: true peer ni: - nid: 24@gni - nid: 11@gni - nid: 13.13.13.13@o2ib - nid: 14.13.13.13@o2ib - nid: 14.15.13.13@o2ib - nid: 15.17.1.5@tcp - nid: 15.17.1.10@tcp - nid: 15.17.1.20@tcp global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 PASS 24 (10s) == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 12:02:09 (1788969729) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni Expected: peer: - primary nid: 25@gni Multi-Rail: True peer ni: - nid: 25@gni global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: peer: - primary nid: 25@gni Multi-Rail: true peer ni: - nid: 25@gni global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 PASS 25 (8s) == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 12:02:17 (1788969737) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 PASS 26 (8s) == sanity-lnet test 27: Import bad config should fail gracefully ========================================================== 12:02:25 (1788969745) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 PASS 27 (5s) == sanity-lnet test 28: Test peer_list =================== 12:02:30 (1788969750) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --ni 7.7.7.7@tcp /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.2@tcp --nid 7.7.7.8@tcp pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 PASS 28 (6s) == sanity-lnet test 40: LNetAddPeer merges a pre-existing non-primary peer NID ========================================================== 12:02:36 (1788969756) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.24@tcp discover: - primary nid: 192.168.203.24@tcp Multi-Rail: true peer_ni: - nid: 192.168.203.24@tcp /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system Added uuid lnet_add_peer_uuid: 192.168.203.24@tcp99 Added uuid lnet_add_peer_uuid: 192.168.203.24@tcp pdsh@oleg324-client: oleg324-client: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-client: ssh exited with exit code 2 PASS 40 (25s) == sanity-lnet test 50: Enable/disable routing via yaml import ========================================================== 12:03:01 (1788969781) Cleaning up LNet LNET unconfigure error 22: Invalid argument unloading modules via unload_modules_local on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing unload_modules_local oleg324-server: oleg324-server: LNET unconfigure error 22: Invalid argument modules unloaded. Loading LNet and configuring DLC /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ../lnet/lnet/lnet options: 'tiny_router_buffers=16 small_router_buffers=8 large_router_buffers=4' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure Enable routing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml Waiting 10s for '1' Updated after 4s: want '1' got '1' Disable routing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml Waiting 10s for '1' Updated after 3s: want '1' got '1' Enable routing w/custom buffer sizses /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-50.yaml 6 10 18 Cleaning up LNet Writer error: failed to resolve Netlink family id unloading modules via unload_modules_local on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing unload_modules_local oleg324-server: oleg324-server: LNET unconfigure error 22: Invalid argument modules unloaded. pdsh@oleg324-client: oleg324-client: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-client: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 PASS 50 (34s) == sanity-lnet test 51: Set numa range via yaml import === 12:03:35 (1788969815) Loading LNet and configuring DLC /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-51.yaml pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 PASS 51 (7s) == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 12:03:43 (1788969823) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure Invalid prim_nid - peer add /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar Invalid prim_nid - peer del /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar Delete non-existing peer /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib Don't provide mandatory argument for peer del /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp Don't provide mandatory argument for peer add /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp Don't provide mandatory arguments peer add /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add peer add: add a peer NID --prim_nid: Primary NID of the peer. --nid: one or more peer NIDs --non_mr: create this peer as not Multi-Rail capable --ip2nets: specify a range of nids per peer --lock_prim: lock primary nid Invalid secondary nids /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar --- add: - peer: errno: -22 descr: ! "\"Unable to parse nidlist: no valid NIDs in string\"" ... can't parse nidrange: "foobar" Exceed max nids per peer /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp --- add: - peer: errno: -7 descr: ! "Unable to parse nidlist: specifies more NIDs than allowed" ... Invalid net type /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo Invalid nid format Check invalid primary nid - '1@tcp' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp Check invalid primary nid - '1@o2ib' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib Check invalid primary nid - '1.1.1.1@gni' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni Check invalid nidstring - '[2-1]@gni' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni --- add: - peer: errno: -34 descr: ! "\"Unable to parse nidlist: range is wrong order\"" ... can't parse nidrange: "[2-1]" Check invalid nidstring - '[a-f/x]@gni' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni --- add: - peer: errno: -34 descr: ! "\"Unable to parse nidlist: range needs number after /\"" ... can't parse nidrange: "[a-f/x]" Check invalid nidstring - '256.256.256.256@tcp' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp --- add: - peer: errno: -22 descr: ! "\"Unable to parse nidlist: invalid NID in nidstr\"" ... can't parse nidrange: "256.256.256.256" Check invalid nidstring - '1.1.1.1.[2-5/f]@tcp' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp --- add: - peer: errno: -34 descr: ! "\"Unable to parse nidlist: range needs number after /\"" ... can't parse nidrange: "1.1.1.1.[2-5/f]" Check invalid nidstring - '1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp --- add: - peer: errno: -22 descr: ! "\"Unable to parse nidlist: improper bracket ordering\"" ... can't parse nidrange: "1.]2[.3.4" Check invalid nidstring - 'foobar' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar --- add: - peer: errno: -22 descr: ! "\"Unable to parse nidlist: no valid NIDs in string\"" ... can't parse nidrange: "foobar" Add non-local gateway /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni --- add: - route: errno: -113 descr: ! "No interface configured on the same net as gateway" ... pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 PASS 99a (7s) == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 12:03:51 (1788969831) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --- add: - import: errno: -22 descr: ! "invalid MR flag param" ... Expected: global: numa_range: 1 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: global: numa_range: 1 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 PASS 99b (8s) == sanity-lnet test 100: Add route with single gw ======== 12:03:58 (1788969838) SKIP: sanity-lnet test_100 Need at least 2 remote nodes found "oleg324-server" SKIP 100 (4s) Total IP for ens2 found [1] Interface:IP are ens2:192.168.203.24 Using GW_NID:192.168.203.25@tcp == sanity-lnet test 101: Add route with multiple gw ====== 12:04:03 (1788969843) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.24 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp1 --gateway 192.168.203.[25-29]@tcp /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl export --backup > /tmp/sanity-lnet-101-actual.yaml expect_gw: 5 actual_gw: 5 expect_gwnids: 192.168.203.25@tcp 192.168.203.26@tcp 192.168.203.27@tcp 192.168.203.28@tcp 192.168.203.29@tcp pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 PASS 101 (9s) == sanity-lnet test 102: Delete route with single gw ===== 12:04:11 (1788969851) SKIP: sanity-lnet test_102 Need at least 2 remote nodes found "oleg324-server" SKIP 102 (4s) == sanity-lnet test 103: Delete route with multiple gw === 12:04:15 (1788969855) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.24 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.203.[25-30/2]@tcp /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.203.[25-30/2]@tcp expect_gw: 0 actual_gw: 0 expect_gwnids: pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 PASS 103 (8s) == sanity-lnet test 104: Set/check response_tracking param ========================================================== 12:04:23 (1788969863) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure Set < 0; Should fail /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 add: - response_tracking: errno: -22 descr: "Valid values are: 0, 1, 2, or 3" /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import Check valid values; Should succeed /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure Set > 3; Should fail /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 add: - response_tracking: errno: -22 descr: "Valid values are: 0, 1, 2, or 3" /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 PASS 104 (11s) == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 12:04:34 (1788969874) SKIP: sanity-lnet test_105 Need at least 2 remote nodes found "oleg324-server" SKIP 105 (5s) == sanity-lnet test 106: Deleting GW peer should fail ==== 12:04:39 (1788969879) SKIP: sanity-lnet test_106 Need at least 2 remote nodes found "oleg324-server" SKIP 106 (4s) == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 12:04:43 (1788969883) Cleaning up LNet Writer error: failed to resolve Netlink family id unloading modules via unload_modules_local on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing unload_modules_local oleg324-server: oleg324-server: LNET unconfigure error 22: Invalid argument modules unloaded. ip link add 'test1pl' type veth peer name test1pg ip link set 'test1pl' up ip addr add "10.1.2.3/31" dev test1pg ip -6 addr add "2001:db8:0:f101::1/64" dev test1pg ip link set test1pg up Waiting 10s for '0' Updated after 3s: want '0' got '0' Loading LNet and configuring DLC /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.24 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.24 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg Cleaning up LNet Writer error: failed to resolve Netlink family id unloading modules via unload_modules_local on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing unload_modules_local oleg324-server: oleg324-server: LNET unconfigure error 22: Invalid argument modules unloaded. pdsh@oleg324-client: oleg324-client: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-client: ssh exited with exit code 2 PASS 107 (34s) == sanity-lnet test 108: Check Multi-Rail setup ========== 12:05:18 (1788969918) Cleaning up LNet LNET unconfigure error 22: Invalid argument unloading modules via unload_modules_local on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing unload_modules_local oleg324-server: oleg324-server: LNET unconfigure error 22: Invalid argument modules unloaded. ip link add 'test1pl' type veth peer name test1pg ip link set 'test1pl' up ip addr add "10.1.2.3/31" dev test1pg ip -6 addr add "2001:db8:0:f101::1/64" dev test1pg ip link set test1pg up Waiting 10s for '0' Updated after 2s: want '0' got '0' Loading LNet and configuring DLC /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.24 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.24 Expected: net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.203.24@tcp status: up interfaces: 0: ens2 - nid: 10.1.2.3@tcp status: up interfaces: 0: test1pg Actual: net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.203.24@tcp status: up interfaces: 0: ens2 - nid: 10.1.2.3@tcp status: up interfaces: 0: test1pg Cleaning up LNet Writer error: failed to resolve Netlink family id unloading modules via unload_modules_local on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing unload_modules_local oleg324-server: oleg324-server: LNET unconfigure error 22: Invalid argument modules unloaded. pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-client: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-client: ssh exited with exit code 2 PASS 108 (30s) == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 12:05:48 (1788969948) Cleaning up LNet LNET unconfigure error 22: Invalid argument unloading modules via unload_modules_local on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing unload_modules_local oleg324-server: oleg324-server: LNET unconfigure error 22: Invalid argument modules unloaded. ip link add 'test1pl' type veth peer name test1pg ip link set 'test1pl' up ip addr add "10.1.2.3/31" dev test1pg ip -6 addr add "2001:db8:0:f101::1/64" dev test1pg ip link set test1pg up Waiting 10s for '0' Updated after 2s: want '0' got '0' ip addr add "10.1.2.31/31" dev test1pg label test1pg:0 Loading LNet and configuring DLC /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg 10.1.2.2/31 dev test1pg proto kernel scope link src 10.1.2.3 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg 10.1.2.2/31 dev test1pg proto kernel scope link src 10.1.2.3 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 10.1.2.2/31 dev test1pg proto kernel scope link src 10.1.2.3 10.1.2.2/31 dev test1pg proto kernel scope link src 10.1.2.3 Cleaning up LNet Writer error: failed to resolve Netlink family id unloading modules via unload_modules_local on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing unload_modules_local oleg324-server: oleg324-server: LNET unconfigure error 22: Invalid argument modules unloaded. pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-client: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-client: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 PASS 109 (36s) == sanity-lnet test 110: Configure NI using lnetctl net add --nid ========================================================== 12:06:24 (1788969984) Loading LNet and configuring DLC /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.24 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.203.24@tcp default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.24 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.203.24@tcp Cleaning up LNet Writer error: failed to resolve Netlink family id unloading modules via unload_modules_local on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing unload_modules_local oleg324-server: oleg324-server: LNET unconfigure error 22: Invalid argument modules unloaded. pdsh@oleg324-client: oleg324-client: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-client: ssh exited with exit code 2 PASS 110 (18s) == sanity-lnet test 111: Test many routes ================ 12:06:43 (1788970003) SKIP: sanity-lnet test_111 Need at least 2 remote nodes found "oleg324-server" SKIP 111 (4s) == sanity-lnet test 112: multiple net configurations ===== 12:06:46 (1788970006) Cleaning up LNet LNET unconfigure error 22: Invalid argument unloading modules via unload_modules_local on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing unload_modules_local oleg324-server: oleg324-server: LNET unconfigure error 22: Invalid argument modules unloaded. ip link add 'test1pl' type veth peer name test1pg ip link set 'test1pl' up ip addr add "10.1.2.3/31" dev test1pg ip -6 addr add "2001:db8:0:f101::1/64" dev test1pg ip link set test1pg up Waiting 10s for '0' Updated after 3s: want '0' got '0' Loading LNet and configuring DLC /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.24 default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.24 Cleaning up LNet Writer error: failed to resolve Netlink family id unloading modules via unload_modules_local on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing unload_modules_local oleg324-server: oleg324-server: LNET unconfigure error 22: Invalid argument modules unloaded. pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-client: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-client: ssh exited with exit code 2 PASS 112 (28s) == sanity-lnet test 150: Check import of ip2nets yaml sequence ========================================================== 12:07:15 (1788970035) Loading LNet and configuring DLC /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.24 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure ip2nets: - net-spec: tcp interfaces: 0: ens2 ip-range: 0: 192.168.203.24 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet.yaml Expected: net: - net type: tcp local NI(s): - interfaces: 0: ens2 tunables: peer_timeout: 180 peer_credits: 8 peer_buffer_credits: 0 credits: 256 lnd tunables: conns_per_peer: 1 tos: -1 CPT: "[0]" global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: net: - net type: tcp local NI(s): - interfaces: 0: ens2 tunables: peer_timeout: 180 peer_credits: 8 peer_buffer_credits: 0 credits: 256 lnd tunables: conns_per_peer: 1 tos: -1 CPT: "[0]" global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 PASS 150 (9s) == sanity-lnet test 151: Check correct application of multiple ip2nets rules ========================================================== 12:07:24 (1788970044) Cleaning up LNet Writer error: failed to resolve Netlink family id unloading modules via unload_modules_local on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing unload_modules_local oleg324-server: oleg324-server: LNET unconfigure error 22: Invalid argument modules unloaded. ip link add 'test1pl' type veth peer name test1pg ip link set 'test1pl' up ip addr add "10.1.2.3/31" dev test1pg ip -6 addr add "2001:db8:0:f101::1/64" dev test1pg ip link set test1pg up Waiting 10s for '0' Updated after 2s: want '0' got '0' Loading LNet and configuring DLC /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.24 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.24 Check import of ip2nets with two matching interfaces /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure ip2nets: - net-spec: tcp interfaces: 0: ens2 1: test1pg ip-range: 0: 192.168.203.24 1: 10.1.2.3 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml Expected: net: - net type: tcp local NI(s): - interfaces: 0: ens2 tunables: peer_timeout: 180 peer_credits: 8 peer_buffer_credits: 0 credits: 256 lnd tunables: conns_per_peer: 1 tos: -1 CPT: "[0,1]" - interfaces: 0: test1pg tunables: peer_timeout: 180 peer_credits: 8 peer_buffer_credits: 0 credits: 256 lnd tunables: conns_per_peer: 2 tos: -1 CPT: "[0,1]" global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: net: - net type: tcp local NI(s): - interfaces: 0: ens2 tunables: peer_timeout: 180 peer_credits: 8 peer_buffer_credits: 0 credits: 256 lnd tunables: conns_per_peer: 1 tos: -1 CPT: "[0,1]" - interfaces: 0: test1pg tunables: peer_timeout: 180 peer_credits: 8 peer_buffer_credits: 0 credits: 256 lnd tunables: conns_per_peer: 2 tos: -1 CPT: "[0,1]" global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Additional rules for same net should not change config /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure ip2nets: - net-spec: tcp interfaces: 0: ens2 1: test1pg ip-range: 0: 192.168.203.24 1: 10.1.2.3 - net-spec: tcp interfaces: 0: ens2 - net-spec: tcp interfaces: 0: test1pg ip-range: 0: 10.1.2.3 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml Expected: net: - net type: tcp local NI(s): - interfaces: 0: ens2 tunables: peer_timeout: 180 peer_credits: 8 peer_buffer_credits: 0 credits: 256 lnd tunables: conns_per_peer: 1 tos: -1 CPT: "[0,1]" - interfaces: 0: test1pg tunables: peer_timeout: 180 peer_credits: 8 peer_buffer_credits: 0 credits: 256 lnd tunables: conns_per_peer: 2 tos: -1 CPT: "[0,1]" global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: net: - net type: tcp local NI(s): - interfaces: 0: ens2 tunables: peer_timeout: 180 peer_credits: 8 peer_buffer_credits: 0 credits: 256 lnd tunables: conns_per_peer: 1 tos: -1 CPT: "[0,1]" - interfaces: 0: test1pg tunables: peer_timeout: 180 peer_credits: 8 peer_buffer_credits: 0 credits: 256 lnd tunables: conns_per_peer: 2 tos: -1 CPT: "[0,1]" global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.24 10.1.2.2/31 dev test1pg proto kernel scope link src 10.1.2.3 default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.24 Additional rule for second net should apply /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure ip2nets: - net-spec: tcp interfaces: 0: ens2 1: test1pg ip-range: 0: 192.168.203.24 1: 10.1.2.3 - net-spec: tcp interfaces: 0: ens2 - net-spec: tcp interfaces: 0: test1pg ip-range: 0: 10.1.2.3 - net-spec: tcp2 interfaces: 0: ens2 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-151.yaml Expected: net: - net type: tcp local NI(s): - interfaces: 0: ens2 tunables: peer_timeout: 180 peer_credits: 8 peer_buffer_credits: 0 credits: 256 lnd tunables: conns_per_peer: 1 tos: -1 CPT: "[0,1]" - interfaces: 0: test1pg tunables: peer_timeout: 180 peer_credits: 8 peer_buffer_credits: 0 credits: 256 lnd tunables: conns_per_peer: 2 tos: -1 CPT: "[0,1]" - net type: tcp2 local NI(s): - interfaces: 0: ens2 tunables: peer_timeout: 180 peer_credits: 8 peer_buffer_credits: 0 credits: 256 lnd tunables: conns_per_peer: 1 tos: -1 CPT: "[0,1]" global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: net: - net type: tcp local NI(s): - interfaces: 0: ens2 tunables: peer_timeout: 180 peer_credits: 8 peer_buffer_credits: 0 credits: 256 lnd tunables: conns_per_peer: 1 tos: -1 CPT: "[0,1]" - interfaces: 0: test1pg tunables: peer_timeout: 180 peer_credits: 8 peer_buffer_credits: 0 credits: 256 lnd tunables: conns_per_peer: 2 tos: -1 CPT: "[0,1]" - net type: tcp2 local NI(s): - interfaces: 0: ens2 tunables: peer_timeout: 180 peer_credits: 8 peer_buffer_credits: 0 credits: 256 lnd tunables: conns_per_peer: 1 tos: -1 CPT: "[0,1]" global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 PASS 151 (26s) == sanity-lnet test 152: Check import of ip2nets with wildcard IP patterns ========================================================== 12:07:50 (1788970070) Cleaning up LNet Writer error: failed to resolve Netlink family id unloading modules via unload_modules_local on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing unload_modules_local oleg324-server: oleg324-server: LNET unconfigure error 22: Invalid argument modules unloaded. Loading LNet and configuring DLC /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.24 Check import of ip2nets with wildcard IP patterns /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure ip2nets: - net-spec: tcp interfaces: 0: ens2 ip-range: 0: 192.168.203.* /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml Expected: net: - net type: tcp local NI(s): - interfaces: 0: ens2 tunables: peer_timeout: 180 peer_credits: 8 peer_buffer_credits: 0 credits: 256 lnd tunables: conns_per_peer: 1 tos: -1 CPT: "[0,1]" global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: net: - net type: tcp local NI(s): - interfaces: 0: ens2 tunables: peer_timeout: 180 peer_credits: 8 peer_buffer_credits: 0 credits: 256 lnd tunables: conns_per_peer: 1 tos: -1 CPT: "[0,1]" global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure ip2nets: - net-spec: tcp interfaces: 0: ens2 ip-range: 0: 192.168.*.* /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml Expected: net: - net type: tcp local NI(s): - interfaces: 0: ens2 tunables: peer_timeout: 180 peer_credits: 8 peer_buffer_credits: 0 credits: 256 lnd tunables: conns_per_peer: 1 tos: -1 CPT: "[0,1]" global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: net: - net type: tcp local NI(s): - interfaces: 0: ens2 tunables: peer_timeout: 180 peer_credits: 8 peer_buffer_credits: 0 credits: 256 lnd tunables: conns_per_peer: 1 tos: -1 CPT: "[0,1]" global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure ip2nets: - net-spec: tcp interfaces: 0: ens2 ip-range: 0: 192.*.*.* /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml Expected: net: - net type: tcp local NI(s): - interfaces: 0: ens2 tunables: peer_timeout: 180 peer_credits: 8 peer_buffer_credits: 0 credits: 256 lnd tunables: conns_per_peer: 1 tos: -1 CPT: "[0,1]" global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: net: - net type: tcp local NI(s): - interfaces: 0: ens2 tunables: peer_timeout: 180 peer_credits: 8 peer_buffer_credits: 0 credits: 256 lnd tunables: conns_per_peer: 1 tos: -1 CPT: "[0,1]" global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 ip link add 'test1pl' type veth peer name test1pg ip link set 'test1pl' up ip addr add "10.1.2.3/31" dev test1pg ip -6 addr add "2001:db8:0:f101::1/64" dev test1pg ip link set test1pg up Waiting 10s for '0' Check match-all expression mask later rules /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure ip2nets: - net-spec: tcp interfaces: 0: ens2 ip-range: 0: "*.*.*.*" - net-spec: tcp interfaces: 0: ens2 1: test1pg - net-spec: tcp interfaces: 0: ens2 1: test1pg ip-range: 0: 192.168.203.24 1: 10.1.2.3 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-152.yaml Expected: net: - net type: tcp local NI(s): - interfaces: 0: ens2 tunables: peer_timeout: 180 peer_credits: 8 peer_buffer_credits: 0 credits: 256 lnd tunables: conns_per_peer: 1 tos: -1 CPT: "[0,1]" global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: net: - net type: tcp local NI(s): - interfaces: 0: ens2 tunables: peer_timeout: 180 peer_credits: 8 peer_buffer_credits: 0 credits: 256 lnd tunables: conns_per_peer: 1 tos: -1 CPT: "[0,1]" global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 PASS 152 (21s) == sanity-lnet test 153: Check import of ip2nets with IP address ranges ========================================================== 12:08:12 (1788970092) Cleaning up LNet Writer error: failed to resolve Netlink family id unloading modules via unload_modules_local on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing unload_modules_local oleg324-server: oleg324-server: LNET unconfigure error 22: Invalid argument modules unloaded. Loading LNet and configuring DLC /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.24 Check import of ip2nets with IP address ranges /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure ip2nets: - net-spec: tcp interfaces: 0: ens2 ip-range: 0: 192.168.203.[20-28] /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-153.yaml Expected: net: - net type: tcp local NI(s): - interfaces: 0: ens2 tunables: peer_timeout: 180 peer_credits: 8 peer_buffer_credits: 0 credits: 256 lnd tunables: conns_per_peer: 1 tos: -1 CPT: "[0,1]" global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: net: - net type: tcp local NI(s): - interfaces: 0: ens2 tunables: peer_timeout: 180 peer_credits: 8 peer_buffer_credits: 0 credits: 256 lnd tunables: conns_per_peer: 1 tos: -1 CPT: "[0,1]" global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 PASS 153 (17s) == sanity-lnet test 154: Check import of ip2nets with stepped IP address ranges ========================================================== 12:08:29 (1788970109) Cleaning up LNet Writer error: failed to resolve Netlink family id unloading modules via unload_modules_local on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing unload_modules_local oleg324-server: oleg324-server: LNET unconfigure error 22: Invalid argument modules unloaded. ip link add 'test1pl' type veth peer name test1pg ip link set 'test1pl' up ip addr add "10.1.2.3/31" dev test1pg ip -6 addr add "2001:db8:0:f101::1/64" dev test1pg ip link set test1pg up Waiting 10s for '0' Updated after 2s: want '0' got '0' Loading LNet and configuring DLC /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.24 Check import of ip2nets with stepped IP address ranges /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure ip2nets: - net-spec: tcp interfaces: 0: ens2 ip-range: 0: 192.168.203.[20-28/2] /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-154.yaml Expected: net: - net type: tcp local NI(s): - interfaces: 0: ens2 tunables: peer_timeout: 180 peer_credits: 8 peer_buffer_credits: 0 credits: 256 lnd tunables: conns_per_peer: 1 tos: -1 CPT: "[0,1]" global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: net: - net type: tcp local NI(s): - interfaces: 0: ens2 tunables: peer_timeout: 180 peer_credits: 8 peer_buffer_credits: 0 credits: 256 lnd tunables: conns_per_peer: 1 tos: -1 CPT: "[0,1]" global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 PASS 154 (20s) == sanity-lnet test 155: Check ip2nets import failure with no matching rules ========================================================== 12:08:49 (1788970129) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure Check import with no matching rules (should fail) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-155.yaml --- add: - import: errno: -22 descr: ! "No ip2nets rules applied" ... pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 PASS 155 (12s) == sanity-lnet test 156: Check import of ip2nets with comma-separated IP ranges ========================================================== 12:09:02 (1788970142) Cleaning up LNet Writer error: failed to resolve Netlink family id unloading modules via unload_modules_local on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing unload_modules_local oleg324-server: oleg324-server: LNET unconfigure error 22: Invalid argument modules unloaded. Loading LNet and configuring DLC /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.24 Check import of ip2nets with comma-separated IP list /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure ip2nets: - net-spec: tcp interfaces: 0: ens2 ip-range: 0: 192.168.203.[26,28,24,30,32] /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-156.yaml Expected: net: - net type: tcp local NI(s): - interfaces: 0: ens2 tunables: peer_timeout: 180 peer_credits: 8 peer_buffer_credits: 0 credits: 256 lnd tunables: conns_per_peer: 1 tos: -1 CPT: "[0,1]" global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: net: - net type: tcp local NI(s): - interfaces: 0: ens2 tunables: peer_timeout: 180 peer_credits: 8 peer_buffer_credits: 0 credits: 256 lnd tunables: conns_per_peer: 1 tos: -1 CPT: "[0,1]" global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 PASS 156 (20s) == sanity-lnet test 157: Check import failure with malformed ip2nets YAML ========================================================== 12:09:22 (1788970162) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure Check import failure with missing net-spec /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-157.yaml --- add: - import: errno: -22 descr: ! "Malformed input. Expect 'net-spec' found 'interfaces'" ... pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 PASS 157 (7s) == sanity-lnet test 158: Check import of ip2nets with single IPv6 address ========================================================== 12:09:29 (1788970169) SKIP: sanity-lnet test_158 Interface has no IPv6 SKIP 158 (3s) == sanity-lnet test 159: Check import of ip2nets with IPv6 /128 CIDR notation ========================================================== 12:09:32 (1788970172) SKIP: sanity-lnet test_159 Interface has no IPv6 SKIP 159 (4s) == sanity-lnet test 160: Check correct application of multiple IPv6 ip2nets rules ========================================================== 12:09:36 (1788970176) SKIP: sanity-lnet test_160 Interface has no IPv6 SKIP 160 (3s) == sanity-lnet test 161: Check import of ip2nets with various IPv6 prefix lengths ========================================================== 12:09:39 (1788970179) SKIP: sanity-lnet test_161 Interface has no IPv6 SKIP 161 (4s) == sanity-lnet test 162: Check ip2nets import failure with non-matching IPv6 ========================================================== 12:09:43 (1788970183) SKIP: sanity-lnet test_162 Interface has no IPv6 SKIP 162 (3s) == sanity-lnet test 170: Check CPTs and tunables parsing when importing ========================================================== 12:09:46 (1788970186) Cleaning up LNet Writer error: failed to resolve Netlink family id unloading modules via unload_modules_local on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing unload_modules_local oleg324-server: oleg324-server: LNET unconfigure error 22: Invalid argument modules unloaded. ip link add 'test1pl' type veth peer name test1pg ip link set 'test1pl' up ip addr add "10.1.2.3/31" dev test1pg ip -6 addr add "2001:db8:0:f101::1/64" dev test1pg ip link set test1pg up Waiting 10s for '0' Updated after 2s: want '0' got '0' Loading LNet and configuring DLC /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure Different CPTs /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure lnd_tunables not read if no tunables /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure lnd_tunables cannot be different /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure Wrong tunables (lnd_tunables) read from NI if first is empty /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure Tunables can be set directly in the net /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure Show warning if we try to set different tunables (not lnd_tunables) for NIs of a same net /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --old-api Cleaning up LNet Writer error: failed to resolve Netlink family id unloading modules via unload_modules_local on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing unload_modules_local oleg324-server: oleg324-server: LNET unconfigure error 22: Invalid argument modules unloaded. pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-client: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-client: ssh exited with exit code 2 PASS 170 (27s) == sanity-lnet test 180: FastReg map error should not leak descriptors ========================================================== 12:10:14 (1788970214) SKIP: sanity-lnet test_180 Need o2ib NETTYPE SKIP 180 (4s) == sanity-lnet test 199: load lnet w/o module option, configure interface twice ========================================================== 12:10:17 (1788970217) Loading LNet and configuring DLC /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure Add interface ens2 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.24 Add IP address 192.168.203.24 for interface ens2 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --ip2net tcp 192.168.203.24 --- add: - net: errno: -17 descr: ! "cannot setup ip2nets" ... pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 PASS 199 (9s) == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 12:10:26 (1788970226) Cleaning up LNet Writer error: failed to resolve Netlink family id unloading modules via unload_modules_local on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing unload_modules_local oleg324-server: oleg324-server: LNET unconfigure error 22: Invalid argument modules unloaded. ip link add 'test1pl' type veth peer name test1pg netns test_ns ip link set 'test1pl' up ip netns exec test_ns ip addr add "10.1.2.3/31" dev test1pg ip netns exec test_ns ip -6 addr add "2001:db8:0:f101::1/64" dev test1pg ip netns exec test_ns ip link set test1pg up Waiting 10s for '0' Updated after 2s: want '0' got '0' /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ../lnet/lnet/lnet options: 'networks=""' 1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: ens2: mtu 9000 qdisc fq_codel state UP group default qlen 1000 link/ether 56:be:5a:26:b8:c8 brd ff:ff:ff:ff:ff:ff altname enp0s2 inet 192.168.203.24/24 brd 192.168.203.255 scope global dynamic ens2 valid_lft 58955sec preferred_lft 58955sec inet6 fe80::54be:5aff:fe26:b8c8/64 scope link valid_lft forever preferred_lft forever 19: test1pl@if2: mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 4a:d6:8d:d7:df:0e brd ff:ff:ff:ff:ff:ff link-netns test_ns inet6 fe80::48d6:8dff:fed7:df0e/64 scope link valid_lft forever preferred_lft forever Netid State Recv-Q Send-Q Local Address:Port Peer Address:PortProcess udp UNCONN 0 0 0.0.0.0:68 0.0.0.0:* users:(("dhclient",pid=1058,fd=6)) udp UNCONN 0 0 0.0.0.0:111 0.0.0.0:* users:(("rpcbind",pid=999,fd=5),("systemd",pid=1,fd=29)) udp UNCONN 0 0 0.0.0.0:54301 0.0.0.0:* users:(("rsyslogd",pid=1095,fd=5)) udp UNCONN 0 0 [::]:111 [::]:* users:(("rpcbind",pid=999,fd=7),("systemd",pid=1,fd=31)) tcp LISTEN 0 128 0.0.0.0:111 0.0.0.0:* users:(("rpcbind",pid=999,fd=4),("systemd",pid=1,fd=27)) tcp LISTEN 0 128 0.0.0.0:22 0.0.0.0:* users:(("sshd",pid=1025,fd=3)) tcp LISTEN 0 128 [::]:111 [::]:* users:(("rpcbind",pid=999,fd=6),("systemd",pid=1,fd=30)) tcp LISTEN 0 128 [::]:22 [::]:* users:(("sshd",pid=1025,fd=4)) ip netns exec test_ns ip a 1: lo: mtu 65536 qdisc noop state DOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2: test1pg@if19: mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 32:08:63:ad:7f:e1 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 10.1.2.3/31 scope global test1pg valid_lft forever preferred_lft forever inet6 2001:db8:0:f101::1/64 scope global valid_lft forever preferred_lft forever inet6 fe80::3008:63ff:fead:7fe1/64 scope link valid_lft forever preferred_lft forever ip netns exec test_ns ss -ltunp Netid State Recv-Q Send-Q Local Address:Port Peer Address:PortProcess lnet 1495040 1 ksocklnd libcfs 225280 2 lnet,ksocklnd ip netns exec test_ns /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 PASS 200 (20s) == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 12:10:46 (1788970246) Cleaning up LNet Writer error: failed to resolve Netlink family id unloading modules via unload_modules_local on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing unload_modules_local oleg324-server: oleg324-server: LNET unconfigure error 22: Invalid argument modules unloaded. ip link add 'test1pl' type veth peer name test1pg netns test_ns ip link set 'test1pl' up ip netns exec test_ns ip addr add "10.1.2.3/31" dev test1pg ip netns exec test_ns ip -6 addr add "2001:db8:0:f101::1/64" dev test1pg ip netns exec test_ns ip link set test1pg up Waiting 10s for '0' Updated after 2s: want '0' got '0' /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ../lnet/lnet/lnet options: 'networks=tcp(test1pg)' ip netns exec test_ns /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 PASS 201 (18s) == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 12:11:04 (1788970264) SKIP: sanity-lnet test_202 kernel ip2nets deprecated in 2.17.53 SKIP 202 (3s) == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 12:11:08 (1788970268) Cleaning up LNet Writer error: failed to resolve Netlink family id unloading modules via unload_modules_local on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing unload_modules_local oleg324-server: oleg324-server: LNET unconfigure error 22: Invalid argument modules unloaded. ip link add 'test1pl' type veth peer name test1pg netns test_ns ip link set 'test1pl' up ip netns exec test_ns ip addr add "10.1.2.3/31" dev test1pg ip netns exec test_ns ip -6 addr add "2001:db8:0:f101::1/64" dev test1pg ip netns exec test_ns ip link set test1pg up Waiting 10s for '0' Updated after 2s: want '0' got '0' /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure ip netns exec test_ns /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp0 --if test1pg pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 PASS 203 (21s) == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 12:11:28 (1788970288) Cleaning up LNet Writer error: failed to resolve Netlink family id unloading modules via unload_modules_local on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing unload_modules_local oleg324-server: oleg324-server: LNET unconfigure error 22: Invalid argument modules unloaded. /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a oleg324-server: Writer error: failed to resolve Netlink family id oleg324-server: opening /dev/lnet failed: No such file or directory oleg324-server: hint: the kernel modules may not be loaded oleg324-server: IOC_LIBCFS_GET_NI error 2: No such file or directory pdsh@oleg324-client: oleg324-server: ssh exited with exit code 1 oleg324-server: oleg324-server.virtnet: executing load_lnet oleg324-server: Loading modules from /home/green/git/lustre-release/lustre oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg324-server: detected 4 online CPUs by sysfs oleg324-server: MODOPTS_LIBCFS= oleg324-server: Force libcfs to create 2 CPU partitions /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.124@tcp discover: - primary nid: 192.168.203.124@tcp Multi-Rail: true peer_ni: - nid: 192.168.203.124@tcp net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.203.24@tcp status: up interfaces: 0: ens2 - primary nid: 192.168.203.124@tcp - nid: 192.168.203.124@tcp health stats: health value: 1000 debug=+net Simulate local_interrupt /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp -m GET -e local_interrupt -r 1 /home/green/git/lustre-release/lustre/utils/lctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp -m GET -e local_interrupt -r 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.124@tcp manage: - ping: errno: -5 descr: ! 'failed to ping 192.168.203.124@tcp: Input/output error' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a Check that no resends took place Check that local NI health has been changed Simulate local_dropped /home/green/git/lustre-release/lustre/utils/lctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp -m GET -e local_dropped -r 1 /home/green/git/lustre-release/lustre/utils/lctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp -m GET -e local_dropped -r 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.124@tcp manage: - ping: errno: -5 descr: ! 'failed to ping 192.168.203.124@tcp: Input/output error' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a Check that no resends took place Check that local NI health has been changed Simulate local_aborted /home/green/git/lustre-release/lustre/utils/lctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp -m GET -e local_aborted -r 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp -m GET -e local_aborted -r 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.124@tcp manage: - ping: errno: -5 descr: ! 'failed to ping 192.168.203.124@tcp: Input/output error' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a Check that no resends took place Check that local NI health has been changed Simulate local_no_route /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp -m GET -e local_no_route -r 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp -m GET -e local_no_route -r 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.124@tcp manage: - ping: errno: -5 descr: ! 'failed to ping 192.168.203.124@tcp: Input/output error' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a Check that no resends took place Check that local NI health has been changed Simulate local_timeout /home/green/git/lustre-release/lustre/utils/lctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp -m GET -e local_timeout -r 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp -m GET -e local_timeout -r 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.124@tcp manage: - ping: errno: -5 descr: ! 'failed to ping 192.168.203.124@tcp: Input/output error' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a Check that no resends took place Check that local NI health has been changed Simulate local_error /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp -m GET -e local_error -r 1 /home/green/git/lustre-release/lustre/utils/lctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp -m GET -e local_error -r 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.124@tcp manage: - ping: errno: -5 descr: ! 'failed to ping 192.168.203.124@tcp: Input/output error' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a Check that no resends took place Check that local NI health has been changed Writer error: failed to resolve Netlink family id unloading modules via unload_modules_local on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing unload_modules_local oleg324-server: modules unloaded. oleg324-server: oleg324-server.virtnet: executing unload_modules_local oleg324-server: oleg324-server: LNET unconfigure error 22: Invalid argument pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-client: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-client: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 PASS 204 (46s) == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 12:12:14 (1788970334) Cleaning up LNet LNET unconfigure error 22: Invalid argument unloading modules via unload_modules_local on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing unload_modules_local oleg324-server: oleg324-server: LNET unconfigure error 22: Invalid argument modules unloaded. /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a oleg324-server: Writer error: failed to resolve Netlink family id oleg324-server: opening /dev/lnet failed: No such file or directory oleg324-server: hint: the kernel modules may not be loaded oleg324-server: IOC_LIBCFS_GET_NI error 2: No such file or directory pdsh@oleg324-client: oleg324-server: ssh exited with exit code 1 oleg324-server: oleg324-server.virtnet: executing load_lnet oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg324-server: Loading modules from /home/green/git/lustre-release/lustre oleg324-server: detected 4 online CPUs by sysfs oleg324-server: MODOPTS_LIBCFS= oleg324-server: Force libcfs to create 2 CPU partitions /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.124@tcp discover: - primary nid: 192.168.203.124@tcp Multi-Rail: true peer_ni: - nid: 192.168.203.124@tcp oleg324-server: oleg324-server.virtnet: executing lnet_if_list oleg324-server: oleg324-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.24 default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.24 net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.203.24@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.203.24@tcp1 status: up interfaces: 0: ens2 - primary nid: 192.168.203.124@tcp - nid: 192.168.203.124@tcp health stats: health value: 1000 - nid: 192.168.203.124@tcp1 health stats: health value: 1000 debug=+net Simulate local_interrupt /home/green/git/lustre-release/lustre/utils/lctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp -m GET -e local_interrupt -r 1 /home/green/git/lustre-release/lustre/utils/lctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp1 -m GET -e local_interrupt -r 1 /home/green/git/lustre-release/lustre/utils/lctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp -m GET -e local_interrupt -r 1 /home/green/git/lustre-release/lustre/utils/lctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp1 -m GET -e local_interrupt -r 1 /home/green/git/lustre-release/lustre/utils/lctl net_drop add -s 192.168.203.24@tcp1 -d 192.168.203.124@tcp -m GET -e local_interrupt -r 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp1 -d 192.168.203.124@tcp1 -m GET -e local_interrupt -r 1 /home/green/git/lustre-release/lustre/utils/lctl net_drop add -s 192.168.203.24@tcp1 -d 192.168.203.24@tcp -m GET -e local_interrupt -r 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp1 -d 192.168.203.24@tcp1 -m GET -e local_interrupt -r 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.124@tcp manage: - ping: errno: -5 descr: ! 'failed to ping 192.168.203.124@tcp: Input/output error' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all /home/green/git/lustre-release/lustre/utils/lctl net_drop del -a Check that 2 resends took place Check for 1 failed resend Check for 0 successful resends Check that local NI health has been changed Simulate local_dropped /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp -m GET -e local_dropped -r 1 /home/green/git/lustre-release/lustre/utils/lctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp1 -m GET -e local_dropped -r 1 /home/green/git/lustre-release/lustre/utils/lctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp -m GET -e local_dropped -r 1 /home/green/git/lustre-release/lustre/utils/lctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp1 -m GET -e local_dropped -r 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp1 -d 192.168.203.124@tcp -m GET -e local_dropped -r 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp1 -d 192.168.203.124@tcp1 -m GET -e local_dropped -r 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp1 -d 192.168.203.24@tcp -m GET -e local_dropped -r 1 /home/green/git/lustre-release/lustre/utils/lctl net_drop add -s 192.168.203.24@tcp1 -d 192.168.203.24@tcp1 -m GET -e local_dropped -r 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.124@tcp manage: - ping: errno: -5 descr: ! 'failed to ping 192.168.203.124@tcp: Input/output error' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all /home/green/git/lustre-release/lustre/utils/lctl net_drop del -a Check that 2 resends took place Check for 1 failed resend Check for 0 successful resends Check that local NI health has been changed Simulate local_aborted /home/green/git/lustre-release/lustre/utils/lctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp -m GET -e local_aborted -r 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp1 -m GET -e local_aborted -r 1 /home/green/git/lustre-release/lustre/utils/lctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp -m GET -e local_aborted -r 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp1 -m GET -e local_aborted -r 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp1 -d 192.168.203.124@tcp -m GET -e local_aborted -r 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp1 -d 192.168.203.124@tcp1 -m GET -e local_aborted -r 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp1 -d 192.168.203.24@tcp -m GET -e local_aborted -r 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp1 -d 192.168.203.24@tcp1 -m GET -e local_aborted -r 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.124@tcp manage: - ping: errno: -5 descr: ! 'failed to ping 192.168.203.124@tcp: Input/output error' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all /home/green/git/lustre-release/lustre/utils/lctl net_drop del -a Check that 2 resends took place Check for 1 failed resend Check for 0 successful resends Check that local NI health has been changed Simulate local_no_route /home/green/git/lustre-release/lustre/utils/lctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp -m GET -e local_no_route -r 1 /home/green/git/lustre-release/lustre/utils/lctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp1 -m GET -e local_no_route -r 1 /home/green/git/lustre-release/lustre/utils/lctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp -m GET -e local_no_route -r 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp1 -m GET -e local_no_route -r 1 /home/green/git/lustre-release/lustre/utils/lctl net_drop add -s 192.168.203.24@tcp1 -d 192.168.203.124@tcp -m GET -e local_no_route -r 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp1 -d 192.168.203.124@tcp1 -m GET -e local_no_route -r 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp1 -d 192.168.203.24@tcp -m GET -e local_no_route -r 1 /home/green/git/lustre-release/lustre/utils/lctl net_drop add -s 192.168.203.24@tcp1 -d 192.168.203.24@tcp1 -m GET -e local_no_route -r 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.124@tcp manage: - ping: errno: -5 descr: ! 'failed to ping 192.168.203.124@tcp: Input/output error' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a Check that 2 resends took place Check for 1 failed resend Check for 0 successful resends Check that local NI health has been changed Simulate local_timeout /home/green/git/lustre-release/lustre/utils/lctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp -m GET -e local_timeout -r 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp1 -m GET -e local_timeout -r 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp -m GET -e local_timeout -r 1 /home/green/git/lustre-release/lustre/utils/lctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp1 -m GET -e local_timeout -r 1 /home/green/git/lustre-release/lustre/utils/lctl net_drop add -s 192.168.203.24@tcp1 -d 192.168.203.124@tcp -m GET -e local_timeout -r 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp1 -d 192.168.203.124@tcp1 -m GET -e local_timeout -r 1 /home/green/git/lustre-release/lustre/utils/lctl net_drop add -s 192.168.203.24@tcp1 -d 192.168.203.24@tcp -m GET -e local_timeout -r 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp1 -d 192.168.203.24@tcp1 -m GET -e local_timeout -r 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.124@tcp manage: - ping: errno: -5 descr: ! 'failed to ping 192.168.203.124@tcp: Input/output error' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a Check that 2 resends took place Check for 1 failed resend Check for 0 successful resends Check that local NI health has been changed Simulate local_error /home/green/git/lustre-release/lustre/utils/lctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp -m GET -e local_error -r 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp1 -m GET -e local_error -r 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp -m GET -e local_error -r 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp1 -m GET -e local_error -r 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp1 -d 192.168.203.124@tcp -m GET -e local_error -r 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp1 -d 192.168.203.124@tcp1 -m GET -e local_error -r 1 /home/green/git/lustre-release/lustre/utils/lctl net_drop add -s 192.168.203.24@tcp1 -d 192.168.203.24@tcp -m GET -e local_error -r 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp1 -d 192.168.203.24@tcp1 -m GET -e local_error -r 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.124@tcp manage: - ping: errno: -5 descr: ! 'failed to ping 192.168.203.124@tcp: Input/output error' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a Check that no resends took place Check that local NI health has been changed oleg324-server: oleg324-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp1 --if ens2 Writer error: failed to resolve Netlink family id unloading modules via unload_modules_local on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing unload_modules_local oleg324-server: modules unloaded. oleg324-server: oleg324-server.virtnet: executing unload_modules_local oleg324-server: oleg324-server: LNET unconfigure error 22: Invalid argument pdsh@oleg324-client: oleg324-client: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-client: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 PASS 205 (63s) == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 12:13:17 (1788970397) Cleaning up LNet LNET unconfigure error 22: Invalid argument unloading modules via unload_modules_local on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing unload_modules_local oleg324-server: oleg324-server: LNET unconfigure error 22: Invalid argument modules unloaded. /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a oleg324-server: Writer error: failed to resolve Netlink family id oleg324-server: opening /dev/lnet failed: No such file or directory oleg324-server: hint: the kernel modules may not be loaded oleg324-server: IOC_LIBCFS_GET_NI error 2: No such file or directory pdsh@oleg324-client: oleg324-server: ssh exited with exit code 1 oleg324-server: oleg324-server.virtnet: executing load_lnet oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg324-server: Loading modules from /home/green/git/lustre-release/lustre oleg324-server: detected 4 online CPUs by sysfs oleg324-server: MODOPTS_LIBCFS= oleg324-server: Force libcfs to create 2 CPU partitions /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.124@tcp discover: - primary nid: 192.168.203.124@tcp Multi-Rail: true peer_ni: - nid: 192.168.203.124@tcp net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.203.24@tcp status: up interfaces: 0: ens2 - primary nid: 192.168.203.124@tcp - nid: 192.168.203.124@tcp health stats: health value: 1000 debug=+net Simulate remote_dropped /home/green/git/lustre-release/lustre/utils/lctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp -m GET -e remote_dropped -r 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp -m GET -e remote_dropped -r 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.124@tcp manage: - ping: errno: -5 descr: ! 'failed to ping 192.168.203.124@tcp: Input/output error' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all /home/green/git/lustre-release/lustre/utils/lctl net_drop del -a Check that no resends took place Check that local NI health is unchanged Check that remote NI health is unchanged Simulate remote_error /home/green/git/lustre-release/lustre/utils/lctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp -m GET -e remote_error -r 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp -m GET -e remote_error -r 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.124@tcp manage: - ping: errno: -5 descr: ! 'failed to ping 192.168.203.124@tcp: Input/output error' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a Check that no resends took place Check that local NI health is unchanged Check that remote NI health is unchanged Simulate remote_timeout /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp -m GET -e remote_timeout -r 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp -m GET -e remote_timeout -r 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.124@tcp manage: - ping: errno: -5 descr: ! 'failed to ping 192.168.203.124@tcp: Input/output error' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all /home/green/git/lustre-release/lustre/utils/lctl net_drop del -a Check that no resends took place Check that local NI health is unchanged Check that remote NI health is unchanged Writer error: failed to resolve Netlink family id unloading modules via unload_modules_local on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing unload_modules_local oleg324-server: modules unloaded. oleg324-server: oleg324-server.virtnet: executing unload_modules_local oleg324-server: oleg324-server: LNET unconfigure error 22: Invalid argument pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-client: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-client: ssh exited with exit code 2 PASS 206 (37s) == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 12:13:54 (1788970434) Cleaning up LNet LNET unconfigure error 22: Invalid argument unloading modules via unload_modules_local on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing unload_modules_local oleg324-server: oleg324-server: LNET unconfigure error 22: Invalid argument modules unloaded. /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a oleg324-server: Writer error: failed to resolve Netlink family id oleg324-server: opening /dev/lnet failed: No such file or directory oleg324-server: hint: the kernel modules may not be loaded oleg324-server: IOC_LIBCFS_GET_NI error 2: No such file or directory pdsh@oleg324-client: oleg324-server: ssh exited with exit code 1 oleg324-server: oleg324-server.virtnet: executing load_lnet oleg324-server: Loading modules from /home/green/git/lustre-release/lustre oleg324-server: detected 4 online CPUs by sysfs oleg324-server: MODOPTS_LIBCFS= oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg324-server: Force libcfs to create 2 CPU partitions /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.124@tcp discover: - primary nid: 192.168.203.124@tcp Multi-Rail: true peer_ni: - nid: 192.168.203.124@tcp oleg324-server: oleg324-server.virtnet: executing lnet_if_list oleg324-server: oleg324-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 oleg324-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.124 oleg324-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.124 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.24 default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.24 net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.203.24@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.203.24@tcp1 status: up interfaces: 0: ens2 - primary nid: 192.168.203.124@tcp - nid: 192.168.203.124@tcp health stats: health value: 1000 - nid: 192.168.203.124@tcp1 health stats: health value: 1000 debug=+net Simulate remote_dropped /home/green/git/lustre-release/lustre/utils/lctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp -m GET -e remote_dropped -r 1 /home/green/git/lustre-release/lustre/utils/lctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp1 -m GET -e remote_dropped -r 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp -m GET -e remote_dropped -r 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp1 -m GET -e remote_dropped -r 1 /home/green/git/lustre-release/lustre/utils/lctl net_drop add -s 192.168.203.24@tcp1 -d 192.168.203.124@tcp -m GET -e remote_dropped -r 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp1 -d 192.168.203.124@tcp1 -m GET -e remote_dropped -r 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp1 -d 192.168.203.24@tcp -m GET -e remote_dropped -r 1 /home/green/git/lustre-release/lustre/utils/lctl net_drop add -s 192.168.203.24@tcp1 -d 192.168.203.24@tcp1 -m GET -e remote_dropped -r 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.124@tcp manage: - ping: errno: -5 descr: ! 'failed to ping 192.168.203.124@tcp: Input/output error' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a Check that 2 resends took place Check for 1 failed resend Check for 0 successful resends Check that local NI health is unchanged Check that remote NI health has been changed /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all Simulate remote_error /home/green/git/lustre-release/lustre/utils/lctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp -m GET -e remote_error -r 1 /home/green/git/lustre-release/lustre/utils/lctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp1 -m GET -e remote_error -r 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp -m GET -e remote_error -r 1 /home/green/git/lustre-release/lustre/utils/lctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp1 -m GET -e remote_error -r 1 /home/green/git/lustre-release/lustre/utils/lctl net_drop add -s 192.168.203.24@tcp1 -d 192.168.203.124@tcp -m GET -e remote_error -r 1 /home/green/git/lustre-release/lustre/utils/lctl net_drop add -s 192.168.203.24@tcp1 -d 192.168.203.124@tcp1 -m GET -e remote_error -r 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp1 -d 192.168.203.24@tcp -m GET -e remote_error -r 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp1 -d 192.168.203.24@tcp1 -m GET -e remote_error -r 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.124@tcp manage: - ping: errno: -5 descr: ! 'failed to ping 192.168.203.124@tcp: Input/output error' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all /home/green/git/lustre-release/lustre/utils/lctl net_drop del -a Check that no resends took place Check that local NI health is unchanged Check that remote NI health has been changed /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all Simulate remote_timeout /home/green/git/lustre-release/lustre/utils/lctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp -m GET -e remote_timeout -r 1 /home/green/git/lustre-release/lustre/utils/lctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp1 -m GET -e remote_timeout -r 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp -m GET -e remote_timeout -r 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp1 -m GET -e remote_timeout -r 1 /home/green/git/lustre-release/lustre/utils/lctl net_drop add -s 192.168.203.24@tcp1 -d 192.168.203.124@tcp -m GET -e remote_timeout -r 1 /home/green/git/lustre-release/lustre/utils/lctl net_drop add -s 192.168.203.24@tcp1 -d 192.168.203.124@tcp1 -m GET -e remote_timeout -r 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp1 -d 192.168.203.24@tcp -m GET -e remote_timeout -r 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp1 -d 192.168.203.24@tcp1 -m GET -e remote_timeout -r 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.124@tcp manage: - ping: errno: -5 descr: ! 'failed to ping 192.168.203.124@tcp: Input/output error' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all /home/green/git/lustre-release/lustre/utils/lctl net_drop del -a Check that no resends took place Check that local NI health is unchanged Check that remote NI health has been changed /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all oleg324-server: oleg324-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp1 --if ens2 Writer error: failed to resolve Netlink family id unloading modules via unload_modules_local on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing unload_modules_local oleg324-server: modules unloaded. oleg324-server: oleg324-server.virtnet: executing unload_modules_local oleg324-server: oleg324-server: LNET unconfigure error 22: Invalid argument pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-client: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-client: ssh exited with exit code 2 PASS 207 (55s) == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 12:14:49 (1788970489) SKIP: sanity-lnet test_208 kernel ip2nets deprecated in 2.17.53 SKIP 208 (3s) == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 12:14:52 (1788970492) Cleaning up LNet LNET unconfigure error 22: Invalid argument unloading modules via unload_modules_local on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing unload_modules_local oleg324-server: oleg324-server: LNET unconfigure error 22: Invalid argument modules unloaded. /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a oleg324-server: Writer error: failed to resolve Netlink family id oleg324-server: opening /dev/lnet failed: No such file or directory oleg324-server: hint: the kernel modules may not be loaded oleg324-server: IOC_LIBCFS_GET_NI error 2: No such file or directory pdsh@oleg324-client: oleg324-server: ssh exited with exit code 1 oleg324-server: oleg324-server.virtnet: executing load_lnet oleg324-server: Loading modules from /home/green/git/lustre-release/lustre oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg324-server: detected 4 online CPUs by sysfs oleg324-server: MODOPTS_LIBCFS= oleg324-server: Force libcfs to create 2 CPU partitions /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.124@tcp discover: - primary nid: 192.168.203.124@tcp Multi-Rail: true peer_ni: - nid: 192.168.203.124@tcp net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.203.24@tcp status: up interfaces: 0: ens2 - primary nid: 192.168.203.124@tcp - nid: 192.168.203.124@tcp health stats: health value: 1000 debug=+net Simulate network_timeout w/SR config /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp -m GET -e network_timeout -r 1 /home/green/git/lustre-release/lustre/utils/lctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp -m GET -e network_timeout -r 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.124@tcp manage: - discover: errno: -5 descr: ! 'failed to discover 192.168.203.124@tcp: Input/output error' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all Check that no resends took place Check that local NI health has been changed Check that remote NI health is unchanged Writer error: failed to resolve Netlink family id unloading modules via unload_modules_local on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing unload_modules_local oleg324-server: modules unloaded. oleg324-server: oleg324-server.virtnet: executing unload_modules_local oleg324-server: oleg324-server: LNET unconfigure error 22: Invalid argument Cleaning up LNet LNET unconfigure error 22: Invalid argument unloading modules via unload_modules_local on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing unload_modules_local oleg324-server: oleg324-server: LNET unconfigure error 22: Invalid argument modules unloaded. /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a oleg324-server: Writer error: failed to resolve Netlink family id oleg324-server: opening /dev/lnet failed: No such file or directory oleg324-server: hint: the kernel modules may not be loaded oleg324-server: IOC_LIBCFS_GET_NI error 2: No such file or directory pdsh@oleg324-client: oleg324-server: ssh exited with exit code 1 oleg324-server: oleg324-server.virtnet: executing load_lnet oleg324-server: Loading modules from /home/green/git/lustre-release/lustre oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg324-server: detected 4 online CPUs by sysfs oleg324-server: MODOPTS_LIBCFS= oleg324-server: Force libcfs to create 2 CPU partitions /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.124@tcp discover: - primary nid: 192.168.203.124@tcp Multi-Rail: true peer_ni: - nid: 192.168.203.124@tcp oleg324-server: oleg324-server.virtnet: executing lnet_if_list oleg324-server: oleg324-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 oleg324-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.124 oleg324-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.124 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.24 default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.24 net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.203.24@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.203.24@tcp1 status: up interfaces: 0: ens2 - primary nid: 192.168.203.124@tcp - nid: 192.168.203.124@tcp health stats: health value: 1000 - nid: 192.168.203.124@tcp1 health stats: health value: 1000 debug=+net Simulate network_timeout w/MR config /home/green/git/lustre-release/lustre/utils/lctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp -m GET -e network_timeout -r 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp1 -m GET -e network_timeout -r 1 /home/green/git/lustre-release/lustre/utils/lctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp -m GET -e network_timeout -r 1 /home/green/git/lustre-release/lustre/utils/lctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp1 -m GET -e network_timeout -r 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp1 -d 192.168.203.124@tcp -m GET -e network_timeout -r 1 /home/green/git/lustre-release/lustre/utils/lctl net_drop add -s 192.168.203.24@tcp1 -d 192.168.203.124@tcp1 -m GET -e network_timeout -r 1 /home/green/git/lustre-release/lustre/utils/lctl net_drop add -s 192.168.203.24@tcp1 -d 192.168.203.24@tcp -m GET -e network_timeout -r 1 /home/green/git/lustre-release/lustre/utils/lctl net_drop add -s 192.168.203.24@tcp1 -d 192.168.203.24@tcp1 -m GET -e network_timeout -r 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.124@tcp manage: - discover: errno: -5 descr: ! 'failed to discover 192.168.203.124@tcp: Input/output error' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all Check that no resends took place Check that local NI health has been changed Check that remote NI health has been changed oleg324-server: oleg324-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp1 --if ens2 Writer error: failed to resolve Netlink family id unloading modules via unload_modules_local on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing unload_modules_local oleg324-server: modules unloaded. oleg324-server: oleg324-server.virtnet: executing unload_modules_local oleg324-server: oleg324-server: LNET unconfigure error 22: Invalid argument pdsh@oleg324-client: oleg324-client: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-client: ssh exited with exit code 2 PASS 209 (78s) == sanity-lnet test 210: Local NI recovery checks ======== 12:16:10 (1788970570) Loading LNet and configuring DLC /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.24 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.24 default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.24 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.24@tcp discover: - primary nid: 192.168.203.24@tcp Multi-Rail: true peer_ni: - nid: 192.168.203.24@tcp - nid: 192.168.203.24@tcp1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 debug=+net /home/green/git/lustre-release/lustre/utils/lctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error /home/green/git/lustre-release/lustre/utils/lctl net_drop add -s *@tcp -d *@tcp -r 1 /home/green/git/lustre-release/lustre/utils/lctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.24@tcp Check ping counts: Waiting 10s for '2' Updated after 4s: want '2' got '2' ping_count: 0 - nid: 192.168.203.24@tcp ping_count: 2 - nid: 192.168.203.24@tcp1 ping_count: 0 -l recovery queue should have 192.168.203.24@tcp /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l Local NI recovery: nid-0: 192.168.203.24@tcp Check ping counts: Waiting 10s for '3' Updated after 3s: want '3' got '3' ping_count: 0 - nid: 192.168.203.24@tcp ping_count: 3 - nid: 192.168.203.24@tcp1 ping_count: 0 -l recovery queue should have 192.168.203.24@tcp /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l Local NI recovery: nid-0: 192.168.203.24@tcp /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.24 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.24 default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.24 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.24@tcp discover: - primary nid: 192.168.203.24@tcp Multi-Rail: true peer_ni: - nid: 192.168.203.24@tcp - nid: 192.168.203.24@tcp1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 debug=+net /home/green/git/lustre-release/lustre/utils/lctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e local_error /home/green/git/lustre-release/lustre/utils/lctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e local_error /home/green/git/lustre-release/lustre/utils/lctl net_drop add -s *@tcp -d *@tcp -r 1 /home/green/git/lustre-release/lustre/utils/lctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.24@tcp Check ping counts: Waiting 10s for '2' Updated after 3s: want '2' got '2' ping_count: 0 - nid: 192.168.203.24@tcp ping_count: 2 - nid: 192.168.203.24@tcp1 ping_count: 0 -l recovery queue should have 192.168.203.24@tcp /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l Local NI recovery: nid-0: 192.168.203.24@tcp Check ping counts: Waiting 10s for '4' Updated after 9s: want '4' got '4' ping_count: 0 - nid: 192.168.203.24@tcp ping_count: 4 - nid: 192.168.203.24@tcp1 ping_count: 0 -l recovery queue should have 192.168.203.24@tcp /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l Local NI recovery: nid-0: 192.168.203.24@tcp /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 PASS 210 (32s) == sanity-lnet test 211: Remote NI recovery checks ======= 12:16:42 (1788970602) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.24 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.24 default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.24 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.24@tcp discover: - primary nid: 192.168.203.24@tcp Multi-Rail: true peer_ni: - nid: 192.168.203.24@tcp - nid: 192.168.203.24@tcp1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error /home/green/git/lustre-release/lustre/utils/lctl net_drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error /home/green/git/lustre-release/lustre/utils/lctl net_drop add -s *@tcp -d *@tcp -r 1 /home/green/git/lustre-release/lustre/utils/lctl net_drop add -s *@tcp1 -d *@tcp1 -r 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.24@tcp --health 0 -p recovery queue should have 192.168.203.24@tcp /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p Peer NI recovery: nid-0: 192.168.203.24@tcp -p recovery queue should be empty Waiting 20s for '0' Updated after 16s: want '0' got '0' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p Check ping counts: - nid: 192.168.203.24@tcp ping_count: 0 - nid: 192.168.203.24@tcp1 ping_count: 0 /home/green/git/lustre-release/lustre/utils/lctl net_drop del -a /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.24@tcp --health 500 -p recovery queue should have 192.168.203.24@tcp /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p Peer NI recovery: nid-0: 192.168.203.24@tcp -p recovery queue should be empty Waiting 20s for '0' Updated after 7s: want '0' got '0' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.24 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.24 default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.24 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.24@tcp discover: - primary nid: 192.168.203.24@tcp Multi-Rail: true peer_ni: - nid: 192.168.203.24@tcp - nid: 192.168.203.24@tcp1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 /home/green/git/lustre-release/lustre/utils/lctl net_drop add -s *@tcp -d *@tcp -m GET -r 1 -e remote_error /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -m GET -r 1 -e remote_error /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp1 -d *@tcp1 -r 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.203.24@tcp --health 0 Check ping counts: Waiting 4s for '1' - nid: 192.168.203.24@tcp ping_count: 1 - nid: 192.168.203.24@tcp1 ping_count: 0 -p recovery queue should have 192.168.203.24@tcp /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p Peer NI recovery: nid-0: 192.168.203.24@tcp Check ping counts: Waiting 14s for '4' Updated after 9s: want '4' got '4' - nid: 192.168.203.24@tcp ping_count: 4 - nid: 192.168.203.24@tcp1 ping_count: 0 -p recovery queue should have 192.168.203.24@tcp /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p Peer NI recovery: nid-0: 192.168.203.24@tcp /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 PASS 211 (44s) == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 12:17:27 (1788970647) Cleaning up LNet Writer error: failed to resolve Netlink family id unloading modules via unload_modules_local on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing unload_modules_local oleg324-server: oleg324-server: LNET unconfigure error 22: Invalid argument modules unloaded. /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a oleg324-server: Writer error: failed to resolve Netlink family id oleg324-server: opening /dev/lnet failed: No such file or directory oleg324-server: hint: the kernel modules may not be loaded oleg324-server: IOC_LIBCFS_GET_NI error 2: No such file or directory pdsh@oleg324-client: oleg324-server: ssh exited with exit code 1 oleg324-server: oleg324-server.virtnet: executing load_lnet oleg324-server: Loading modules from /home/green/git/lustre-release/lustre oleg324-server: detected 4 online CPUs by sysfs oleg324-server: MODOPTS_LIBCFS= oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg324-server: Force libcfs to create 2 CPU partitions Initial discovery /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.124@tcp discover: - primary nid: 192.168.203.124@tcp Multi-Rail: true peer_ni: - nid: 192.168.203.124@tcp discover: - primary nid: 192.168.203.24@tcp Multi-Rail: true peer_ni: - nid: 192.168.203.24@tcp Fail local discover ping to set LNET_PEER_REDISCOVER flag /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.124@tcp manage: - discover: errno: -5 descr: ! 'failed to discover 192.168.203.124@tcp: Input/output error' /home/green/git/lustre-release/lustre/utils/lctl net_drop del -a /home/green/git/lustre-release/lustre/utils/lctl net_delay add -s *@tcp -d 192.168.203.124@tcp -r 1 -m GET -l 3 /home/green/git/lustre-release/lustre/utils/lctl net_drop add -s *@tcp -d 192.168.203.124@tcp -r 1 -m GET -e local_error /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault delay add -s *@tcp -d 192.168.203.124@tcp -r 1 -m PUT -l 6 Force oleg324-client.virtnet to discover 192.168.203.124@tcp (in background) Force oleg324-server to discover 192.168.203.24@tcp /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.203.124@tcp discover: - primary nid: 192.168.203.24@tcp Multi-Rail: true peer_ni: - nid: 192.168.203.24@tcp Wait for 72108 manage: - discover: errno: -5 descr: ! 'failed to discover 192.168.203.124@tcp: Input/output error' Finished wait on 72108 /home/green/git/lustre-release/lustre/utils/lctl net_delay del -a /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a Writer error: failed to resolve Netlink family id unloading modules via unload_modules_local on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing unload_modules_local oleg324-server: modules unloaded. oleg324-server: oleg324-server.virtnet: executing unload_modules_local oleg324-server: oleg324-server: LNET unconfigure error 22: Invalid argument pdsh@oleg324-client: oleg324-client: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-client: ssh exited with exit code 2 PASS 212 (40s) == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 12:18:07 (1788970687) Cleaning up LNet LNET unconfigure error 22: Invalid argument unloading modules via unload_modules_local on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing unload_modules_local oleg324-server: oleg324-server: LNET unconfigure error 22: Invalid argument modules unloaded. ip link add 'test1pl' type veth peer name test1pg ip link set 'test1pl' up ip addr add "10.1.2.3/31" dev test1pg ip -6 addr add "2001:db8:0:f101::1/64" dev test1pg ip link set test1pg up Waiting 10s for '0' Updated after 2s: want '0' got '0' Loading LNet and configuring DLC /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.24 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.24 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 PASS 213 (16s) == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 12:18:23 (1788970703) Cleaning up LNet Writer error: failed to resolve Netlink family id unloading modules via unload_modules_local on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing unload_modules_local oleg324-server: oleg324-server: LNET unconfigure error 22: Invalid argument modules unloaded. ip link add 'test1pl' type veth peer name test1pg RTNETLINK answers: File exists ip link set 'test1pl' up ip addr add "10.1.2.3/31" dev test1pg RTNETLINK answers: File exists ip -6 addr add "2001:db8:0:f101::1/64" dev test1pg RTNETLINK answers: File exists ip link set test1pg up Loading LNet and configuring DLC /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.24 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.24 10.1.2.2/31 dev test1pg proto kernel scope link src 10.1.2.3 NI 0@lo expect status "up" found "up" NI 192.168.203.24@tcp expect status "up" found "up" NI 10.1.2.3@tcp expect status "up" found "up" /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.203.24@tcp ping: - primary nid: 192.168.203.24@tcp Multi-Rail: false peer_ni: - nid: 192.168.203.24@tcp - nid: 10.1.2.3@tcp Set test1pg down ip link set dev test1pg down NI 0@lo expect status "up" found "up" NI 192.168.203.24@tcp expect status "up" found "up" NI 10.1.2.3@tcp expect status "down" found "down" pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 PASS 214 (16s) == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 12:18:39 (1788970719) Cleaning up LNet Writer error: failed to resolve Netlink family id unloading modules via unload_modules_local on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing unload_modules_local oleg324-server: oleg324-server: LNET unconfigure error 22: Invalid argument modules unloaded. Loading LNet and configuring DLC /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.24 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if ens2 default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.24 default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.24 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.203.24@tcp1 --nid 192.168.203.24@tcp2 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.24@tcp1 x 25 192.168.203.24@tcp1 pre send_count 0 post send_count 26 192.168.203.24@tcp1 pre recv_count 0 post recv_count 26 192.168.203.24@tcp2 pre send_count 0 post send_count 24 192.168.203.24@tcp2 pre recv_count 0 post recv_count 24 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.24@tcp1 192.168.203.24@tcp1 x 25 192.168.203.24@tcp1 pre send_count 26 post send_count 76 192.168.203.24@tcp1 pre recv_count 26 post recv_count 76 192.168.203.24@tcp2 pre send_count 24 post send_count 24 192.168.203.24@tcp2 pre recv_count 24 post recv_count 24 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.24@tcp2 x 50 192.168.203.24@tcp1 pre send_count 76 post send_count 126 192.168.203.24@tcp1 pre recv_count 76 post recv_count 126 192.168.203.24@tcp2 pre send_count 24 post send_count 74 192.168.203.24@tcp2 pre recv_count 24 post recv_count 74 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.24@tcp2 192.168.203.24@tcp2 x 50 192.168.203.24@tcp1 pre send_count 126 post send_count 126 192.168.203.24@tcp1 pre recv_count 126 post recv_count 126 192.168.203.24@tcp2 pre send_count 74 post send_count 174 192.168.203.24@tcp2 pre recv_count 74 post recv_count 174 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.24@tcp1 192.168.203.24@tcp2 manage: - ping: errno: -113 descr: ! 'failed to ping 192.168.203.24@tcp2: No route to host' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.24@tcp2 192.168.203.24@tcp1 manage: - ping: errno: -113 descr: ! 'failed to ping 192.168.203.24@tcp1: No route to host' pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 PASS 215 (20s) == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 12:18:59 (1788970739) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.24 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.24 default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.24 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.24@tcp discover: - primary nid: 192.168.203.24@tcp Multi-Rail: true peer_ni: - nid: 192.168.203.24@tcp - nid: 192.168.203.24@tcp1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.24@tcp 192.168.203.24@tcp ping: - primary nid: 192.168.203.24@tcp Multi-Rail: true peer_ni: - nid: 192.168.203.24@tcp - nid: 192.168.203.24@tcp1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.24@tcp1 192.168.203.24@tcp1 ping: - primary nid: 192.168.203.24@tcp Multi-Rail: true peer_ni: - nid: 192.168.203.24@tcp - nid: 192.168.203.24@tcp1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.24@tcp -d 192.168.203.24@tcp -e network_timeout /home/green/git/lustre-release/lustre/utils/lctl net_drop add -r 1 -s 192.168.203.24@tcp -d 192.168.203.24@tcp1 -e network_timeout /home/green/git/lustre-release/lustre/utils/lctl net_drop add -r 1 -s 192.168.203.24@tcp1 -d 192.168.203.24@tcp -e network_timeout /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -r 1 -s 192.168.203.24@tcp1 -d 192.168.203.24@tcp1 -e network_timeout /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.24@tcp manage: - ping: errno: -5 descr: ! 'failed to ping 192.168.203.24@tcp: Input/output error' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop del -a -p recovery queue should be empty /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p -l recovery queue should have 192.168.203.24@tcp /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l Local NI recovery: nid-0: 192.168.203.24@tcp pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 PASS 216 (10s) == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 12:19:09 (1788970749) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo discover: - primary nid: 0@lo Multi-Rail: true peer_ni: Writer error: failed to resolve Netlink family id unloading modules via unload_modules_local on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing unload_modules_local oleg324-server: oleg324-server: LNET unconfigure error 22: Invalid argument modules unloaded. pdsh@oleg324-client: oleg324-client: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-client: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 PASS 217 (19s) == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 12:19:28 (1788970768) Loading LNet and configuring DLC /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure SKIP: sanity-lnet test_218 Need two LNet interfaces SKIP 218 (4s) == sanity-lnet test 219: Consolidate peer entries ======== 12:19:33 (1788970773) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.24 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.24 default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.24 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.24@tcp ping: - primary nid: 192.168.203.24@tcp Multi-Rail: false peer_ni: - nid: 192.168.203.24@tcp - nid: 192.168.203.24@tcp1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.24@tcp1 ping: - primary nid: 192.168.203.24@tcp1 Multi-Rail: false peer_ni: - nid: 192.168.203.24@tcp - nid: 192.168.203.24@tcp1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.24@tcp1 discover: - primary nid: 192.168.203.24@tcp Multi-Rail: true peer_ni: - nid: 192.168.203.24@tcp - nid: 192.168.203.24@tcp1 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 PASS 219 (8s) == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 12:19:41 (1788970781) SKIP: sanity-lnet test_220 Need at least 2 remote nodes found "oleg324-server" SKIP 220 (3s) == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 12:19:45 (1788970785) SKIP: sanity-lnet test_221 Need at least 2 remote nodes found "oleg324-server" SKIP 221 (4s) == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 12:19:48 (1788970788) SKIP: sanity-lnet test_222 Need at least 2 remote nodes found "oleg324-server" SKIP 222 (4s) == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 12:19:52 (1788970792) SKIP: sanity-lnet test_223 Need at least 2 remote nodes found "oleg324-server" SKIP 223 (4s) == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 12:19:56 (1788970796) SKIP: sanity-lnet test_224 Need at least 2 remote nodes found "oleg324-server" SKIP 224 (4s) == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 12:20:00 (1788970800) SKIP: sanity-lnet test_225 Need at least 2 remote nodes found "oleg324-server" SKIP 225 (5s) == sanity-lnet test 226: test missing route for 1 of 2 routers ========================================================== 12:20:05 (1788970805) SKIP: sanity-lnet test_226 Need at least 3 remote nodes found "oleg324-server" SKIP 226 (4s) == sanity-lnet test 227: Check router peer health w/DD disabled ========================================================== 12:20:09 (1788970809) SKIP: sanity-lnet test_227 Need at least 3 remote nodes found "oleg324-server" SKIP 227 (4s) == sanity-lnet test 228: Routes should stay up when health is decremented ========================================================== 12:20:13 (1788970813) SKIP: sanity-lnet test_228 Need at least 3 remote nodes found "oleg324-server" SKIP 228 (4s) == sanity-lnet test 230: Test setting conns-per-peer ===== 12:20:17 (1788970817) Check valid values; Should succeed /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.24 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.24@tcp ping: - primary nid: 192.168.203.24@tcp Multi-Rail: false peer_ni: - nid: 192.168.203.24@tcp /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 5 default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.24 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.24@tcp ping: - primary nid: 192.168.203.24@tcp Multi-Rail: false peer_ni: - nid: 192.168.203.24@tcp Waiting 10s for '12' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.24 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.24@tcp ping: - primary nid: 192.168.203.24@tcp Multi-Rail: false peer_ni: - nid: 192.168.203.24@tcp /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 7 default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.24 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.24@tcp ping: - primary nid: 192.168.203.24@tcp Multi-Rail: false peer_ni: - nid: 192.168.203.24@tcp /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.24 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.24@tcp ping: - primary nid: 192.168.203.24@tcp Multi-Rail: false peer_ni: - nid: 192.168.203.24@tcp /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 9 default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.24 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.24@tcp ping: - primary nid: 192.168.203.24@tcp Multi-Rail: false peer_ni: - nid: 192.168.203.24@tcp /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.24 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.24@tcp ping: - primary nid: 192.168.203.24@tcp Multi-Rail: false peer_ni: - nid: 192.168.203.24@tcp /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 11 default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.24 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.24@tcp ping: - primary nid: 192.168.203.24@tcp Multi-Rail: false peer_ni: - nid: 192.168.203.24@tcp /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.24 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.24@tcp ping: - primary nid: 192.168.203.24@tcp Multi-Rail: false peer_ni: - nid: 192.168.203.24@tcp /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 13 default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.24 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.24@tcp ping: - primary nid: 192.168.203.24@tcp Multi-Rail: false peer_ni: - nid: 192.168.203.24@tcp /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.24 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.24@tcp ping: - primary nid: 192.168.203.24@tcp Multi-Rail: false peer_ni: - nid: 192.168.203.24@tcp /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 15 default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.24 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.24@tcp ping: - primary nid: 192.168.203.24@tcp Multi-Rail: false peer_ni: - nid: 192.168.203.24@tcp /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.24 Set > 127; Should fail /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 net: - set: errno: -1 descr: invalid conns-per-peer value (valid range 0-127) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.24 Set < 0; Should be ignored /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure Add > 127; Should fail and not configure NI /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 128 ni: - add: errno: -1 descr: invalid conns-per-peer value (valid range 0-127) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure Add overflow value; Should fail and not configure NI /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 1000000000000000000000000 ni: - add: errno: -1 descr: invalid conns-per-peer value (valid range 0-127) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure Add non-numeric value; Should fail and not configure NI /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer foo ni: - add: errno: -1 descr: invalid conns-per-peer value (valid range 0-127) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure Import > 127 conns-per-peer; Should fail and not configure NI /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 --conns-per-peer 8 default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.24 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --- add: - net: errno: -34 descr: ! "failed to parse lnd tunables" ... --- add: - import: errno: -22 descr: ! "Failed to configure NI on net 'tcp' rc = -22" ... pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 PASS 230 (36s) == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 12:20:53 (1788970853) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.24 Expected: net: - net type: tcp231 local NI(s): - interfaces: 0: ens2 tunables: peer_timeout: 180 peer_credits: 8 peer_buffer_credits: 0 credits: 256 lnd tunables: conns_per_peer: 1 tos: -1 CPT: "[0]" global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: net: - net type: tcp231 local NI(s): - interfaces: 0: ens2 tunables: peer_timeout: 180 peer_credits: 8 peer_buffer_credits: 0 credits: 256 lnd tunables: conns_per_peer: 1 tos: -1 CPT: "[0]" global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml Expected: net: - net type: tcp231 local NI(s): - interfaces: 0: ens2 tunables: peer_timeout: 180 peer_credits: 8 peer_buffer_credits: 0 credits: 256 lnd tunables: conns_per_peer: 1 tos: -1 CPT: "[0]" global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: net: - net type: tcp231 local NI(s): - interfaces: 0: ens2 tunables: peer_timeout: 180 peer_credits: 8 peer_buffer_credits: 0 credits: 256 lnd tunables: conns_per_peer: 1 tos: -1 CPT: "[0]" global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=-1 default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.24 Expected: net: - net type: tcp231 local NI(s): - interfaces: 0: ens2 tunables: peer_timeout: 180 peer_credits: 8 peer_buffer_credits: 0 credits: 256 lnd tunables: conns_per_peer: 1 tos: -1 CPT: "[0]" global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: net: - net type: tcp231 local NI(s): - interfaces: 0: ens2 tunables: peer_timeout: 180 peer_credits: 8 peer_buffer_credits: 0 credits: 256 lnd tunables: conns_per_peer: 1 tos: -1 CPT: "[0]" global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml Expected: net: - net type: tcp231 local NI(s): - interfaces: 0: ens2 tunables: peer_timeout: 0 peer_credits: 8 peer_buffer_credits: 0 credits: 256 lnd tunables: conns_per_peer: 1 tos: -1 CPT: "[0]" global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: net: - net type: tcp231 local NI(s): - interfaces: 0: ens2 tunables: peer_timeout: 0 peer_credits: 8 peer_buffer_credits: 0 credits: 256 lnd tunables: conns_per_peer: 1 tos: -1 CPT: "[0]" global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=0 default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.24 Expected: net: - net type: tcp231 local NI(s): - interfaces: 0: ens2 tunables: peer_timeout: 0 peer_credits: 8 peer_buffer_credits: 0 credits: 256 lnd tunables: conns_per_peer: 1 tos: -1 CPT: "[0]" global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: net: - net type: tcp231 local NI(s): - interfaces: 0: ens2 tunables: peer_timeout: 0 peer_credits: 8 peer_buffer_credits: 0 credits: 256 lnd tunables: conns_per_peer: 1 tos: -1 CPT: "[0]" global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml Expected: net: - net type: tcp231 local NI(s): - interfaces: 0: ens2 tunables: peer_timeout: 60 peer_credits: 8 peer_buffer_credits: 0 credits: 256 lnd tunables: conns_per_peer: 1 tos: -1 CPT: "[0]" global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: net: - net type: tcp231 local NI(s): - interfaces: 0: ens2 tunables: peer_timeout: 60 peer_credits: 8 peer_buffer_credits: 0 credits: 256 lnd tunables: conns_per_peer: 1 tos: -1 CPT: "[0]" global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=60 default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.24 Expected: net: - net type: tcp231 local NI(s): - interfaces: 0: ens2 tunables: peer_timeout: 60 peer_credits: 8 peer_buffer_credits: 0 credits: 256 lnd tunables: conns_per_peer: 1 tos: -1 CPT: "[0]" global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: net: - net type: tcp231 local NI(s): - interfaces: 0: ens2 tunables: peer_timeout: 60 peer_credits: 8 peer_buffer_credits: 0 credits: 256 lnd tunables: conns_per_peer: 1 tos: -1 CPT: "[0]" global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml Expected: net: - net type: tcp231 local NI(s): - interfaces: 0: ens2 tunables: peer_timeout: 180 peer_credits: 8 peer_buffer_credits: 0 credits: 256 lnd tunables: conns_per_peer: 1 tos: -1 CPT: "[0]" global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: net: - net type: tcp231 local NI(s): - interfaces: 0: ens2 tunables: peer_timeout: 180 peer_credits: 8 peer_buffer_credits: 0 credits: 256 lnd tunables: conns_per_peer: 1 tos: -1 CPT: "[0]" global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if ens2 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if ens2 --peer-timeout=180 default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.24 Expected: net: - net type: tcp231 local NI(s): - interfaces: 0: ens2 tunables: peer_timeout: 180 peer_credits: 8 peer_buffer_credits: 0 credits: 256 lnd tunables: conns_per_peer: 1 tos: -1 CPT: "[0]" global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: net: - net type: tcp231 local NI(s): - interfaces: 0: ens2 tunables: peer_timeout: 180 peer_credits: 8 peer_buffer_credits: 0 credits: 256 lnd tunables: conns_per_peer: 1 tos: -1 CPT: "[0]" global: numa_range: 0 max_interfaces: 4096 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 PASS 231 (16s) == sanity-lnet test 232: Test setting ToS value ========== 12:21:09 (1788970869) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.24 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos -1 default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.24 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 104 default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.24 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 106 default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.24 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp232 --if ens2 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp232 --if ens2 --tos 300 ni: - add: errno: -1 descr: Invalid ToS argument pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 PASS 232 (12s) == sanity-lnet test 233: Check for successful resends ==== 12:21:21 (1788970881) Cleaning up LNet Writer error: failed to resolve Netlink family id unloading modules via unload_modules_local on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing unload_modules_local oleg324-server: oleg324-server: LNET unconfigure error 22: Invalid argument modules unloaded. /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a oleg324-server: Writer error: failed to resolve Netlink family id oleg324-server: opening /dev/lnet failed: No such file or directory oleg324-server: hint: the kernel modules may not be loaded oleg324-server: IOC_LIBCFS_GET_NI error 2: No such file or directory pdsh@oleg324-client: oleg324-server: ssh exited with exit code 1 oleg324-server: oleg324-server.virtnet: executing load_lnet oleg324-server: Loading modules from /home/green/git/lustre-release/lustre oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg324-server: detected 4 online CPUs by sysfs oleg324-server: MODOPTS_LIBCFS= oleg324-server: Force libcfs to create 2 CPU partitions /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.124@tcp discover: - primary nid: 192.168.203.124@tcp Multi-Rail: true peer_ni: - nid: 192.168.203.124@tcp oleg324-server: oleg324-server.virtnet: executing lnet_if_list oleg324-server: oleg324-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 oleg324-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.124 oleg324-server: default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.124 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.24 default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.24 net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.203.24@tcp status: up interfaces: 0: ens2 - net type: tcp1 local NI(s): - nid: 192.168.203.24@tcp1 status: up interfaces: 0: ens2 - primary nid: 192.168.203.124@tcp - nid: 192.168.203.124@tcp health stats: health value: 1000 - nid: 192.168.203.124@tcp1 health stats: health value: 1000 debug=+net Simulate intermittent local_interrupt /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp -m GET -e local_interrupt -r 2 /home/green/git/lustre-release/lustre/utils/lctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp1 -m GET -e local_interrupt -r 2 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp -m GET -e local_interrupt -r 2 /home/green/git/lustre-release/lustre/utils/lctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp1 -m GET -e local_interrupt -r 2 /home/green/git/lustre-release/lustre/utils/lctl net_drop add -s 192.168.203.24@tcp1 -d 192.168.203.124@tcp -m GET -e local_interrupt -r 2 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp1 -d 192.168.203.124@tcp1 -m GET -e local_interrupt -r 2 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp1 -d 192.168.203.24@tcp -m GET -e local_interrupt -r 2 /home/green/git/lustre-release/lustre/utils/lctl net_drop add -s 192.168.203.24@tcp1 -d 192.168.203.24@tcp1 -m GET -e local_interrupt -r 2 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.24@tcp 192.168.203.124@tcp ping: - primary nid: 192.168.203.124@tcp Multi-Rail: true peer_ni: - nid: 192.168.203.124@tcp - nid: 192.168.203.124@tcp1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all Check for 1 successful resend Simulate intermittent local_dropped /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp -m GET -e local_dropped -r 2 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp1 -m GET -e local_dropped -r 2 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp -m GET -e local_dropped -r 2 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp1 -m GET -e local_dropped -r 2 /home/green/git/lustre-release/lustre/utils/lctl net_drop add -s 192.168.203.24@tcp1 -d 192.168.203.124@tcp -m GET -e local_dropped -r 2 /home/green/git/lustre-release/lustre/utils/lctl net_drop add -s 192.168.203.24@tcp1 -d 192.168.203.124@tcp1 -m GET -e local_dropped -r 2 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp1 -d 192.168.203.24@tcp -m GET -e local_dropped -r 2 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp1 -d 192.168.203.24@tcp1 -m GET -e local_dropped -r 2 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.24@tcp 192.168.203.124@tcp ping: - primary nid: 192.168.203.124@tcp Multi-Rail: true peer_ni: - nid: 192.168.203.124@tcp - nid: 192.168.203.124@tcp1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all Check for 1 successful resend Simulate intermittent local_aborted /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp -m GET -e local_aborted -r 2 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp1 -m GET -e local_aborted -r 2 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp -m GET -e local_aborted -r 2 /home/green/git/lustre-release/lustre/utils/lctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp1 -m GET -e local_aborted -r 2 /home/green/git/lustre-release/lustre/utils/lctl net_drop add -s 192.168.203.24@tcp1 -d 192.168.203.124@tcp -m GET -e local_aborted -r 2 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp1 -d 192.168.203.124@tcp1 -m GET -e local_aborted -r 2 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp1 -d 192.168.203.24@tcp -m GET -e local_aborted -r 2 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp1 -d 192.168.203.24@tcp1 -m GET -e local_aborted -r 2 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.24@tcp 192.168.203.124@tcp ping: - primary nid: 192.168.203.124@tcp Multi-Rail: true peer_ni: - nid: 192.168.203.124@tcp - nid: 192.168.203.124@tcp1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all Check for 1 successful resend Simulate intermittent local_no_route /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp -m GET -e local_no_route -r 2 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp1 -m GET -e local_no_route -r 2 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp -m GET -e local_no_route -r 2 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp1 -m GET -e local_no_route -r 2 /home/green/git/lustre-release/lustre/utils/lctl net_drop add -s 192.168.203.24@tcp1 -d 192.168.203.124@tcp -m GET -e local_no_route -r 2 /home/green/git/lustre-release/lustre/utils/lctl net_drop add -s 192.168.203.24@tcp1 -d 192.168.203.124@tcp1 -m GET -e local_no_route -r 2 /home/green/git/lustre-release/lustre/utils/lctl net_drop add -s 192.168.203.24@tcp1 -d 192.168.203.24@tcp -m GET -e local_no_route -r 2 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp1 -d 192.168.203.24@tcp1 -m GET -e local_no_route -r 2 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.24@tcp 192.168.203.124@tcp ping: - primary nid: 192.168.203.124@tcp Multi-Rail: true peer_ni: - nid: 192.168.203.124@tcp - nid: 192.168.203.124@tcp1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all Check for 1 successful resend Simulate intermittent local_timeout /home/green/git/lustre-release/lustre/utils/lctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp -m GET -e local_timeout -r 2 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp1 -m GET -e local_timeout -r 2 /home/green/git/lustre-release/lustre/utils/lctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp -m GET -e local_timeout -r 2 /home/green/git/lustre-release/lustre/utils/lctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp1 -m GET -e local_timeout -r 2 /home/green/git/lustre-release/lustre/utils/lctl net_drop add -s 192.168.203.24@tcp1 -d 192.168.203.124@tcp -m GET -e local_timeout -r 2 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp1 -d 192.168.203.124@tcp1 -m GET -e local_timeout -r 2 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp1 -d 192.168.203.24@tcp -m GET -e local_timeout -r 2 /home/green/git/lustre-release/lustre/utils/lctl net_drop add -s 192.168.203.24@tcp1 -d 192.168.203.24@tcp1 -m GET -e local_timeout -r 2 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.24@tcp 192.168.203.124@tcp ping: - primary nid: 192.168.203.124@tcp Multi-Rail: true peer_ni: - nid: 192.168.203.124@tcp - nid: 192.168.203.124@tcp1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all Check for 1 successful resend Simulate intermittent remote_dropped /home/green/git/lustre-release/lustre/utils/lctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp -m GET -e remote_dropped -r 2 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.124@tcp1 -m GET -e remote_dropped -r 2 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp -m GET -e remote_dropped -r 2 /home/green/git/lustre-release/lustre/utils/lctl net_drop add -s 192.168.203.24@tcp -d 192.168.203.24@tcp1 -m GET -e remote_dropped -r 2 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp1 -d 192.168.203.124@tcp -m GET -e remote_dropped -r 2 /home/green/git/lustre-release/lustre/utils/lctl net_drop add -s 192.168.203.24@tcp1 -d 192.168.203.124@tcp1 -m GET -e remote_dropped -r 2 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp1 -d 192.168.203.24@tcp -m GET -e remote_dropped -r 2 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl fault drop add -s 192.168.203.24@tcp1 -d 192.168.203.24@tcp1 -m GET -e remote_dropped -r 2 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.203.24@tcp 192.168.203.124@tcp ping: - primary nid: 192.168.203.124@tcp Multi-Rail: true peer_ni: - nid: 192.168.203.124@tcp - nid: 192.168.203.124@tcp1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set numa_range 0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_interfaces 4096 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set discovery 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set drop_asym_route 0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set retry_count 2 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set transaction_timeout 150 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_interval 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set router_sensitivity 100 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all Check for 1 successful resend oleg324-server: oleg324-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp1 --if ens2 Writer error: failed to resolve Netlink family id unloading modules via unload_modules_local on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing unload_modules_local oleg324-server: modules unloaded. oleg324-server: oleg324-server.virtnet: executing unload_modules_local oleg324-server: oleg324-server: LNET unconfigure error 22: Invalid argument pdsh@oleg324-client: oleg324-client: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-client: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 PASS 233 (61s) == sanity-lnet test 234: Verify NI status changes with manual health value transitions ========================================================== 12:22:22 (1788970942) Loading LNet and configuring DLC /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.24 NI 192.168.203.24@tcp expect status "up" found "up" Initial health value: 1000 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 0 Setting health value to 0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.203.24@tcp NI 192.168.203.24@tcp expect status "down" found "down" Health value after setting to 0: 0 Setting health value to 1000 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --nid 192.168.203.24@tcp NI 192.168.203.24@tcp expect status "up" found "up" Health value after setting to 1000: 1000 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set health_sensitivity 100 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 PASS 234 (8s) == sanity-lnet test 235: Verify NI status DOWN when health reaches 0 after failures ========================================================== 12:22:30 (1788970950) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.24 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if ens2 default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.24 default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.24 NI 192.168.203.24@tcp expect status "up" found "up" NI 192.168.203.24@tcp1 expect status "up" found "up" Adding drop rule for 192.168.203.24@tcp Setting health to 100 for 192.168.203.24@tcp /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 100 --nid 192.168.203.24@tcp Waiting 60s for 'down' NI 192.168.203.24@tcp1 expect status "up" found "up" Restoring health to 900 for 192.168.203.24@tcp /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 900 --nid 192.168.203.24@tcp Waiting 60s for 'up' Updated after 2s: want 'up' got 'up' pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 PASS 235 (10s) == sanity-lnet test 236: Local NI state propagates to routers ========================================================== 12:22:40 (1788970960) SKIP: sanity-lnet test_236 Need at least 2 remote nodes found "oleg324-server" SKIP 236 (4s) == sanity-lnet test 237: Check MR forwarding feature ===== 12:22:44 (1788970964) SKIP: sanity-lnet test_237 Need at least 2 remote nodes found "oleg324-server" SKIP 237 (4s) == sanity-lnet test 238: Check MR forwarding feature skipped for non-MR source ========================================================== 12:22:48 (1788970968) SKIP: sanity-lnet test_238 Need at least 2 remote nodes found "oleg324-server" SKIP 238 (4s) == sanity-lnet test 241: Check setting LND timeout value via lnetctl updates tunables ========================================================== 12:22:52 (1788970972) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.24 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 51 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v net: - net type: tcp local NI(s): - nid: 192.168.203.24@tcp status: up interfaces: 0: ens2 statistics: send_count: 0 recv_count: 0 drop_count: 0 put_egress_samples: 0 put_egress_min_nsec: 0 put_egress_max_nsec: 0 put_egress_avg_nsec: 0 get_rtt_samples: 0 get_rtt_min_nsec: 0 get_rtt_max_nsec: 0 get_rtt_avg_nsec: 0 put_ack_samples: 0 put_ack_min_nsec: 0 put_ack_max_nsec: 0 put_ack_avg_nsec: 0 tunables: peer_timeout: 180 peer_credits: 8 peer_buffer_credits: 0 credits: 256 lnd tunables: conns_per_peer: 1 timeout: 51 tos: -1 dev cpt: -1 CPT: "[0]" check parameter timeout value 51 on 1 NI(s) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show --net tcp -v net: - net type: tcp local NI(s): - nid: 192.168.203.24@tcp status: up interfaces: 0: ens2 statistics: send_count: 0 recv_count: 0 drop_count: 0 put_egress_samples: 0 put_egress_min_nsec: 0 put_egress_max_nsec: 0 put_egress_avg_nsec: 0 get_rtt_samples: 0 get_rtt_min_nsec: 0 get_rtt_max_nsec: 0 get_rtt_avg_nsec: 0 put_ack_samples: 0 put_ack_min_nsec: 0 put_ack_max_nsec: 0 put_ack_avg_nsec: 0 tunables: peer_timeout: 180 peer_credits: 8 peer_buffer_credits: 0 credits: 256 lnd tunables: conns_per_peer: 1 timeout: 149 tos: -1 dev cpt: -1 CPT: "[0]" check parameter timeout value 149 on 1 NI(s) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --net tcp --lnd-timeout 50 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 PASS 241 (9s) == sanity-lnet test 245: Check read-only params do not appear in export --backup ========================================================== 12:23:01 (1788970981) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.24 Check timeout present in net show -v Check timeout absent from export -b pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 PASS 245 (8s) == sanity-lnet test 250: test that linux routes are added ========================================================== 12:23:09 (1788970989) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if ens2 default via 192.168.203.254 dev ens2 192.168.203.0/24 dev ens2 proto kernel scope link src 192.168.203.24 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 PASS 250 (6s) == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 12:23:15 (1788970995) SKIP: sanity-lnet test_251 Need kfi NETTYPE SKIP 251 (3s) == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 12:23:18 (1788970998) Cleaning up LNet Writer error: failed to resolve Netlink family id unloading modules via unload_modules_local on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing unload_modules_local oleg324-server: oleg324-server: LNET unconfigure error 22: Invalid argument modules unloaded. /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a oleg324-server: Writer error: failed to resolve Netlink family id oleg324-server: opening /dev/lnet failed: No such file or directory oleg324-server: hint: the kernel modules may not be loaded oleg324-server: IOC_LIBCFS_GET_NI error 2: No such file or directory pdsh@oleg324-client: oleg324-server: ssh exited with exit code 1 oleg324-server: oleg324-server.virtnet: executing load_lnet oleg324-server: Loading modules from /home/green/git/lustre-release/lustre oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg324-server: detected 4 online CPUs by sysfs oleg324-server: MODOPTS_LIBCFS= oleg324-server: Force libcfs to create 2 CPU partitions /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.124@tcp discover: - primary nid: 192.168.203.124@tcp Multi-Rail: true peer_ni: - nid: 192.168.203.124@tcp net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.203.24@tcp status: up interfaces: 0: ens2 - primary nid: 192.168.203.124@tcp - nid: 192.168.203.124@tcp health stats: health value: 1000 debug=+net oleg324-server: oleg324-server.virtnet: executing unload_modules_local oleg324-server: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.203.124@tcp manage: - ping: errno: -111 descr: ! 'failed to ping 192.168.203.124@tcp: Connection refused' Writer error: failed to resolve Netlink family id unloading modules via unload_modules_local on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing unload_modules_local oleg324-server: oleg324-server: LNET unconfigure error 22: Invalid argument modules unloaded. pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-client: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-client: ssh exited with exit code 2 PASS 252 (39s) == sanity-lnet test 258: TCP reset cancels an outstanding discovery Ping ========================================================== 12:23:57 (1788971037) Cleaning up LNet LNET unconfigure error 22: Invalid argument unloading modules via unload_modules_local on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing unload_modules_local oleg324-server: oleg324-server: LNET unconfigure error 22: Invalid argument modules unloaded. /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure -a oleg324-server: Writer error: failed to resolve Netlink family id oleg324-server: opening /dev/lnet failed: No such file or directory oleg324-server: hint: the kernel modules may not be loaded oleg324-server: IOC_LIBCFS_GET_NI error 2: No such file or directory pdsh@oleg324-client: oleg324-server: ssh exited with exit code 1 oleg324-server: oleg324-server.virtnet: executing load_lnet oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg324-server: Loading modules from /home/green/git/lustre-release/lustre oleg324-server: detected 4 online CPUs by sysfs oleg324-server: MODOPTS_LIBCFS= oleg324-server: Force libcfs to create 2 CPU partitions /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.203.124@tcp discover: - primary nid: 192.168.203.124@tcp Multi-Rail: true peer_ni: - nid: 192.168.203.124@tcp net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.203.24@tcp status: up interfaces: 0: ens2 - primary nid: 192.168.203.124@tcp - nid: 192.168.203.124@tcp health stats: health value: 1000 debug=+net 258: client oleg324-client.virtnet (192.168.203.24@tcp), server oleg324-server /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.203.124@tcp ping: - primary nid: 192.168.203.124@tcp Multi-Rail: true peer_ni: - nid: 192.168.203.124@tcp 258: initial ping complete; cleared client debug buffer 258: remove stale remote GET drop rule pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 failed to delete drop rule: Object not found 258: install remote GET drop rule for forced discovery 258: start forced discovery (log /tmp/sanity-lnet-258-discover.log) 258: forced discovery pid 97864; deadline in 180s 258: client sent the forced discovery GET 258: client peer state after GET drop (PING_SENT=1) peer: - primary nid: 192.168.203.124@tcp Multi-Rail: true peer state: 16977 peer ni: - nid: 192.168.203.124@tcp udsp info: net priority: -1 nid priority: -1 state: NA max_ni_tx_credits: 8 available_tx_credits: 8 min_tx_credits: 7 tx_q_num_of_buf: 0 available_rtr_credits: 8 min_rtr_credits: 8 refcount: 2 statistics: send_count: 5 recv_count: 4 drop_count: 0 put_egress_samples: 1 put_egress_min_nsec: 4268251 put_egress_max_nsec: 4268251 put_egress_avg_nsec: 4268251 get_rtt_samples: 2 get_rtt_min_nsec: 7703674 get_rtt_max_nsec: 23840728 get_rtt_avg_nsec: 15772201 put_ack_samples: 1 put_ack_min_nsec: 11733807 put_ack_max_nsec: 11733807 put_ack_avg_nsec: 11733807 sent_stats: put: 2 get: 3 reply: 0 ack: 0 hello: 0 received_stats: put: 1 get: 0 reply: 2 ack: 1 hello: 0 dropped_stats: put: 0 get: 0 reply: 0 ack: 0 hello: 0 health stats: health value: 1000 dropped: 0 timeout: 0 error: 0 network timeout: 0 ping_count: 0 next_ping: 0 258: forced discovery remains active 258: server SockLND sockets before ss -K 0 0 192.168.203.124:988 192.168.203.24:1021 timer:(keepalive,7.859ms,0) ino:180231 sk:1 <-> 0 0 192.168.203.124:988 192.168.203.24:1023 timer:(keepalive,7.828ms,0) ino:179538 sk:2 <-> 0 0 192.168.203.124:988 192.168.203.24:1022 timer:(keepalive,7.850ms,0) ino:179540 sk:3 <-> 258: ss -K rc 0; output follows 258: invoking ss -K 258: server SockLND sockets after ss -K 0 0 192.168.203.124:988 192.168.203.24:1021 timer:(keepalive,1.654ms,0) ino:180231 sk:1 <-> 0 0 192.168.203.124:988 192.168.203.24:1023 timer:(keepalive,1.623ms,0) ino:179538 sk:2 <-> 0 0 192.168.203.124:988 192.168.203.24:1022 timer:(keepalive,1.645ms,0) ino:179540 sk:3 <-> 258: every selected server socket identity survived 258: stop forced discovery pid 97864 258: remove remote GET drop rule for 192.168.203.24@tcp Writer error: failed to resolve Netlink family id unloading modules via unload_modules_local on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing unload_modules_local oleg324-server: modules unloaded. oleg324-server: oleg324-server.virtnet: executing unload_modules_local oleg324-server: oleg324-server: LNET unconfigure error 22: Invalid argument SKIP: sanity-lnet test_258 ss -K did not destroy selected SockLND socket for 192.168.203.124@tcp Cleaning up LNet LNET unconfigure error 22: Invalid argument unloading modules via unload_modules_local on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing unload_modules_local oleg324-server: oleg324-server: LNET unconfigure error 22: Invalid argument modules unloaded. Checking servers environments Checking clients oleg324-client.virtnet environments /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing load_modules_local oleg324-server: Loading modules from /home/green/git/lustre-release/lustre oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg324-server: detected 4 online CPUs by sysfs oleg324-server: MODOPTS_LIBCFS= oleg324-server: Force libcfs to create 2 CPU partitions oleg324-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg324-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg324-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Setup mgs, mdt, osts Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg324-server: oleg324-server.virtnet: executing set_default_debug neterror net nettrace malloc all pdsh@oleg324-client: oleg324-server: ssh exited with exit code 1 Started lustre-MDT0000 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg324-server: oleg324-server.virtnet: executing set_default_debug neterror net nettrace malloc all pdsh@oleg324-client: oleg324-server: ssh exited with exit code 1 Started lustre-OST0000 Start ost2: mount -t lustre -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg324-server: oleg324-server.virtnet: executing set_default_debug neterror net nettrace malloc all pdsh@oleg324-client: oleg324-server: ssh exited with exit code 1 Started lustre-OST0001 Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre Starting client oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre Started clients oleg324-client.virtnet: 192.168.203.124@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff98bb44a52000.idle_timeout=debug osc.lustre-OST0001-osc-ffff98bb44a52000.idle_timeout=debug disable quota as required SKIP 258 (156s) SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 12:26:37 (1788971197) SKIP: sanity-lnet test_255 kernel ip2nets deprecated in 2.17.53 SKIP 255 (4s) == sanity-lnet test 256: Router should not drop messages that are past the deadline ========================================================== 12:26:41 (1788971201) SKIP: sanity-lnet test_256 Need at least 2 remote nodes found "oleg324-server" SKIP 256 (3s) == sanity-lnet test 257: Test graceful router shutdown/startup ========================================================== 12:26:44 (1788971204) SKIP: sanity-lnet test_257 Need at least 3 remote nodes found "oleg324-server" SKIP 257 (4s) == sanity-lnet test 260: test that linux sysctl parameter are set correctly ========================================================== 12:26:48 (1788971208) Setting default values and create backup for check Check default configuration found: net.ipv4.neigh.default.gc_thresh1 128 found: net.ipv4.neigh.default.gc_thresh2 512 found: net.ipv4.neigh.default.gc_thresh3 1024 found: net.ipv6.neigh.default.gc_thresh1 128 found: net.ipv6.neigh.default.gc_thresh2 512 found: net.ipv6.neigh.default.gc_thresh3 1024 /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing load_modules_local oleg324-server: Loading modules from /home/green/git/lustre-release/lustre oleg324-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg324-server: detected 4 online CPUs by sysfs oleg324-server: MODOPTS_LIBCFS= oleg324-server: Force libcfs to create 2 CPU partitions enable_sysctl_setup set to 0 New configuration enable_sysctl_setup set to 1 /home/green/git/lustre-release/lustre/scripts/lnet-sysctl-config: line 28: /etc/lnet-sysctl.conf: No such file or directory Check new configuration /home/green/git/lustre-release/lustre/tests/sanity-lnet.sh: line 5967: /etc/lnet-sysctl.conf: No such file or directory Reset to original values enable_sysctl_setup set to 0 Check original configuration found: net.ipv4.neigh.default.gc_thresh1 128 found: net.ipv4.neigh.default.gc_thresh2 512 found: net.ipv4.neigh.default.gc_thresh3 1024 found: net.ipv6.neigh.default.gc_thresh1 128 found: net.ipv6.neigh.default.gc_thresh2 512 found: net.ipv6.neigh.default.gc_thresh3 1024 lustre_rmmod: wait 41s for 6 OBD devices lustre_rmmod: OBD devices after 60s: 0 UP mgc MGC192.168.203.124@tcp 94d249cb-8ce8-4694-9530-465f9f595f02 5 1 UP lov lustre-clilov-ffff98bb44a52000 bb578dba-66a2-4424-b6c3-0338dda03657 4 2 UP lmv lustre-clilmv-ffff98bb44a52000 bb578dba-66a2-4424-b6c3-0338dda03657 5 3 UP mdc lustre-MDT0000-mdc-ffff98bb44a52000 bb578dba-66a2-4424-b6c3-0338dda03657 5 4 UP osc lustre-OST0000-osc-ffff98bb44a52000 bb578dba-66a2-4424-b6c3-0338dda03657 5 5 UP osc lustre-OST0001-osc-ffff98bb44a52000 bb578dba-66a2-4424-b6c3-0338dda03657 5 rmmod: ERROR: Module mdd is in use unloading modules via unload_modules_local on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing unload_modules_local oleg324-server: lustre_rmmod: wait 41s for 18 OBD devices oleg324-server: lustre_rmmod: OBD devices after 60s: oleg324-server: 0 UP osd-zfs lustre-MDT0000-osd lustre-MDT0000-osd_UUID 10 oleg324-server: 1 UP mgs MGS MGS 7 oleg324-server: 2 UP mgc MGC192.168.203.124@tcp c2a4ea9e-b180-40d2-b7dd-3b0d82230c17 5 oleg324-server: 3 UP mds MDS MDS_uuid 3 oleg324-server: 4 UP lod lustre-MDT0000-mdtlov lustre-MDT0000-mdtlov_UUID 4 oleg324-server: 5 UP mdt lustre-MDT0000 lustre-MDT0000_UUID 11 oleg324-server: 6 UP mdd lustre-MDD0000 lustre-MDD0000_UUID 4 oleg324-server: 7 UP qmt lustre-QMT0000 lustre-QMT0000_UUID 4 oleg324-server: 8 UP osp lustre-OST0000-osc-MDT0000 lustre-MDT0000-mdtlov_UUID 5 oleg324-server: 9 UP osp lustre-OST0001-osc-MDT0000 lustre-MDT0000-mdtlov_UUID 5 oleg324-server: 10 UP lwp lustre-MDT0000-lwp-MDT0000 lustre-MDT0000-lwp-MDT0000_UUID 5 oleg324-server: 11 UP osd-zfs lustre-OST0000-osd lustre-OST0000-osd_UUID 5 oleg324-server: 12 UP ost OSS OSS_uuid 3 oleg324-server: 13 UP obdfilter lustre-OST0000 lustre-OST0000_UUID 5 oleg324-server: 14 UP lwp lustre-MDT0000-lwp-OST0000 lustre-MDT0000-lwp-OST0000_UUID 5 oleg324-server: 15 UP osd-zfs lustre-OST0001-osd lustre-OST0001-osd_UUID 5 oleg324-server: 16 UP obdfilter lustre-OST0001 lustre-OST0001_UUID 5 oleg324-server: 17 UP lwp lustre-MDT0000-lwp-OST0001 lustre-MDT0000-lwp-OST0001_UUID 5 oleg324-server: oleg324-server: rmmod: ERROR: Module mdd is in use pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 sanity-lnet test_260: @@@@@@ FAIL: Failed to cleanup Modules Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7496:error() = /home/green/git/lustre-release/lustre/tests/sanity-lnet.sh:6067:test_260() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7873:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7936:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7750:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-lnet.sh:6069:main() Dumping lctl log to /tmp/testlogs//sanity-lnet.test_260.*.1788971358.log rsync: chown "/tmp/testlogs/.sanity-lnet.test_260.debug_log.oleg324-server.1788971358.log.mHvsZ3" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-lnet.test_260.dmesg.oleg324-server.1788971358.log.e575Kn" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3] FAIL 260 (153s) == sanity-lnet test 265: Import of invalid CPT should fail ========================================================== 12:29:27 (1788971367) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure --- configure: - lnet: errno: -16 descr: ! "LNet configure error: Object busy" ... sanity-lnet test_265: @@@@@@ FAIL: lnetctl lnet unconfigure failed 240 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7496:error() = /home/green/git/lustre-release/lustre/tests/sanity-lnet.sh:147:reinit_dlc() = /home/green/git/lustre-release/lustre/tests/sanity-lnet.sh:6072:test_265() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7873:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7936:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7750:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-lnet.sh:6102:main() Dumping lctl log to /tmp/testlogs//sanity-lnet.test_265.*.1788971370.log rsync: chown "/tmp/testlogs/.sanity-lnet.test_265.debug_log.oleg324-server.1788971370.log.giNqx9" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-lnet.test_265.dmesg.oleg324-server.1788971370.log.adC9KE" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3] FAIL 265 (6s) == sanity-lnet test 266: Validate CPT parsing in network sequence ========================================================== 12:29:40 (1788971380) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure --- configure: - lnet: errno: -16 descr: ! "LNet configure error: Object busy" ... sanity-lnet test_266: @@@@@@ FAIL: lnetctl lnet unconfigure failed 240 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7496:error() = /home/green/git/lustre-release/lustre/tests/sanity-lnet.sh:147:reinit_dlc() = /home/green/git/lustre-release/lustre/tests/sanity-lnet.sh:6105:test_266() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7873:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7936:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7750:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-lnet.sh:6158:main() Dumping lctl log to /tmp/testlogs//sanity-lnet.test_266.*.1788971384.log rsync: chown "/tmp/testlogs/.sanity-lnet.test_266.debug_log.oleg324-server.1788971384.log.IeW3st" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-lnet.test_266.dmesg.oleg324-server.1788971384.log.e3hPc1" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3] FAIL 266 (6s) == sanity-lnet test 270: Test NID metadata =============== 12:29:53 (1788971393) Cleaning up LNet lustre_rmmod: wait 41s for 6 OBD devices lustre_rmmod: OBD devices after 60s: 0 UP mgc MGC192.168.203.124@tcp 94d249cb-8ce8-4694-9530-465f9f595f02 5 1 UP lov lustre-clilov-ffff98bb44a52000 bb578dba-66a2-4424-b6c3-0338dda03657 4 2 UP lmv lustre-clilmv-ffff98bb44a52000 bb578dba-66a2-4424-b6c3-0338dda03657 5 3 UP mdc lustre-MDT0000-mdc-ffff98bb44a52000 bb578dba-66a2-4424-b6c3-0338dda03657 5 4 UP osc lustre-OST0000-osc-ffff98bb44a52000 bb578dba-66a2-4424-b6c3-0338dda03657 5 5 UP osc lustre-OST0001-osc-ffff98bb44a52000 bb578dba-66a2-4424-b6c3-0338dda03657 5 rmmod: ERROR: Module mdd is in use unloading modules via unload_modules_local on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing unload_modules_local oleg324-server: lustre_rmmod: wait 41s for 18 OBD devices oleg324-server: lustre_rmmod: OBD devices after 60s: oleg324-server: 0 UP osd-zfs lustre-MDT0000-osd lustre-MDT0000-osd_UUID 10 oleg324-server: 1 UP mgs MGS MGS 7 oleg324-server: 2 UP mgc MGC192.168.203.124@tcp c2a4ea9e-b180-40d2-b7dd-3b0d82230c17 5 oleg324-server: 3 UP mds MDS MDS_uuid 3 oleg324-server: 4 UP lod lustre-MDT0000-mdtlov lustre-MDT0000-mdtlov_UUID 4 oleg324-server: 5 UP mdt lustre-MDT0000 lustre-MDT0000_UUID 11 oleg324-server: 6 UP mdd lustre-MDD0000 lustre-MDD0000_UUID 4 oleg324-server: 7 UP qmt lustre-QMT0000 lustre-QMT0000_UUID 4 oleg324-server: 8 UP osp lustre-OST0000-osc-MDT0000 lustre-MDT0000-mdtlov_UUID 5 oleg324-server: 9 UP osp lustre-OST0001-osc-MDT0000 lustre-MDT0000-mdtlov_UUID 5 oleg324-server: 10 UP lwp lustre-MDT0000-lwp-MDT0000 lustre-MDT0000-lwp-MDT0000_UUID 5 oleg324-server: 11 UP osd-zfs lustre-OST0000-osd lustre-OST0000-osd_UUID 5 oleg324-server: 12 UP ost OSS OSS_uuid 3 oleg324-server: 13 UP obdfilter lustre-OST0000 lustre-OST0000_UUID 5 oleg324-server: 14 UP lwp lustre-MDT0000-lwp-OST0000 lustre-MDT0000-lwp-OST0000_UUID 5 oleg324-server: 15 UP osd-zfs lustre-OST0001-osd lustre-OST0001-osd_UUID 5 oleg324-server: 16 UP obdfilter lustre-OST0001 lustre-OST0001_UUID 5 oleg324-server: 17 UP lwp lustre-MDT0000-lwp-OST0001 lustre-MDT0000-lwp-OST0001_UUID 5 oleg324-server: oleg324-server: rmmod: ERROR: Module mdd is in use pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 sanity-lnet test_270: @@@@@@ FAIL: Failed to unload modules before test execution Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7496:error() = /home/green/git/lustre-release/lustre/tests/sanity-lnet.sh:6163:test_270() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7873:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7936:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7750:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-lnet.sh:6180:main() Dumping lctl log to /tmp/testlogs//sanity-lnet.test_270.*.1788971526.log rsync: chown "/tmp/testlogs/.sanity-lnet.test_270.debug_log.oleg324-server.1788971526.log.pcV24e" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-lnet.test_270.dmesg.oleg324-server.1788971526.log.3JDISi" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3] FAIL 270 (136s) == sanity-lnet test 280: Don't panic when request_module fails ========================================================== 12:32:17 (1788971537) Cleaning up LNet lustre_rmmod: wait 41s for 6 OBD devices lustre_rmmod: OBD devices after 60s: 0 UP mgc MGC192.168.203.124@tcp 94d249cb-8ce8-4694-9530-465f9f595f02 5 1 UP lov lustre-clilov-ffff98bb44a52000 bb578dba-66a2-4424-b6c3-0338dda03657 4 2 UP lmv lustre-clilmv-ffff98bb44a52000 bb578dba-66a2-4424-b6c3-0338dda03657 5 3 UP mdc lustre-MDT0000-mdc-ffff98bb44a52000 bb578dba-66a2-4424-b6c3-0338dda03657 5 4 UP osc lustre-OST0000-osc-ffff98bb44a52000 bb578dba-66a2-4424-b6c3-0338dda03657 5 5 UP osc lustre-OST0001-osc-ffff98bb44a52000 bb578dba-66a2-4424-b6c3-0338dda03657 5 rmmod: ERROR: Module mdd is in use unloading modules via unload_modules_local on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing unload_modules_local oleg324-server: lustre_rmmod: wait 41s for 18 OBD devices oleg324-server: lustre_rmmod: OBD devices after 60s: oleg324-server: 0 UP osd-zfs lustre-MDT0000-osd lustre-MDT0000-osd_UUID 10 oleg324-server: 1 UP mgs MGS MGS 7 oleg324-server: 2 UP mgc MGC192.168.203.124@tcp c2a4ea9e-b180-40d2-b7dd-3b0d82230c17 5 oleg324-server: 3 UP mds MDS MDS_uuid 3 oleg324-server: 4 UP lod lustre-MDT0000-mdtlov lustre-MDT0000-mdtlov_UUID 4 oleg324-server: 5 UP mdt lustre-MDT0000 lustre-MDT0000_UUID 11 oleg324-server: 6 UP mdd lustre-MDD0000 lustre-MDD0000_UUID 4 oleg324-server: 7 UP qmt lustre-QMT0000 lustre-QMT0000_UUID 4 oleg324-server: 8 UP osp lustre-OST0000-osc-MDT0000 lustre-MDT0000-mdtlov_UUID 5 oleg324-server: 9 UP osp lustre-OST0001-osc-MDT0000 lustre-MDT0000-mdtlov_UUID 5 oleg324-server: 10 UP lwp lustre-MDT0000-lwp-MDT0000 lustre-MDT0000-lwp-MDT0000_UUID 5 oleg324-server: 11 UP osd-zfs lustre-OST0000-osd lustre-OST0000-osd_UUID 5 oleg324-server: 12 UP ost OSS OSS_uuid 3 oleg324-server: 13 UP obdfilter lustre-OST0000 lustre-OST0000_UUID 5 oleg324-server: 14 UP lwp lustre-MDT0000-lwp-OST0000 lustre-MDT0000-lwp-OST0000_UUID 5 oleg324-server: 15 UP osd-zfs lustre-OST0001-osd lustre-OST0001-osd_UUID 5 oleg324-server: 16 UP obdfilter lustre-OST0001 lustre-OST0001_UUID 5 oleg324-server: 17 UP lwp lustre-MDT0000-lwp-OST0001 lustre-MDT0000-lwp-OST0001_UUID 5 oleg324-server: oleg324-server: rmmod: ERROR: Module mdd is in use pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 sanity-lnet test_280: @@@@@@ FAIL: Failed to unload modules before test execution Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7496:error() = /home/green/git/lustre-release/lustre/tests/sanity-lnet.sh:6183:test_280() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7873:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7936:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7750:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-lnet.sh:6210:main() Dumping lctl log to /tmp/testlogs//sanity-lnet.test_280.*.1788971671.log rsync: chown "/tmp/testlogs/.sanity-lnet.test_280.debug_log.oleg324-server.1788971671.log.RZMo1H" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-lnet.test_280.dmesg.oleg324-server.1788971671.log.Qdicyv" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3] FAIL 280 (138s) == sanity-lnet test 290: Check that lnet acceptor is using explicit address binding ========================================================== 12:34:40 (1788971680) Cleaning up LNet lustre_rmmod: wait 41s for 6 OBD devices lustre_rmmod: OBD devices after 60s: 0 UP mgc MGC192.168.203.124@tcp 94d249cb-8ce8-4694-9530-465f9f595f02 5 1 UP lov lustre-clilov-ffff98bb44a52000 bb578dba-66a2-4424-b6c3-0338dda03657 4 2 UP lmv lustre-clilmv-ffff98bb44a52000 bb578dba-66a2-4424-b6c3-0338dda03657 5 3 UP mdc lustre-MDT0000-mdc-ffff98bb44a52000 bb578dba-66a2-4424-b6c3-0338dda03657 5 4 UP osc lustre-OST0000-osc-ffff98bb44a52000 bb578dba-66a2-4424-b6c3-0338dda03657 5 5 UP osc lustre-OST0001-osc-ffff98bb44a52000 bb578dba-66a2-4424-b6c3-0338dda03657 5 rmmod: ERROR: Module mdd is in use unloading modules via unload_modules_local on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing unload_modules_local oleg324-server: lustre_rmmod: wait 41s for 18 OBD devices oleg324-server: lustre_rmmod: OBD devices after 60s: oleg324-server: 0 UP osd-zfs lustre-MDT0000-osd lustre-MDT0000-osd_UUID 10 oleg324-server: 1 UP mgs MGS MGS 7 oleg324-server: oleg324-server: 2 UP mgc MGC192.168.203.124@tcp c2a4ea9e-b180-40d2-b7dd-3b0d82230c17 5 oleg324-server: 3 UP mds MDS MDS_uuid 3 oleg324-server: 4 UP lod lustre-MDT0000-mdtlov lustre-MDT0000-mdtlov_UUID 4 oleg324-server: 5 UP mdt lustre-MDT0000 lustre-MDT0000_UUID 11 oleg324-server: 6 UP mdd lustre-MDD0000 lustre-MDD0000_UUID 4 oleg324-server: 7 UP qmt lustre-QMT0000 lustre-QMT0000_UUID 4 oleg324-server: 8 UP osp lustre-OST0000-osc-MDT0000 lustre-MDT0000-mdtlov_UUID 5 oleg324-server: 9 UP osp lustre-OST0001-osc-MDT0000 lustre-MDT0000-mdtlov_UUID 5 oleg324-server: 10 UP lwp lustre-MDT0000-lwp-MDT0000 lustre-MDT0000-lwp-MDT0000_UUID 5 oleg324-server: 11 UP osd-zfs lustre-OST0000-osd lustre-OST0000-osd_UUID 5 oleg324-server: 12 UP ost OSS OSS_uuid 3 oleg324-server: 13 UP obdfilter lustre-OST0000 lustre-OST0000_UUID 5 oleg324-server: 14 UP lwp lustre-MDT0000-lwp-OST0000 lustre-MDT0000-lwp-OST0000_UUID 5 oleg324-server: 15 UP osd-zfs lustre-OST0001-osd lustre-OST0001-osd_UUID 5 oleg324-server: 16 UP obdfilter lustre-OST0001 lustre-OST0001_UUID 5 oleg324-server: 17 UP lwp lustre-MDT0000-lwp-OST0001 lustre-MDT0000-lwp-OST0001_UUID 5 oleg324-server: rmmod: ERROR: Module mdd is in use pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 sanity-lnet test_290: @@@@@@ FAIL: Failed to unload modules before test execution Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7496:error() = /home/green/git/lustre-release/lustre/tests/sanity-lnet.sh:6218:test_290() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7873:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7936:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7750:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-lnet.sh:6234:main() Dumping lctl log to /tmp/testlogs//sanity-lnet.test_290.*.1788971815.log rsync: chown "/tmp/testlogs/.sanity-lnet.test_290.debug_log.oleg324-server.1788971815.log.emlDFi" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-lnet.test_290.dmesg.oleg324-server.1788971815.log.ZLHYJS" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3] FAIL 290 (138s) == sanity-lnet test 291: load lnet using custom accept port for bulk tcp connections ========================================================== 12:37:06 (1788971826) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure --- configure: - lnet: errno: -16 descr: ! "LNet configure error: Object busy" ... sanity-lnet test_291: @@@@@@ FAIL: lnetctl lnet unconfigure failed 240 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7496:error() = /home/green/git/lustre-release/lustre/tests/sanity-lnet.sh:147:reinit_dlc() = /home/green/git/lustre-release/lustre/tests/sanity-lnet.sh:6240:test_291() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7873:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7936:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7750:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-lnet.sh:6273:main() Dumping lctl log to /tmp/testlogs//sanity-lnet.test_291.*.1788971830.log rsync: chown "/tmp/testlogs/.sanity-lnet.test_291.debug_log.oleg324-server.1788971830.log.tvz0dZ" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-lnet.test_291.dmesg.oleg324-server.1788971830.log.6glOoF" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3] FAIL 291 (7s) == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 12:37:24 (1788971844) Cleaning up LNet lustre_rmmod: wait 41s for 6 OBD devices lustre_rmmod: OBD devices after 60s: 0 UP mgc MGC192.168.203.124@tcp 94d249cb-8ce8-4694-9530-465f9f595f02 5 1 UP lov lustre-clilov-ffff98bb44a52000 bb578dba-66a2-4424-b6c3-0338dda03657 4 2 UP lmv lustre-clilmv-ffff98bb44a52000 bb578dba-66a2-4424-b6c3-0338dda03657 5 3 UP mdc lustre-MDT0000-mdc-ffff98bb44a52000 bb578dba-66a2-4424-b6c3-0338dda03657 5 4 UP osc lustre-OST0000-osc-ffff98bb44a52000 bb578dba-66a2-4424-b6c3-0338dda03657 5 5 UP osc lustre-OST0001-osc-ffff98bb44a52000 bb578dba-66a2-4424-b6c3-0338dda03657 5 rmmod: ERROR: Module mdd is in use unloading modules via unload_modules_local on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing unload_modules_local oleg324-server: lustre_rmmod: wait 41s for 18 OBD devices oleg324-server: lustre_rmmod: OBD devices after 60s: oleg324-server: 0 UP osd-zfs lustre-MDT0000-osd lustre-MDT0000-osd_UUID 10 oleg324-server: 1 UP mgs MGS MGS 7 oleg324-server: 2 UP mgc MGC192.168.203.124@tcp c2a4ea9e-b180-40d2-b7dd-3b0d82230c17 5 oleg324-server: 3 UP mds MDS MDS_uuid 3 oleg324-server: 4 UP lod lustre-MDT0000-mdtlov lustre-MDT0000-mdtlov_UUID 4 oleg324-server: 5 UP mdt lustre-MDT0000 lustre-MDT0000_UUID 11 oleg324-server: 6 UP mdd lustre-MDD0000 lustre-MDD0000_UUID 4 oleg324-server: 7 UP qmt lustre-QMT0000 lustre-QMT0000_UUID 4 oleg324-server: 8 UP osp lustre-OST0000-osc-MDT0000 lustre-MDT0000-mdtlov_UUID 5 oleg324-server: 9 UP osp lustre-OST0001-osc-MDT0000 lustre-MDT0000-mdtlov_UUID 5 oleg324-server: 10 UP lwp lustre-MDT0000-lwp-MDT0000 lustre-MDT0000-lwp-MDT0000_UUID 5 oleg324-server: 11 UP osd-zfs lustre-OST0000-osd lustre-OST0000-osd_UUID 5 oleg324-server: 12 UP ost OSS OSS_uuid 3 oleg324-server: 13 UP obdfilter lustre-OST0000 lustre-OST0000_UUID 5 oleg324-server: 14 UP lwp lustre-MDT0000-lwp-OST0000 lustre-MDT0000-lwp-OST0000_UUID 5 oleg324-server: 15 UP osd-zfs lustre-OST0001-osd lustre-OST0001-osd_UUID 5 oleg324-server: 16 UP obdfilter lustre-OST0001 lustre-OST0001_UUID 5 oleg324-server: 17 UP lwp lustre-MDT0000-lwp-OST0001 lustre-MDT0000-lwp-OST0001_UUID 5 oleg324-server: oleg324-server: rmmod: ERROR: Module mdd is in use pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 sanity-lnet test_300: @@@@@@ FAIL: test_300 failed with 2 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7496:error() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7873:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7936:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7750:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-lnet.sh:6309:main() Dumping lctl log to /tmp/testlogs//sanity-lnet.test_300.*.1788971980.log rsync: chown "/tmp/testlogs/.sanity-lnet.test_300.debug_log.oleg324-server.1788971980.log.8xoKxM" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-lnet.test_300.dmesg.oleg324-server.1788971980.log.w6VJ8H" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3] FAIL 300 (139s) == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 12:39:50 (1788971990) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure --- configure: - lnet: errno: -16 descr: ! "LNet configure error: Object busy" ... sanity-lnet test_301: @@@@@@ FAIL: lnetctl lnet unconfigure failed 240 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7496:error() = /home/green/git/lustre-release/lustre/tests/sanity-lnet.sh:147:reinit_dlc() = /home/green/git/lustre-release/lustre/tests/sanity-lnet.sh:6314:test_301() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7873:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7936:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7750:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-lnet.sh:6323:main() Dumping lctl log to /tmp/testlogs//sanity-lnet.test_301.*.1788971993.log rsync: chown "/tmp/testlogs/.sanity-lnet.test_301.debug_log.oleg324-server.1788971993.log.IY93aj" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-lnet.test_301.dmesg.oleg324-server.1788971993.log.iXYOXX" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3] FAIL 301 (6s) == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 12:40:03 (1788972003) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure --- configure: - lnet: errno: -16 descr: ! "LNet configure error: Object busy" ... sanity-lnet test_302: @@@@@@ FAIL: lnetctl lnet unconfigure failed 240 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7496:error() = /home/green/git/lustre-release/lustre/tests/sanity-lnet.sh:147:reinit_dlc() = /home/green/git/lustre-release/lustre/tests/sanity-lnet.sh:6328:test_302() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7873:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7936:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7750:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-lnet.sh:6339:main() Dumping lctl log to /tmp/testlogs//sanity-lnet.test_302.*.1788972006.log rsync: chown "/tmp/testlogs/.sanity-lnet.test_302.debug_log.oleg324-server.1788972006.log.PzfpEf" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-lnet.test_302.dmesg.oleg324-server.1788972006.log.QUb1jx" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3] FAIL 302 (6s) == sanity-lnet test 303: Check peer NI health after link down ========================================================== 12:40:16 (1788972016) Cleaning up LNet lustre_rmmod: wait 41s for 6 OBD devices lustre_rmmod: OBD devices after 60s: 0 UP mgc MGC192.168.203.124@tcp 94d249cb-8ce8-4694-9530-465f9f595f02 5 1 UP lov lustre-clilov-ffff98bb44a52000 bb578dba-66a2-4424-b6c3-0338dda03657 4 2 UP lmv lustre-clilmv-ffff98bb44a52000 bb578dba-66a2-4424-b6c3-0338dda03657 5 3 UP mdc lustre-MDT0000-mdc-ffff98bb44a52000 bb578dba-66a2-4424-b6c3-0338dda03657 5 4 UP osc lustre-OST0000-osc-ffff98bb44a52000 bb578dba-66a2-4424-b6c3-0338dda03657 5 5 UP osc lustre-OST0001-osc-ffff98bb44a52000 bb578dba-66a2-4424-b6c3-0338dda03657 5 rmmod: ERROR: Module mdd is in use unloading modules via unload_modules_local on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing unload_modules_local oleg324-server: lustre_rmmod: wait 40s for 18 OBD devices oleg324-server: lustre_rmmod: OBD devices after 60s: oleg324-server: 0 UP osd-zfs lustre-MDT0000-osd lustre-MDT0000-osd_UUID 10 oleg324-server: 1 UP mgs MGS MGS 7 oleg324-server: 2 UP mgc MGC192.168.203.124@tcp c2a4ea9e-b180-40d2-b7dd-3b0d82230c17 5 oleg324-server: 3 UP mds MDS MDS_uuid 3 oleg324-server: 4 UP lod lustre-MDT0000-mdtlov lustre-MDT0000-mdtlov_UUID 4 oleg324-server: 5 UP mdt lustre-MDT0000 lustre-MDT0000_UUID 11 oleg324-server: 6 UP mdd lustre-MDD0000 lustre-MDD0000_UUID 4 oleg324-server: 7 UP qmt lustre-QMT0000 lustre-QMT0000_UUID 4 oleg324-server: 8 UP osp lustre-OST0000-osc-MDT0000 lustre-MDT0000-mdtlov_UUID 5 oleg324-server: 9 UP osp lustre-OST0001-osc-MDT0000 lustre-MDT0000-mdtlov_UUID 5 oleg324-server: 10 UP lwp lustre-MDT0000-lwp-MDT0000 lustre-MDT0000-lwp-MDT0000_UUID 5 oleg324-server: 11 UP osd-zfs lustre-OST0000-osd lustre-OST0000-osd_UUID 5 oleg324-server: 12 UP ost OSS OSS_uuid 3 oleg324-server: 13 UP obdfilter lustre-OST0000 lustre-OST0000_UUID 5 oleg324-server: 14 UP lwp lustre-MDT0000-lwp-OST0000 lustre-MDT0000-lwp-OST0000_UUID 5 oleg324-server: 15 UP osd-zfs lustre-OST0001-osd lustre-OST0001-osd_UUID 5 oleg324-server: 16 UP obdfilter lustre-OST0001 lustre-OST0001_UUID 5 oleg324-server: 17 UP lwp lustre-MDT0000-lwp-OST0001 lustre-MDT0000-lwp-OST0001_UUID 5 oleg324-server: oleg324-server: rmmod: ERROR: Module mdd is in use pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 sanity-lnet test_303: @@@@@@ FAIL: Failed to cleanup before test execution Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7496:error() = /home/green/git/lustre-release/lustre/tests/sanity-lnet.sh:3028:setup_health_test() = /home/green/git/lustre-release/lustre/tests/sanity-lnet.sh:6344:test_303() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7873:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7936:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7750:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-lnet.sh:6378:main() Dumping lctl log to /tmp/testlogs//sanity-lnet.test_303.*.1788972150.log rsync: chown "/tmp/testlogs/.sanity-lnet.test_303.debug_log.oleg324-server.1788972150.log.tloeOW" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-lnet.test_303.dmesg.oleg324-server.1788972150.log.D8cjRJ" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3] FAIL 303 (137s) == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 12:42:40 (1788972160) Cleaning up LNet lustre_rmmod: wait 41s for 6 OBD devices lustre_rmmod: OBD devices after 60s: 0 UP mgc MGC192.168.203.124@tcp 94d249cb-8ce8-4694-9530-465f9f595f02 5 1 UP lov lustre-clilov-ffff98bb44a52000 bb578dba-66a2-4424-b6c3-0338dda03657 4 2 UP lmv lustre-clilmv-ffff98bb44a52000 bb578dba-66a2-4424-b6c3-0338dda03657 5 3 UP mdc lustre-MDT0000-mdc-ffff98bb44a52000 bb578dba-66a2-4424-b6c3-0338dda03657 5 4 UP osc lustre-OST0000-osc-ffff98bb44a52000 bb578dba-66a2-4424-b6c3-0338dda03657 5 5 UP osc lustre-OST0001-osc-ffff98bb44a52000 bb578dba-66a2-4424-b6c3-0338dda03657 5 rmmod: ERROR: Module mdd is in use unloading modules via unload_modules_local on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing unload_modules_local oleg324-server: lustre_rmmod: wait 40s for 18 OBD devices oleg324-server: lustre_rmmod: OBD devices after 60s: oleg324-server: 0 UP osd-zfs lustre-MDT0000-osd lustre-MDT0000-osd_UUID 10 oleg324-server: 1 UP mgs MGS MGS 7 oleg324-server: 2 UP mgc MGC192.168.203.124@tcp c2a4ea9e-b180-40d2-b7dd-3b0d82230c17 5 oleg324-server: 3 UP mds MDS MDS_uuid 3 oleg324-server: 4 UP lod lustre-MDT0000-mdtlov lustre-MDT0000-mdtlov_UUID 4 oleg324-server: 5 UP mdt lustre-MDT0000 lustre-MDT0000_UUID 11 oleg324-server: 6 UP mdd lustre-MDD0000 lustre-MDD0000_UUID 4 oleg324-server: 7 UP qmt lustre-QMT0000 lustre-QMT0000_UUID 4 oleg324-server: 8 UP osp lustre-OST0000-osc-MDT0000 lustre-MDT0000-mdtlov_UUID 5 oleg324-server: 9 UP osp lustre-OST0001-osc-MDT0000 lustre-MDT0000-mdtlov_UUID 5 oleg324-server: 10 UP lwp lustre-MDT0000-lwp-MDT0000 lustre-MDT0000-lwp-MDT0000_UUID 5 oleg324-server: 11 UP osd-zfs lustre-OST0000-osd lustre-OST0000-osd_UUID 5 oleg324-server: 12 UP ost OSS OSS_uuid 3 oleg324-server: 13 UP obdfilter lustre-OST0000 lustre-OST0000_UUID 5 oleg324-server: 14 UP lwp lustre-MDT0000-lwp-OST0000 lustre-MDT0000-lwp-OST0000_UUID 5 oleg324-server: 15 UP osd-zfs lustre-OST0001-osd lustre-OST0001-osd_UUID 5 oleg324-server: 16 UP obdfilter lustre-OST0001 lustre-OST0001_UUID 5 oleg324-server: 17 UP lwp lustre-MDT0000-lwp-OST0001 lustre-MDT0000-lwp-OST0001_UUID 5 oleg324-server: oleg324-server: rmmod: ERROR: Module mdd is in use pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 sanity-lnet test_304: @@@@@@ FAIL: Failed to unload modules before test execution Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7496:error() = /home/green/git/lustre-release/lustre/tests/sanity-lnet.sh:6383:test_304() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7873:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7936:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7750:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-lnet.sh:6444:main() Dumping lctl log to /tmp/testlogs//sanity-lnet.test_304.*.1788972293.log rsync: chown "/tmp/testlogs/.sanity-lnet.test_304.debug_log.oleg324-server.1788972293.log.Hdqebv" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-lnet.test_304.dmesg.oleg324-server.1788972293.log.m5Rzhn" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3] FAIL 304 (136s) == sanity-lnet test 305: Resolve hostname before lnetctl ping ========================================================== 12:45:04 (1788972304) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure --- configure: - lnet: errno: -16 descr: ! "LNet configure error: Object busy" ... sanity-lnet test_305: @@@@@@ FAIL: lnetctl lnet unconfigure failed 240 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7496:error() = /home/green/git/lustre-release/lustre/tests/sanity-lnet.sh:147:reinit_dlc() = /home/green/git/lustre-release/lustre/tests/sanity-lnet.sh:6449:test_305() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7873:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7936:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7750:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-lnet.sh:6465:main() Dumping lctl log to /tmp/testlogs//sanity-lnet.test_305.*.1788972308.log rsync: chown "/tmp/testlogs/.sanity-lnet.test_305.debug_log.oleg324-server.1788972308.log.XI3qHe" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-lnet.test_305.dmesg.oleg324-server.1788972308.log.GHhVlX" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3] FAIL 305 (8s) == sanity-lnet test 310: Set timeout and verify ========== 12:45:21 (1788972321) Cleaning up LNet lustre_rmmod: wait 41s for 6 OBD devices lustre_rmmod: OBD devices after 60s: 0 UP mgc MGC192.168.203.124@tcp 94d249cb-8ce8-4694-9530-465f9f595f02 5 1 UP lov lustre-clilov-ffff98bb44a52000 bb578dba-66a2-4424-b6c3-0338dda03657 4 2 UP lmv lustre-clilmv-ffff98bb44a52000 bb578dba-66a2-4424-b6c3-0338dda03657 5 3 UP mdc lustre-MDT0000-mdc-ffff98bb44a52000 bb578dba-66a2-4424-b6c3-0338dda03657 5 4 UP osc lustre-OST0000-osc-ffff98bb44a52000 bb578dba-66a2-4424-b6c3-0338dda03657 5 5 UP osc lustre-OST0001-osc-ffff98bb44a52000 bb578dba-66a2-4424-b6c3-0338dda03657 5 rmmod: ERROR: Module mdd is in use unloading modules via unload_modules_local on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing unload_modules_local oleg324-server: lustre_rmmod: wait 41s for 18 OBD devices oleg324-server: lustre_rmmod: OBD devices after 60s: oleg324-server: 0 UP osd-zfs lustre-MDT0000-osd lustre-MDT0000-osd_UUID 10 oleg324-server: 1 UP mgs MGS MGS 7 oleg324-server: 2 UP mgc MGC192.168.203.124@tcp c2a4ea9e-b180-40d2-b7dd-3b0d82230c17 5 oleg324-server: 3 UP mds MDS MDS_uuid 3 oleg324-server: 4 UP lod lustre-MDT0000-mdtlov lustre-MDT0000-mdtlov_UUID 4 oleg324-server: 5 UP mdt lustre-MDT0000 lustre-MDT0000_UUID 11 oleg324-server: 6 UP mdd lustre-MDD0000 lustre-MDD0000_UUID 4 oleg324-server: 7 UP qmt lustre-QMT0000 lustre-QMT0000_UUID 4 oleg324-server: 8 UP osp lustre-OST0000-osc-MDT0000 lustre-MDT0000-mdtlov_UUID 5 oleg324-server: 9 UP osp lustre-OST0001-osc-MDT0000 lustre-MDT0000-mdtlov_UUID 5 oleg324-server: 10 UP lwp lustre-MDT0000-lwp-MDT0000 lustre-MDT0000-lwp-MDT0000_UUID 5 oleg324-server: 11 UP osd-zfs lustre-OST0000-osd lustre-OST0000-osd_UUID 5 oleg324-server: 12 UP ost OSS OSS_uuid 3 oleg324-server: 13 UP obdfilter lustre-OST0000 lustre-OST0000_UUID 5 oleg324-server: 14 UP lwp lustre-MDT0000-lwp-OST0000 lustre-MDT0000-lwp-OST0000_UUID 5 oleg324-server: 15 UP osd-zfs lustre-OST0001-osd lustre-OST0001-osd_UUID 5 oleg324-server: 16 UP obdfilter lustre-OST0001 lustre-OST0001_UUID 5 oleg324-server: 17 UP lwp lustre-MDT0000-lwp-OST0001 lustre-MDT0000-lwp-OST0001_UUID 5 oleg324-server: oleg324-server: rmmod: ERROR: Module mdd is in use pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 sanity-lnet test_310: @@@@@@ FAIL: Failed to cleanup LNet Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7496:error() = /home/green/git/lustre-release/lustre/tests/sanity-lnet.sh:6471:static_config() = /home/green/git/lustre-release/lustre/tests/sanity-lnet.sh:6493:test_310() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7873:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7936:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7750:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-lnet.sh:6509:main() Dumping lctl log to /tmp/testlogs//sanity-lnet.test_310.*.1788972459.log rsync: chown "/tmp/testlogs/.sanity-lnet.test_310.debug_log.oleg324-server.1788972459.log.fhefIP" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-lnet.test_310.dmesg.oleg324-server.1788972459.log.TdI2Zz" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3] FAIL 310 (141s) == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 12:47:50 (1788972470) SKIP: sanity-lnet test_311 Need kfi network type SKIP 311 (3s) == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 12:47:54 (1788972474) SKIP: sanity-lnet test_312 Need kfi network type SKIP 312 (2s) == sanity-lnet test 350: Check import --exec of ping/discover ========================================================== 12:47:57 (1788972477) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure --- configure: - lnet: errno: -16 descr: ! "LNet configure error: Object busy" ... sanity-lnet test_350: @@@@@@ FAIL: lnetctl lnet unconfigure failed 240 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7496:error() = /home/green/git/lustre-release/lustre/tests/sanity-lnet.sh:147:reinit_dlc() = /home/green/git/lustre-release/lustre/tests/sanity-lnet.sh:6565:test_350() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7873:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7936:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7750:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-lnet.sh:6603:main() Dumping lctl log to /tmp/testlogs//sanity-lnet.test_350.*.1788972480.log rsync: chown "/tmp/testlogs/.sanity-lnet.test_350.debug_log.oleg324-server.1788972480.log.wXxR0h" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-lnet.test_350.dmesg.oleg324-server.1788972480.log.KMo0vv" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3] FAIL 350 (6s) == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 12:48:09 (1788972489) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure --- configure: - lnet: errno: -16 descr: ! "LNet configure error: Object busy" ... sanity-lnet test_400: @@@@@@ FAIL: lnetctl lnet unconfigure failed 240 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7496:error() = /home/green/git/lustre-release/lustre/tests/sanity-lnet.sh:147:reinit_dlc() = /home/green/git/lustre-release/lustre/tests/sanity-lnet.sh:6662:test_400() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7873:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7936:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7750:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-lnet.sh:6670:main() Dumping lctl log to /tmp/testlogs//sanity-lnet.test_400.*.1788972493.log rsync: chown "/tmp/testlogs/.sanity-lnet.test_400.debug_log.oleg324-server.1788972493.log.h2VyCW" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-lnet.test_400.dmesg.oleg324-server.1788972493.log.ChcZVj" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3] FAIL 400 (6s) == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 12:48:23 (1788972503) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure --- configure: - lnet: errno: -16 descr: ! "LNet configure error: Object busy" ... sanity-lnet test_401: @@@@@@ FAIL: lnetctl lnet unconfigure failed 240 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7496:error() = /home/green/git/lustre-release/lustre/tests/sanity-lnet.sh:147:reinit_dlc() = /home/green/git/lustre-release/lustre/tests/sanity-lnet.sh:6673:test_401() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7873:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7936:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7750:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-lnet.sh:6688:main() Dumping lctl log to /tmp/testlogs//sanity-lnet.test_401.*.1788972507.log rsync: chown "/tmp/testlogs/.sanity-lnet.test_401.debug_log.oleg324-server.1788972507.log.HNgNd4" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-lnet.test_401.dmesg.oleg324-server.1788972507.log.jZcEzj" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3] FAIL 401 (7s) == sanity-lnet test 402: Destination net rule should not panic ========================================================== 12:48:39 (1788972519) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure --- configure: - lnet: errno: -16 descr: ! "LNet configure error: Object busy" ... sanity-lnet test_402: @@@@@@ FAIL: lnetctl lnet unconfigure failed 240 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7496:error() = /home/green/git/lustre-release/lustre/tests/sanity-lnet.sh:147:reinit_dlc() = /home/green/git/lustre-release/lustre/tests/sanity-lnet.sh:6691:test_402() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7873:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7936:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7750:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-lnet.sh:6701:main() Dumping lctl log to /tmp/testlogs//sanity-lnet.test_402.*.1788972522.log rsync: chown "/tmp/testlogs/.sanity-lnet.test_402.debug_log.oleg324-server.1788972522.log.o24ADj" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-lnet.test_402.dmesg.oleg324-server.1788972522.log.BPljpM" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3] FAIL 402 (6s) == sanity-lnet test 403: Add and delete udsp net and nid rules ========================================================== 12:48:53 (1788972533) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure --- configure: - lnet: errno: -16 descr: ! "LNet configure error: Object busy" ... sanity-lnet test_403: @@@@@@ FAIL: lnetctl lnet unconfigure failed 240 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7496:error() = /home/green/git/lustre-release/lustre/tests/sanity-lnet.sh:147:reinit_dlc() = /home/green/git/lustre-release/lustre/tests/sanity-lnet.sh:6707:test_403() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7873:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7936:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7750:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-lnet.sh:6745:main() Dumping lctl log to /tmp/testlogs//sanity-lnet.test_403.*.1788972537.log rsync: chown "/tmp/testlogs/.sanity-lnet.test_403.debug_log.oleg324-server.1788972537.log.b23KoH" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-lnet.test_403.dmesg.oleg324-server.1788972537.log.jB3mF8" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3] FAIL 403 (6s) == sanity-lnet test 404: Check udsp src/dst net rule ===== 12:49:06 (1788972546) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure --- configure: - lnet: errno: -16 descr: ! "LNet configure error: Object busy" ... sanity-lnet test_404: @@@@@@ FAIL: lnetctl lnet unconfigure failed 240 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7496:error() = /home/green/git/lustre-release/lustre/tests/sanity-lnet.sh:147:reinit_dlc() = /home/green/git/lustre-release/lustre/tests/sanity-lnet.sh:6815:test_404() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7873:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7936:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7750:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-lnet.sh:6839:main() Dumping lctl log to /tmp/testlogs//sanity-lnet.test_404.*.1788972550.log rsync: chown "/tmp/testlogs/.sanity-lnet.test_404.debug_log.oleg324-server.1788972550.log.Z31EAv" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-lnet.test_404.dmesg.oleg324-server.1788972550.log.bGwy5f" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3] FAIL 404 (7s) SKIP: sanity-lnet test_410 skipping ALWAYS excluded test 410 == sanity-lnet test 450: Check import of multiple interfaces ========================================================== 12:49:22 (1788972562) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure --- configure: - lnet: errno: -16 descr: ! "LNet configure error: Object busy" ... sanity-lnet test_450: @@@@@@ FAIL: lnetctl lnet unconfigure failed 240 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7496:error() = /home/green/git/lustre-release/lustre/tests/sanity-lnet.sh:147:reinit_dlc() = /home/green/git/lustre-release/lustre/tests/sanity-lnet.sh:6856:test_450() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7873:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7936:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7750:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-lnet.sh:6884:main() Dumping lctl log to /tmp/testlogs//sanity-lnet.test_450.*.1788972566.log rsync: chown "/tmp/testlogs/.sanity-lnet.test_450.debug_log.oleg324-server.1788972566.log.kmXRNR" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-lnet.test_450.dmesg.oleg324-server.1788972566.log.6QztPk" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3] FAIL 450 (8s) == sanity-lnet test 475: per-operation latency stats populate and reset ========================================================== 12:49:37 (1788972577) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure --- configure: - lnet: errno: -16 descr: ! "LNet configure error: Object busy" ... sanity-lnet test_475: @@@@@@ FAIL: lnetctl lnet unconfigure failed 240 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7496:error() = /home/green/git/lustre-release/lustre/tests/sanity-lnet.sh:147:reinit_dlc() = /home/green/git/lustre-release/lustre/tests/sanity-lnet.sh:6896:test_475() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7873:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7936:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7750:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-lnet.sh:6936:main() Dumping lctl log to /tmp/testlogs//sanity-lnet.test_475.*.1788972583.log rsync: chown "/tmp/testlogs/.sanity-lnet.test_475.debug_log.oleg324-server.1788972583.log.mcG4aO" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-lnet.test_475.dmesg.oleg324-server.1788972583.log.uqIGXy" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3] FAIL 475 (9s) == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 12:49:53 (1788972593) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure --- configure: - lnet: errno: -16 descr: ! "LNet configure error: Object busy" ... sanity-lnet test_500: @@@@@@ FAIL: lnetctl lnet unconfigure failed 240 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7496:error() = /home/green/git/lustre-release/lustre/tests/sanity-lnet.sh:147:reinit_dlc() = /home/green/git/lustre-release/lustre/tests/sanity-lnet.sh:6939:test_500() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7873:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7936:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7750:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-lnet.sh:6963:main() Dumping lctl log to /tmp/testlogs//sanity-lnet.test_500.*.1788972597.log rsync: chown "/tmp/testlogs/.sanity-lnet.test_500.debug_log.oleg324-server.1788972597.log.Ho4JXm" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-lnet.test_500.dmesg.oleg324-server.1788972597.log.fx3OOP" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3] FAIL 500 (7s) == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 12:50:06 (1788972606) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure --- configure: - lnet: errno: -16 descr: ! "LNet configure error: Object busy" ... sanity-lnet test_501: @@@@@@ FAIL: lnetctl lnet unconfigure failed 240 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7496:error() = /home/green/git/lustre-release/lustre/tests/sanity-lnet.sh:147:reinit_dlc() = /home/green/git/lustre-release/lustre/tests/sanity-lnet.sh:7013:test_501() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7873:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7936:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7750:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-lnet.sh:7030:main() Dumping lctl log to /tmp/testlogs//sanity-lnet.test_501.*.1788972610.log rsync: chown "/tmp/testlogs/.sanity-lnet.test_501.debug_log.oleg324-server.1788972610.log.m4QUs8" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-lnet.test_501.dmesg.oleg324-server.1788972610.log.3lQ3UK" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3] FAIL 501 (7s) == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 12:50:20 (1788972620) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure --- configure: - lnet: errno: -16 descr: ! "LNet configure error: Object busy" ... sanity-lnet test_502: @@@@@@ FAIL: lnetctl lnet unconfigure failed 240 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7496:error() = /home/green/git/lustre-release/lustre/tests/sanity-lnet.sh:147:reinit_dlc() = /home/green/git/lustre-release/lustre/tests/sanity-lnet.sh:7033:test_502() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7873:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7936:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7750:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-lnet.sh:7050:main() Dumping lctl log to /tmp/testlogs//sanity-lnet.test_502.*.1788972624.log rsync: chown "/tmp/testlogs/.sanity-lnet.test_502.debug_log.oleg324-server.1788972624.log.57iA9S" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-lnet.test_502.dmesg.oleg324-server.1788972624.log.tmCCl3" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3] FAIL 502 (8s) == sanity-lnet test 525: keep reachable peer on local net delete ========================================================== 12:50:36 (1788972636) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure --- configure: - lnet: errno: -16 descr: ! "LNet configure error: Object busy" ... sanity-lnet test_525: @@@@@@ FAIL: lnetctl lnet unconfigure failed 240 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7496:error() = /home/green/git/lustre-release/lustre/tests/sanity-lnet.sh:147:reinit_dlc() = /home/green/git/lustre-release/lustre/tests/sanity-lnet.sh:7053:test_525() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7873:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7936:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7750:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-lnet.sh:7075:main() Dumping lctl log to /tmp/testlogs//sanity-lnet.test_525.*.1788972640.log rsync: chown "/tmp/testlogs/.sanity-lnet.test_525.debug_log.oleg324-server.1788972640.log.j9w1E5" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-lnet.test_525.dmesg.oleg324-server.1788972640.log.ayzOUX" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3] FAIL 525 (6s) == sanity-lnet test 550: DD-off: keep route up when gw reachable via another local NI ========================================================== 12:50:50 (1788972650) SKIP: sanity-lnet test_550 Need at least 2 local interfaces for multi-rail SKIP 550 (3s) == sanity-lnet test 600: lnet_legacy2yaml: basic ip2nets conversion ========================================================== 12:50:53 (1788972653) Expected: ip2nets: - net-spec: tcp0 interfaces: 0: eth0 1: eth1 ip-range: 0: 10.0.0.* - net-spec: o2ib0 ip-range: 0: 192.168.[1-2].* Actual: ip2nets: - net-spec: tcp0 interfaces: 0: eth0 1: eth1 ip-range: 0: 10.0.0.* - net-spec: o2ib0 ip-range: 0: 192.168.[1-2].* PASS 600 (7s) == sanity-lnet test 601: lnet_legacy2yaml: basic routes conversion ========================================================== 12:51:00 (1788972660) Expected: route: - net: tcp0 gateway: 10.0.0.1@o2ib0 hop: 1 Actual: route: - net: tcp0 gateway: 10.0.0.1@o2ib0 hop: 1 PASS 601 (7s) == sanity-lnet test 602: lnet_legacy2yaml: routes with priority ========================================================== 12:51:07 (1788972667) Expected: route: - net: tcp0 gateway: 10.0.0.1@o2ib0 hop: 2 priority: 7 Actual: route: - net: tcp0 gateway: 10.0.0.1@o2ib0 hop: 2 priority: 7 PASS 602 (7s) == sanity-lnet test 603: lnet_legacy2yaml: default hop flag ========================================================== 12:51:14 (1788972674) Expected: route: - net: tcp0 gateway: 10.0.0.1@o2ib0 hop: 1 Actual: route: - net: tcp0 gateway: 10.0.0.1@o2ib0 hop: 1 PASS 603 (6s) == sanity-lnet test 604: lnet_legacy2yaml: default hop with explicit value ========================================================== 12:51:20 (1788972680) Expected: route: - net: tcp0 gateway: 10.0.0.1@o2ib0 hop: 3 Actual: route: - net: tcp0 gateway: 10.0.0.1@o2ib0 hop: 3 PASS 604 (7s) == sanity-lnet test 605: lnet_legacy2yaml: quote leading star in IP pattern ========================================================== 12:51:27 (1788972687) Expected: ip2nets: - net-spec: tcp ip-range: 0: "*.*.*.*" Actual: ip2nets: - net-spec: tcp ip-range: 0: "*.*.*.*" PASS 605 (9s) == sanity-lnet test 606: lnet_legacy2yaml: file inputs === 12:51:36 (1788972696) Expected: ip2nets: - net-spec: tcp0 ip-range: 0: 10.1.0.* route: - net: tcp0 gateway: 10.1.0.1@o2ib0 hop: 1 priority: 9 Actual: ip2nets: - net-spec: tcp0 ip-range: 0: 10.1.0.* route: - net: tcp0 gateway: 10.1.0.1@o2ib0 hop: 1 priority: 9 PASS 606 (12s) == sanity-lnet test 607: lnet_legacy2yaml: options line with combined ip2nets and routes ========================================================== 12:51:48 (1788972708) Expected: ip2nets: - net-spec: tcp0 ip-range: 0: 10.0.0.* route: - net: tcp0 gateway: 10.0.0.2@o2ib0 hop: 1 Actual: ip2nets: - net-spec: tcp0 ip-range: 0: 10.0.0.* route: - net: tcp0 gateway: 10.0.0.2@o2ib0 hop: 1 PASS 607 (7s) == sanity-lnet test 608: lnet_legacy2yaml: error on invalid rule (missing IP pattern) ========================================================== 12:51:56 (1788972716) PASS 608 (7s) == sanity-lnet test 609: lnet_legacy2yaml: error on no input ========================================================== 12:52:02 (1788972722) PASS 609 (7s) == sanity-lnet test 610: lnet_legacy2yaml: multiple files and inline, check order ========================================================== 12:52:09 (1788972729) Expected: ip2nets: - net-spec: tcp0 ip-range: 0: 10.0.0.* - net-spec: o2ib0 ip-range: 0: 192.168.0.* route: - net: tcp0 gateway: 10.0.0.1@o2ib0 hop: 1 - net: o2ib0 gateway: 192.168.0.1@tcp0 hop: 1 Actual: ip2nets: - net-spec: tcp0 ip-range: 0: 10.0.0.* - net-spec: o2ib0 ip-range: 0: 192.168.0.* route: - net: tcp0 gateway: 10.0.0.1@o2ib0 hop: 1 - net: o2ib0 gateway: 192.168.0.1@tcp0 hop: 1 PASS 610 (8s) == sanity-lnet test 611: lnet_legacy2yaml: invalid hopcount ========================================================== 12:52:18 (1788972738) PASS 611 (9s) == sanity-lnet test 612: lnet_legacy2yaml: malformed IP pattern ========================================================== 12:52:27 (1788972747) PASS 612 (10s) == sanity-lnet test 613: lnet_legacy2yaml: multiple IP patterns in single rule ========================================================== 12:52:37 (1788972757) Expected: ip2nets: - net-spec: tcp0 ip-range: 0: 10.0.0.* 1: 10.0.1.* Actual: ip2nets: - net-spec: tcp0 ip-range: 0: 10.0.0.* 1: 10.0.1.* PASS 613 (10s) == sanity-lnet test 614: lnet_legacy2yaml: interfaces enumeration with multiple IP ranges ========================================================== 12:52:47 (1788972767) Expected: ip2nets: - net-spec: tcp0 interfaces: 0: eth0 1: eth1 ip-range: 0: 10.0.0.* 1: 10.0.1.* Actual: ip2nets: - net-spec: tcp0 interfaces: 0: eth0 1: eth1 ip-range: 0: 10.0.0.* 1: 10.0.1.* PASS 614 (8s) == sanity-lnet test 615: lnet_legacy2yaml: modprobe configuration file input ========================================================== 12:52:55 (1788972775) Expected: ip2nets: - net-spec: tcp0 interfaces: 0: eth0 ip-range: 0: 10.1.0.* - net-spec: o2ib0 ip-range: 0: 192.168.1.* route: - net: tcp0 gateway: 10.1.0.1@o2ib0 hop: 1 priority: 9 Actual: ip2nets: - net-spec: tcp0 interfaces: 0: eth0 ip-range: 0: 10.1.0.* - net-spec: o2ib0 ip-range: 0: 192.168.1.* route: - net: tcp0 gateway: 10.1.0.1@o2ib0 hop: 1 priority: 9 PASS 615 (8s) == sanity-lnet test 616: lnet_legacy2yaml: combining modprobe file with explicit ip2nets file ========================================================== 12:53:03 (1788972783) Expected: ip2nets: - net-spec: tcp0 ip-range: 0: 10.0.0.* - net-spec: o2ib0 ip-range: 0: 192.168.0.* route: - net: tcp0 gateway: 10.0.0.1@o2ib0 hop: 1 Actual: ip2nets: - net-spec: tcp0 ip-range: 0: 10.0.0.* - net-spec: o2ib0 ip-range: 0: 192.168.0.* route: - net: tcp0 gateway: 10.0.0.1@o2ib0 hop: 1 PASS 616 (8s) == sanity-lnet test 617: lnet_legacy2yaml: round-trip tunables from LNet to modprobe to YAML ========================================================== 12:53:11 (1788972791) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure --- configure: - lnet: errno: -16 descr: ! "LNet configure error: Object busy" ... sanity-lnet test_617: @@@@@@ FAIL: lnetctl lnet unconfigure failed 240 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7496:error() = /home/green/git/lustre-release/lustre/tests/sanity-lnet.sh:147:reinit_dlc() = /home/green/git/lustre-release/lustre/tests/sanity-lnet.sh:7523:test_617() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7873:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7936:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7750:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-lnet.sh:7574:main() Dumping lctl log to /tmp/testlogs//sanity-lnet.test_617.*.1788972795.log rsync: chown "/tmp/testlogs/.sanity-lnet.test_617.debug_log.oleg324-server.1788972795.log.NaxyEr" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-lnet.test_617.dmesg.oleg324-server.1788972795.log.PTURXg" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3] FAIL 617 (8s) == sanity-lnet test 618: lnet_legacy2yaml: alternative route format ========================================================== 12:53:27 (1788972807) Expected: route: - net: tcp7 gateway: 10.3.3.[6-12]@tcp hop: 8 priority: 20 Actual: route: - net: tcp7 gateway: 10.3.3.[6-12]@tcp hop: 8 priority: 20 PASS 618 (9s) == sanity-lnet test 619: lnet_legacy2yaml: multiple alternative format routes from file ========================================================== 12:53:36 (1788972816) Expected: route: - net: tcp1 gateway: 10.1.1.2@tcp0 priority: 3 - net: tcp4 gateway: 10.3.3.4@tcp - net: tcp6 gateway: 10.3.3.6@tcp hop: 2 priority: 5 - net: tcp7 gateway: 10.3.3.[6-12]@tcp hop: 8 priority: 20 Actual: route: - net: tcp1 gateway: 10.1.1.2@tcp0 priority: 3 - net: tcp4 gateway: 10.3.3.4@tcp - net: tcp6 gateway: 10.3.3.6@tcp hop: 2 priority: 5 - net: tcp7 gateway: 10.3.3.[6-12]@tcp hop: 8 priority: 20 PASS 619 (11s) == sanity-lnet test 620: lnet_legacy2yaml: mixed legacy and alternative route formats ========================================================== 12:53:47 (1788972827) Expected: route: - net: tcp1 gateway: 10.1.1.2@tcp0 priority: 3 - net: tcp0 gateway: 10.2.2.3@tcp1 hop: 1 - net: tcp6 gateway: 10.3.3.6@tcp hop: 2 priority: 5 - net: tcp2 gateway: 10.4.4.5@tcp0 hop: 2 priority: 7 Actual: route: - net: tcp1 gateway: 10.1.1.2@tcp0 priority: 3 - net: tcp0 gateway: 10.2.2.3@tcp1 hop: 1 - net: tcp6 gateway: 10.3.3.6@tcp hop: 2 priority: 5 - net: tcp2 gateway: 10.4.4.5@tcp0 hop: 2 priority: 7 PASS 620 (10s) == sanity-lnet test 621: lnet_legacy2yaml: alternative format with invalid hopcount ========================================================== 12:53:57 (1788972837) PASS 621 (12s) == sanity-lnet test 622: lnet_legacy2yaml: alternative format with default-hop flag ========================================================== 12:54:09 (1788972849) Expected: route: - net: tcp5 gateway: 10.5.5.5@tcp0 hop: 3 Actual: route: - net: tcp5 gateway: 10.5.5.5@tcp0 hop: 3 PASS 622 (10s) == sanity-lnet test 623: lnet_legacy2yaml: basic networks conversion ========================================================== 12:54:20 (1788972860) Expected: net: - net type: tcp Actual: net: - net type: tcp PASS 623 (12s) == sanity-lnet test 624: lnet_legacy2yaml: networks with interfaces ========================================================== 12:54:32 (1788972872) Expected: net: - net type: tcp0 local NI(s): - interfaces: 0: eth0 Actual: net: - net type: tcp0 local NI(s): - interfaces: 0: eth0 PASS 624 (13s) == sanity-lnet test 625: lnet_legacy2yaml: multiple networks ========================================================== 12:54:45 (1788972885) Expected: net: - net type: tcp0 local NI(s): - interfaces: 0: eth0 - net type: o2ib local NI(s): - interfaces: 0: ib0 Actual: net: - net type: tcp0 local NI(s): - interfaces: 0: eth0 - net type: o2ib local NI(s): - interfaces: 0: ib0 PASS 625 (8s) == sanity-lnet test 626: lnet_legacy2yaml: mixed networks, ip2nets, and routes ========================================================== 12:54:54 (1788972894) Expected: net: - net type: tcp0 local NI(s): - interfaces: 0: eth0 - net type: o2ib local NI(s): - interfaces: 0: ib0 ip2nets: - net-spec: tcp1 interfaces: 0: eth1 ip-range: 0: 192.168.1.* route: - net: tcp0 gateway: 10.0.0.1@o2ib0 hop: 1 Actual: net: - net type: tcp0 local NI(s): - interfaces: 0: eth0 - net type: o2ib local NI(s): - interfaces: 0: ib0 ip2nets: - net-spec: tcp1 interfaces: 0: eth1 ip-range: 0: 192.168.1.* route: - net: tcp0 gateway: 10.0.0.1@o2ib0 hop: 1 PASS 626 (10s) == sanity-lnet test 627: lnet_legacy2yaml: invalid networks format ========================================================== 12:55:04 (1788972904) PASS 627 (10s) == sanity-lnet test 628: lnet_legacy2yaml: missing delimiter error ========================================================== 12:55:14 (1788972914) PASS 628 (8s) == sanity-lnet test 629: lnet_legacy2yaml: multiple gateways error ========================================================== 12:55:22 (1788972922) PASS 629 (9s) == sanity-lnet test complete, duration 3515 sec ========== 12:55:31 (1788972931) sanity-lnet: FAIL: test_260 Failed to cleanup Modules sanity-lnet: FAIL: test_265 lnetctl lnet unconfigure failed 240 sanity-lnet: FAIL: test_266 lnetctl lnet unconfigure failed 240 sanity-lnet: FAIL: test_270 Failed to unload modules before test execution sanity-lnet: FAIL: test_280 Failed to unload modules before test execution sanity-lnet: FAIL: test_290 Failed to unload modules before test execution sanity-lnet: FAIL: test_291 lnetctl lnet unconfigure failed 240 sanity-lnet: FAIL: test_300 test_300 failed with 2 sanity-lnet: FAIL: test_301 lnetctl lnet unconfigure failed 240 sanity-lnet: FAIL: test_302 lnetctl lnet unconfigure failed 240 sanity-lnet: FAIL: test_303 Failed to cleanup before test execution sanity-lnet: FAIL: test_304 Failed to unload modules before test execution sanity-lnet: FAIL: test_305 lnetctl lnet unconfigure failed 240 sanity-lnet: FAIL: test_310 Failed to cleanup LNet sanity-lnet: FAIL: test_350 lnetctl lnet unconfigure failed 240 sanity-lnet: FAIL: test_400 lnetctl lnet unconfigure failed 240 sanity-lnet: FAIL: test_401 lnetctl lnet unconfigure failed 240 sanity-lnet: FAIL: test_402 lnetctl lnet unconfigure failed 240 sanity-lnet: FAIL: test_403 lnetctl lnet unconfigure failed 240 sanity-lnet: FAIL: test_404 lnetctl lnet unconfigure failed 240 sanity-lnet: FAIL: test_450 lnetctl lnet unconfigure failed 240 sanity-lnet: FAIL: test_475 lnetctl lnet unconfigure failed 240 sanity-lnet: FAIL: test_500 lnetctl lnet unconfigure failed 240 sanity-lnet: FAIL: test_501 lnetctl lnet unconfigure failed 240 sanity-lnet: FAIL: test_502 lnetctl lnet unconfigure failed 240 sanity-lnet: FAIL: test_525 lnetctl lnet unconfigure failed 240 sanity-lnet: FAIL: test_617 lnetctl lnet unconfigure failed 240 Cleaning up LNet