-----============= acceptance-small: sanity-lnet ============----- Thu Apr 4 20:05:21 EDT 2024 Stopping clients: oleg133-client.virtnet /mnt/lustre (opts:) Stopping client oleg133-client.virtnet /mnt/lustre opts: Stopping clients: oleg133-client.virtnet /mnt/lustre2 (opts:) Stopping /mnt/lustre-mds1 (opts:-f) on oleg133-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg133-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg133-server unloading modules on: 'oleg133-server' oleg133-server: oleg133-server.virtnet: executing unload_modules_local modules unloaded. Writer error: failed to resolve Netlink family id Stopping clients: oleg133-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg133-client.virtnet /mnt/lustre2 (opts:-f) LNET unconfigure error 22: (null) unloading modules on: 'oleg133-server' oleg133-server: oleg133-server.virtnet: executing unload_modules_local oleg133-server: LNET unconfigure error 22: (null) modules unloaded. ip netns exec test_ns ip addr add 10.1.2.3/31 dev test1pg ip netns exec test_ns ip link set test1pg up Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.201.33@tcp status: up interfaces: 0: eth0 1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0: mtu 9000 qdisc pfifo_fast state UP group default qlen 1000 link/ether 56:60:59:37:fa:54 brd ff:ff:ff:ff:ff:ff inet 192.168.201.33/24 brd 192.168.201.255 scope global dynamic eth0 valid_lft 59901sec preferred_lft 59901sec inet6 fe80::5460:59ff:fe37:fa54/64 scope link noprefixroute valid_lft forever preferred_lft forever 3: test1pl@if2: mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 52:1c:08:20:ee:87 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet6 fe80::501c:8ff:fe20:ee87/64 scope link tentative valid_lft forever preferred_lft forever excepting tests: 253 254 Cleaning up LNet Writer error: failed to resolve Netlink family id unloading modules on: 'oleg133-server' oleg133-server: oleg133-server.virtnet: executing unload_modules_local oleg133-server: LNET unconfigure error 22: (null) modules unloaded. error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg133-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 0: Export empty config, import the config, compare ========================================================== 20:05:57 (1712275557) Loading LNet and configuring DLC Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import Expected: global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 PASS 0 (3s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg133-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 1: Add peer with single nid (tcp) ==== 20:06:01 (1712275561) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp Expected: peer: - primary nid: 1.1.1.1@tcp Multi-Rail: True peer ni: - nid: 1.1.1.1@tcp global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: peer: - primary nid: 1.1.1.1@tcp Multi-Rail: True peer ni: - nid: 1.1.1.1@tcp global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 PASS 1 (2s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg133-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 2: Add peer with single nid (o2ib) === 20:06:05 (1712275565) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 2.2.2.2@o2ib Expected: peer: - primary nid: 2.2.2.2@o2ib Multi-Rail: True peer ni: - nid: 2.2.2.2@o2ib global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: peer: - primary nid: 2.2.2.2@o2ib Multi-Rail: True peer ni: - nid: 2.2.2.2@o2ib global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 PASS 2 (1s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg133-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 3: Add peer with tcp primary o2ib secondary ========================================================== 20:06:08 (1712275568) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 3.3.3.3@tcp --nid 3.3.3.3@o2ib Expected: peer: - primary nid: 3.3.3.3@tcp Multi-Rail: True peer ni: - nid: 3.3.3.3@tcp - nid: 3.3.3.3@o2ib global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: peer: - primary nid: 3.3.3.3@tcp Multi-Rail: True peer ni: - nid: 3.3.3.3@tcp - nid: 3.3.3.3@o2ib global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 PASS 3 (1s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg133-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 4: Add peer with nidrange (tcp) ====== 20:06:11 (1712275571) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure Add peer with nidrange (tcp) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-3]@tcp Expected: peer: - primary nid: 4.4.4.4@tcp Multi-Rail: True peer ni: - nid: 4.4.4.4@tcp - nid: 4.4.4.1@tcp - nid: 4.4.4.2@tcp - nid: 4.4.4.3@tcp global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: peer: - primary nid: 4.4.4.4@tcp Multi-Rail: True peer ni: - nid: 4.4.4.4@tcp - nid: 4.4.4.1@tcp - nid: 4.4.4.2@tcp - nid: 4.4.4.3@tcp global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Add peer with nidrange that overlaps primary nid (tcp) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 4.4.4.4@tcp --nid 4.4.4.[1-4]@tcp Expected: peer: - primary nid: 4.4.4.4@tcp Multi-Rail: True peer ni: - nid: 4.4.4.4@tcp - nid: 4.4.4.1@tcp - nid: 4.4.4.2@tcp - nid: 4.4.4.3@tcp global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: peer: - primary nid: 4.4.4.4@tcp Multi-Rail: True peer ni: - nid: 4.4.4.4@tcp - nid: 4.4.4.1@tcp - nid: 4.4.4.2@tcp - nid: 4.4.4.3@tcp global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 PASS 4 (1s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg133-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 5: Add peer with nidrange (o2ib) ===== 20:06:14 (1712275574) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure Add peer with nidrange (o2ib) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib Expected: peer: - primary nid: 5.5.5.5@o2ib Multi-Rail: True peer ni: - nid: 5.5.5.5@o2ib - nid: 5.5.5.1@o2ib - nid: 5.5.5.2@o2ib - nid: 5.5.5.3@o2ib - nid: 5.5.5.4@o2ib global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: peer: - primary nid: 5.5.5.5@o2ib Multi-Rail: True peer ni: - nid: 5.5.5.5@o2ib - nid: 5.5.5.1@o2ib - nid: 5.5.5.2@o2ib - nid: 5.5.5.3@o2ib - nid: 5.5.5.4@o2ib global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Add peer with nidranage that overlaps primary nid (o2ib) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1-4]@o2ib Expected: peer: - primary nid: 5.5.5.5@o2ib Multi-Rail: True peer ni: - nid: 5.5.5.5@o2ib - nid: 5.5.5.1@o2ib - nid: 5.5.5.2@o2ib - nid: 5.5.5.3@o2ib - nid: 5.5.5.4@o2ib global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: peer: - primary nid: 5.5.5.5@o2ib Multi-Rail: True peer ni: - nid: 5.5.5.5@o2ib - nid: 5.5.5.1@o2ib - nid: 5.5.5.2@o2ib - nid: 5.5.5.3@o2ib - nid: 5.5.5.4@o2ib global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Add peer with nidranage that contain , plus primary nid (o2ib) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 5.5.5.5@o2ib --nid 5.5.5.[1,2,3-4]@o2ib Expected: peer: - primary nid: 5.5.5.5@o2ib Multi-Rail: True peer ni: - nid: 5.5.5.5@o2ib - nid: 5.5.5.1@o2ib - nid: 5.5.5.2@o2ib - nid: 5.5.5.3@o2ib - nid: 5.5.5.4@o2ib global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: peer: - primary nid: 5.5.5.5@o2ib Multi-Rail: True peer ni: - nid: 5.5.5.5@o2ib - nid: 5.5.5.1@o2ib - nid: 5.5.5.2@o2ib - nid: 5.5.5.3@o2ib - nid: 5.5.5.4@o2ib global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 PASS 5 (1s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg133-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 6: Add peer with multiple nidranges == 20:06:17 (1712275577) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 6.6.6.6@tcp --nid 6.6.[6-7].[0-4/2]@tcp,6.6.[1-4/2].[0-6/3]@o2ib,[6-12/4]@gni,[6-12/4]@kfi Expected: peer: - primary nid: 6.6.6.6@tcp Multi-Rail: True peer ni: - nid: 6.6.6.6@tcp - nid: 6.6.6.0@tcp - nid: 6.6.6.2@tcp - nid: 6.6.6.4@tcp - nid: 6.6.7.0@tcp - nid: 6.6.7.2@tcp - nid: 6.6.7.4@tcp - nid: 6.6.1.0@o2ib - nid: 6.6.1.3@o2ib - nid: 6.6.1.6@o2ib - nid: 6.6.3.0@o2ib - nid: 6.6.3.3@o2ib - nid: 6.6.3.6@o2ib - nid: 6@gni - nid: 10@gni - nid: 6@kfi - nid: 10@kfi global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: peer: - primary nid: 6.6.6.6@tcp Multi-Rail: True peer ni: - nid: 6.6.6.6@tcp - nid: 6.6.6.0@tcp - nid: 6.6.6.2@tcp - nid: 6.6.6.4@tcp - nid: 6.6.7.0@tcp - nid: 6.6.7.2@tcp - nid: 6.6.7.4@tcp - nid: 6.6.1.0@o2ib - nid: 6.6.1.3@o2ib - nid: 6.6.1.6@o2ib - nid: 6.6.3.0@o2ib - nid: 6.6.3.3@o2ib - nid: 6.6.3.6@o2ib - nid: 6@gni - nid: 10@gni - nid: 6@kfi - nid: 10@kfi global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 PASS 6 (2s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg133-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 7: Various peer delete tests ========= 20:06:20 (1712275580) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure Delete peer with single nid (tcp) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp Expected: global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Delete peer with single nid (o2ib) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib Expected: global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Delete peer that has multiple nids (tcp) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[8-12]@tcp /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp Expected: global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Delete peer that has multiple nids (o2ib) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@o2ib --nid 7.7.7.[8-12]@o2ib /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@o2ib Expected: global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Delete peer that has both tcp and o2ib nids /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7.7.7.7@tcp --nid 7.7.7.[9-12]@tcp,7.7.7.[13-15]@o2ib /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7.7.7.7@tcp Expected: global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Delete peer with single nid (gni) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni Expected: global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Delete peer that has multiple nids (gni) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni Expected: global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Delete peer with single nid (kfi) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi Expected: global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Delete peer that has multiple nids (kfi) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@kfi --nid [8-12]@kfi /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@kfi Expected: global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Delete peer that has tcp, o2ib, gni and kfi nids /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 7@gni --nid [8-12]@gni,7.7.7.[1-4]@tcp,7.7.7.[5-9]@o2ib,[1-5]@kfi /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 7@gni Expected: global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Delete peer that contains nid equal to primary nid /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --nid 1.1.1.1@tcp,2.2.2.2@tcp,3.3.3.3@tcp,4.4.4.4@tcp Expected: global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 PASS 7 (3s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg133-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 8: Delete single secondary nid from peer (tcp) ========================================================== 20:06:25 (1712275585) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 8.8.8.8@tcp --nid 8.8.8.[10-15]@tcp /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 8.8.8.8@tcp --nid 8.8.8.13@tcp Expected: peer: - primary nid: 8.8.8.8@tcp Multi-Rail: True peer ni: - nid: 8.8.8.8@tcp - nid: 8.8.8.10@tcp - nid: 8.8.8.11@tcp - nid: 8.8.8.12@tcp - nid: 8.8.8.14@tcp - nid: 8.8.8.15@tcp global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: peer: - primary nid: 8.8.8.8@tcp Multi-Rail: True peer ni: - nid: 8.8.8.8@tcp - nid: 8.8.8.10@tcp - nid: 8.8.8.11@tcp - nid: 8.8.8.12@tcp - nid: 8.8.8.14@tcp - nid: 8.8.8.15@tcp global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 PASS 8 (1s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg133-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 9: Delete all secondary nids from peer (tcp) ========================================================== 20:06:28 (1712275588) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 9.9.9.9@tcp --nid 9.9.9.[11-16]@tcp Expected: peer: - primary nid: 9.9.9.9@tcp Multi-Rail: True peer ni: - nid: 9.9.9.9@tcp global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: peer: - primary nid: 9.9.9.9@tcp Multi-Rail: True peer ni: - nid: 9.9.9.9@tcp global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 PASS 9 (1s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg133-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 10: Delete single secondary nid from peer (o2ib) ========================================================== 20:06:31 (1712275591) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.10.10.10@tcp --nid 10.10.10.[12-16]@tcp /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 10.10.10.10@tcp --nid 10.10.10.14@tcp Expected: peer: - primary nid: 10.10.10.10@tcp Multi-Rail: True peer ni: - nid: 10.10.10.10@tcp - nid: 10.10.10.12@tcp - nid: 10.10.10.13@tcp - nid: 10.10.10.15@tcp - nid: 10.10.10.16@tcp global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: peer: - primary nid: 10.10.10.10@tcp Multi-Rail: True peer ni: - nid: 10.10.10.10@tcp - nid: 10.10.10.12@tcp - nid: 10.10.10.13@tcp - nid: 10.10.10.15@tcp - nid: 10.10.10.16@tcp global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 PASS 10 (1s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg133-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 11: Delete all secondary nids from peer (o2ib) ========================================================== 20:06:34 (1712275594) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 11.11.11.11@tcp --nid 11.11.11.[13-17]@tcp Expected: peer: - primary nid: 11.11.11.11@tcp Multi-Rail: True peer ni: - nid: 11.11.11.11@tcp global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: peer: - primary nid: 11.11.11.11@tcp Multi-Rail: True peer ni: - nid: 11.11.11.11@tcp global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 PASS 11 (1s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg133-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 12: Delete a secondary nid from peer (tcp and o2ib) ========================================================== 20:06:37 (1712275597) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 12.12.12.12@o2ib --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 12.12.12.12@o2ib --nid 13.15.13.13@o2ib,15.17.1.15@tcp Expected: peer: - primary nid: 12.12.12.12@o2ib Multi-Rail: True peer ni: - nid: 12.12.12.12@o2ib - nid: 13.13.13.13@o2ib - nid: 14.13.13.13@o2ib - nid: 14.15.13.13@o2ib - nid: 15.17.1.5@tcp - nid: 15.17.1.10@tcp - nid: 15.17.1.20@tcp global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: peer: - primary nid: 12.12.12.12@o2ib Multi-Rail: True peer ni: - nid: 12.12.12.12@o2ib - nid: 13.13.13.13@o2ib - nid: 14.13.13.13@o2ib - nid: 14.15.13.13@o2ib - nid: 15.17.1.5@tcp - nid: 15.17.1.10@tcp - nid: 15.17.1.20@tcp global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 PASS 12 (2s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg133-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 13: Delete all secondary nids from peer (tcp and o2ib) ========================================================== 20:06:40 (1712275600) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 13.13.13.13@o2ib --nid [14-15].[1-2/1].[1].[100-254/10]@tcp,14.14.[254].14@o2ib Expected: peer: - primary nid: 13.13.13.13@o2ib Multi-Rail: True peer ni: - nid: 13.13.13.13@o2ib global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: peer: - primary nid: 13.13.13.13@o2ib Multi-Rail: True peer ni: - nid: 13.13.13.13@o2ib global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 PASS 13 (2s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg133-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 14: import peer create/delete with single nid ========================================================== 20:06:43 (1712275603) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure Create single peer, single nid, using import Generating peer yaml for 1 peers with 0 secondary nids /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import Expected: peer: - primary nid: 1.1.1.1@tcp Multi-Rail: True peer ni: - nid: 1.1.1.1@tcp global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: peer: - primary nid: 1.1.1.1@tcp Multi-Rail: True peer ni: - nid: 1.1.1.1@tcp global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Delete single peer using import --del /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del Expected: global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 PASS 14 (2s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg133-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 15: import multi peer create/delete with single nid per peer ========================================================== 20:06:46 (1712275606) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure Create multiple peers, single nid per peer, using import Generating peer yaml for 5 peers with 0 secondary nids /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import Delete multiple peers, single nid per peer, using import --del /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del Expected: global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 PASS 15 (2s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg133-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 16: import peer create/delete with multiple nids ========================================================== 20:06:49 (1712275609) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure Create single peer, multiple nids, using import Generating peer yaml for 1 peers with 5 secondary nids /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import Delete single peer, multiple nids, using import --del /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del Expected: global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 PASS 16 (2s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg133-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 17: import multi peer create/delete with multiple nids ========================================================== 20:06:52 (1712275612) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure Create multiple peers, multiple nids per peer, using import Generating peer yaml for 5 peers with 7 secondary nids /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import Delete multiple peers, multiple nids per peer, using import --del /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del Expected: global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 PASS 17 (2s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg133-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 18a: Delete a subset of nids from a single peer using import --del ========================================================== 20:06:55 (1712275615) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure Import peer with 5 nids peer: - primary nid: 1.1.1.1@tcp Multi-Rail: True peer ni: - nid: 1.1.1.1@tcp - nid: 2.2.2.2@tcp - nid: 4.4.4.4@tcp - nid: 3.3.3.3@o2ib - nid: 5@gni /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import Delete three of the nids peer: - primary nid: 1.1.1.1@tcp Multi-Rail: True peer ni: - nid: 2.2.2.2@tcp - nid: 3.3.3.3@o2ib - nid: 5@gni /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del Check peer has expected nids remaining Expected: peer: - primary nid: 1.1.1.1@tcp Multi-Rail: True peer ni: - nid: 1.1.1.1@tcp - nid: 4.4.4.4@tcp global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: peer: - primary nid: 1.1.1.1@tcp Multi-Rail: True peer ni: - nid: 1.1.1.1@tcp - nid: 4.4.4.4@tcp global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 PASS 18a (2s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg133-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 18b: Delete multiple nids from multiple peers using import --del ========================================================== 20:06:58 (1712275618) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure Import two peers with 5 nids each peer: - primary nid: 1.1.1.1@tcp Multi-Rail: True peer ni: - nid: 1.1.1.1@tcp - nid: 2.2.2.2@tcp - nid: 4.4.4.4@tcp - nid: 3.3.3.3@o2ib - nid: 5@gni - primary nid: 6.6.6.6@o2ib Multi-Rail: True peer ni: - nid: 6.6.6.6@o2ib - nid: 7.7.7.7@tcp - nid: 8.8.8.8@tcp - nid: 9.9.9.9@tcp - nid: 10@gni /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import Delete three of the nids from each peer peer: - primary nid: 1.1.1.1@tcp Multi-Rail: True peer ni: - nid: 2.2.2.2@tcp - nid: 3.3.3.3@o2ib - nid: 5@gni - primary nid: 6.6.6.6@o2ib Multi-Rail: True peer ni: - nid: 7.7.7.7@tcp - nid: 8.8.8.8@tcp - nid: 10@gni /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import --del Check peers have expected nids remaining Expected: peer: - primary nid: 6.6.6.6@o2ib Multi-Rail: True peer ni: - nid: 6.6.6.6@o2ib - nid: 7.7.7.7@tcp - primary nid: 1.1.1.1@tcp Multi-Rail: True peer ni: - nid: 1.1.1.1@tcp - nid: 4.4.4.4@tcp global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: peer: - primary nid: 1.1.1.1@tcp Multi-Rail: True peer ni: - nid: 1.1.1.1@tcp - nid: 4.4.4.4@tcp - primary nid: 6.6.6.6@o2ib Multi-Rail: True peer ni: - nid: 6.6.6.6@o2ib - nid: 9.9.9.9@tcp global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 PASS 18b (2s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg133-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 19: Add peer with single nid (gni) === 20:07:01 (1712275621) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 19@gni Expected: peer: - primary nid: 19@gni Multi-Rail: True peer ni: - nid: 19@gni global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: peer: - primary nid: 19@gni Multi-Rail: True peer ni: - nid: 19@gni global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 PASS 19 (2s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg133-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 20: Add peer with gni primary and tcp, o2ib secondary ========================================================== 20:07:04 (1712275624) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 20@gni --nid 20.20.20.20@tcp,20.20.20.20@o2ib Expected: peer: - primary nid: 20@gni Multi-Rail: True peer ni: - nid: 20@gni - nid: 20.20.20.20@tcp - nid: 20.20.20.20@o2ib global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: peer: - primary nid: 20@gni Multi-Rail: True peer ni: - nid: 20@gni - nid: 20.20.20.20@tcp - nid: 20.20.20.20@o2ib global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 PASS 20 (2s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg133-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 21: Add peer with nidrange (gni) ===== 20:07:07 (1712275627) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure Add peer with nidrange (gni) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [22-25]@gni Expected: peer: - primary nid: 21@gni Multi-Rail: True peer ni: - nid: 21@gni - nid: 22@gni - nid: 23@gni - nid: 24@gni - nid: 25@gni global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: peer: - primary nid: 21@gni Multi-Rail: True peer ni: - nid: 21@gni - nid: 22@gni - nid: 23@gni - nid: 24@gni - nid: 25@gni global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Add peer with nidrange that overlaps primary nid (gni) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 21@gni --nid [21-25]@gni Expected: peer: - primary nid: 21@gni Multi-Rail: True peer ni: - nid: 21@gni - nid: 22@gni - nid: 23@gni - nid: 24@gni - nid: 25@gni global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: peer: - primary nid: 21@gni Multi-Rail: True peer ni: - nid: 21@gni - nid: 22@gni - nid: 23@gni - nid: 24@gni - nid: 25@gni global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 PASS 21 (2s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg133-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 22: Delete single secondary nid from peer (gni) ========================================================== 20:07:10 (1712275630) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 22@gni --nid [24-29]@gni /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 22@gni --nid 26@gni Expected: peer: - primary nid: 22@gni Multi-Rail: True peer ni: - nid: 22@gni - nid: 24@gni - nid: 25@gni - nid: 27@gni - nid: 28@gni - nid: 29@gni global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: peer: - primary nid: 22@gni Multi-Rail: True peer ni: - nid: 22@gni - nid: 24@gni - nid: 25@gni - nid: 27@gni - nid: 28@gni - nid: 29@gni global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 PASS 22 (2s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg133-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 23: Delete all secondary nids from peer (gni) ========================================================== 20:07:13 (1712275633) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 23@gni --nid [25-29]@gni /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 23@gni --nid [25-29]@gni Expected: peer: - primary nid: 23@gni Multi-Rail: True peer ni: - nid: 23@gni global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: peer: - primary nid: 23@gni Multi-Rail: True peer ni: - nid: 23@gni global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 PASS 23 (2s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg133-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 24: Delete a secondary nid from peer (tcp, o2ib and gni) ========================================================== 20:07:16 (1712275636) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 24@gni --nid [13-14/1].[13-15/2].13.13@o2ib,[15-16/3].[17-19/4].[1].[5-20/5]@tcp,[5-12/6]@gni /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 24@gni --nid 5@gni,13.15.13.13@o2ib,15.17.1.15@tcp Expected: peer: - primary nid: 24@gni Multi-Rail: True peer ni: - nid: 24@gni - nid: 11@gni - nid: 13.13.13.13@o2ib - nid: 14.13.13.13@o2ib - nid: 14.15.13.13@o2ib - nid: 15.17.1.5@tcp - nid: 15.17.1.10@tcp - nid: 15.17.1.20@tcp global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: peer: - primary nid: 24@gni Multi-Rail: True peer ni: - nid: 24@gni - nid: 11@gni - nid: 13.13.13.13@o2ib - nid: 14.13.13.13@o2ib - nid: 14.15.13.13@o2ib - nid: 15.17.1.5@tcp - nid: 15.17.1.10@tcp - nid: 15.17.1.20@tcp global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 PASS 24 (2s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg133-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 25: Delete all secondary nids from peer (tcp, gni and o2ib) ========================================================== 20:07:19 (1712275639) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 25@gni --nid [26-27].[4-10/3].26.26@tcp,26.26.26.26@o2ib,[30-35]@gni Expected: peer: - primary nid: 25@gni Multi-Rail: True peer ni: - nid: 25@gni global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: peer: - primary nid: 25@gni Multi-Rail: True peer ni: - nid: 25@gni global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 PASS 25 (2s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg133-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 26: Delete peer with primary nid locked ========================================================== 20:07:22 (1712275642) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --lock_prim /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@tcp --force /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer show --nid 1.1.1.1@tcp --- show: - peer: errno: -2 descr: ! "Object not found" ... pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 PASS 26 (2s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg133-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 99a: Check various invalid inputs to lnetctl peer ========================================================== 20:07:25 (1712275645) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure Invalid prim_nid - peer add /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid foobar --- add: - peer: errno: -22 descr: ! "Invalid input data or parameter" ... Invalid prim_nid - peer del /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid foobar --- del: - peer: errno: -22 descr: ! "Invalid input data or parameter" ... Delete non-existing peer /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim_nid 1.1.1.1@o2ib --- del: - peer: errno: -2 descr: ! "Object not found" ... Don't provide mandatory argument for peer del /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --nid 1.1.1.1@tcp --- del: - peer: errno: -42 descr: ! "Unspecific failure" ... Don't provide mandatory argument for peer add /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --nid 1.1.1.1@tcp --- add: - peer: errno: -42 descr: ! "Unspecific failure" ... Don't provide mandatory arguments peer add /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add peer add: add a peer NID --prim_nid: Primary NID of the peer. --nid: one or more peer NIDs --non_mr: create this peer as not Multi-Rail capable --ip2nets: specify a range of nids per peer --lock_prim: lock primary nid Invalid secondary nids /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar --- add: - peer: errno: -22 descr: ! "Unable to parse nidlist: no valid NIDs in string" ... can't parse nidrange: "foobar" Exceed max nids per peer /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.[2-255]@tcp --- add: - peer: errno: -7 descr: ! "Unable to parse nidlist: specifies more NIDs than allowed" ... Invalid net type /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@foo --- add: - peer: errno: -22 descr: ! "Invalid input data or parameter" ... Invalid nid format Check invalid primary nid - '1@tcp' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@tcp --- add: - peer: errno: -22 descr: ! "Invalid input data or parameter" ... Check invalid primary nid - '1@o2ib' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1@o2ib --- add: - peer: errno: -22 descr: ! "Invalid input data or parameter" ... Check invalid primary nid - '1.1.1.1@gni' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@gni --- add: - peer: errno: -22 descr: ! "Invalid input data or parameter" ... Check invalid nidstring - '[2-1]@gni' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [2-1]@gni --- add: - peer: errno: -34 descr: ! "Unable to parse nidlist: range is wrong order" ... can't parse nidrange: "[2-1]" Check invalid nidstring - '[a-f/x]@gni' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid [a-f/x]@gni --- add: - peer: errno: -34 descr: ! "Unable to parse nidlist: range needs number after /" ... can't parse nidrange: "[a-f/x]" Check invalid nidstring - '256.256.256.256@tcp' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 256.256.256.256@tcp --- add: - peer: errno: -22 descr: ! "Invalid input data or parameter" ... Check invalid nidstring - '1.1.1.1.[2-5/f]@tcp' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.1.1.1.[2-5/f]@tcp --- add: - peer: errno: -34 descr: ! "Unable to parse nidlist: range needs number after /" ... can't parse nidrange: "1.1.1.1.[2-5/f]" Check invalid nidstring - '1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid 1.]2[.3.4@o2ib1.[2-4,[5-6],7-8].1.1@tcp --- add: - peer: errno: -22 descr: ! "Unable to parse nidlist: improper bracket ordering" ... can't parse nidrange: "1.]2[.3.4" Check invalid nidstring - 'foobar' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 1.1.1.1@tcp --nid foobar --- add: - peer: errno: -22 descr: ! "Unable to parse nidlist: no valid NIDs in string" ... can't parse nidrange: "foobar" Add non-local gateway /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp --gateway 1@gni --- add: - route: errno: -113 descr: ! "Unspecific failure" ... pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 PASS 99a (2s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg133-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 99b: Invalid value for Multi-Rail in yaml import ========================================================== 20:07:28 (1712275648) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import peer: - add: errno: -22 descr: Multi-Rail must be set to "True" or "False" found "Foobar" Expected: global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 PASS 99b (2s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg133-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 100: Add route with single gw (tcp) == 20:07:31 (1712275651) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if eth0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp7 --gateway 192.168.201.34@tcp pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 PASS 100 (2s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg133-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 101: Add route with multiple gw (tcp) ========================================================== 20:07:35 (1712275655) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if eth0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp8 --gateway 192.168.201.[34-38]@tcp pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 PASS 101 (3s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg133-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 102: Delete route with single gw ===== 20:07:40 (1712275660) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if eth0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp2 --gateway 192.168.201.34@tcp /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp2 --gateway 192.168.201.34@tcp pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 PASS 102 (10s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg133-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 103: Delete route with multiple gw === 20:07:52 (1712275672) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if eth0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp103 --gateway 192.168.201.[34-39/2]@tcp /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route del --net tcp103 --gateway 192.168.201.[34-39/2]@tcp pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 PASS 103 (1s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg133-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 104: Set/check response_tracking param ========================================================== 20:07:55 (1712275675) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure Set < 0; Should fail /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking -1 add: - response_tracking: errno: -22 descr: "Valid values are: 0, 1, 2, or 3" /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import add: - response_tracking: errno: -22 descr: "Valid values are: 0, 1, 2, or 3" Check valid values; Should succeed /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 2 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 3 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure Set > 3; Should fail /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set response_tracking 4 add: - response_tracking: errno: -22 descr: "Valid values are: 0, 1, 2, or 3" /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import add: - response_tracking: errno: -22 descr: "Valid values are: 0, 1, 2, or 3" pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 PASS 104 (3s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg133-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 105: Adding duplicate GW peer should fail ========================================================== 20:07:59 (1712275679) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if eth0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp105 --gateway 192.168.201.34@tcp /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.201.34@tcp --- add: - peer: errno: -16 descr: ! "Object busy" ... pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 PASS 105 (2s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg133-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 106: Deleting GW peer should fail ==== 20:08:02 (1712275682) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if eth0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl route add --net tcp106 --gateway 192.168.201.34@tcp /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer del --prim 192.168.201.34@tcp --- del: - peer: errno: -16 descr: ! "Object busy" ... pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 PASS 106 (3s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg133-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 107: Deleting extra interface doesn't crash node ========================================================== 20:08:06 (1712275686) Cleaning up LNet Writer error: failed to resolve Netlink family id unloading modules on: 'oleg133-server' oleg133-server: oleg133-server.virtnet: executing unload_modules_local oleg133-server: LNET unconfigure error 22: (null) modules unloaded. Loading LNet and configuring DLC Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if eth0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg Cleaning up LNet Writer error: failed to resolve Netlink family id unloading modules on: 'oleg133-server' oleg133-server: oleg133-server.virtnet: executing unload_modules_local oleg133-server: LNET unconfigure error 22: (null) modules unloaded. ip netns exec test_ns ip addr add 10.1.2.3/31 dev test1pg ip netns exec test_ns ip link set test1pg up pdsh@oleg133-client: oleg133-client: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-client: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 PASS 107 (5s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg133-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 108: Check Multi-Rail setup ========== 20:08:13 (1712275693) Cleaning up LNet LNET unconfigure error 22: (null) unloading modules on: 'oleg133-server' oleg133-server: oleg133-server.virtnet: executing unload_modules_local oleg133-server: LNET unconfigure error 22: (null) modules unloaded. Loading LNet and configuring DLC Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if eth0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg Expected: net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.201.33@tcp status: up interfaces: 0: eth0 - nid: 10.1.2.3@tcp status: up interfaces: 0: test1pg Actual: net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.201.33@tcp status: up interfaces: 0: eth0 - nid: 10.1.2.3@tcp status: up interfaces: 0: test1pg Cleaning up LNet Writer error: failed to resolve Netlink family id unloading modules on: 'oleg133-server' oleg133-server: oleg133-server.virtnet: executing unload_modules_local oleg133-server: LNET unconfigure error 22: (null) modules unloaded. ip netns exec test_ns ip addr add 10.1.2.3/31 dev test1pg ip netns exec test_ns ip link set test1pg up pdsh@oleg133-client: oleg133-client: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-client: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 PASS 108 (5s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg133-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 109: Add NI using a network interface alias (LU-16859) ========================================================== 20:08:19 (1712275699) Cleaning up LNet LNET unconfigure error 22: (null) unloading modules on: 'oleg133-server' oleg133-server: oleg133-server.virtnet: executing unload_modules_local oleg133-server: LNET unconfigure error 22: (null) modules unloaded. Loading LNet and configuring DLC Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if test1pg:0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg:0 Cleaning up LNet Writer error: failed to resolve Netlink family id unloading modules on: 'oleg133-server' oleg133-server: oleg133-server.virtnet: executing unload_modules_local oleg133-server: LNET unconfigure error 22: (null) modules unloaded. ip netns exec test_ns ip addr add 10.1.2.3/31 dev test1pg ip netns exec test_ns ip link set test1pg up pdsh@oleg133-client: oleg133-client: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-client: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 PASS 109 (6s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg133-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 110: Add NI using a specific TCP / IP address ========================================================== 20:08:27 (1712275707) Cleaning up LNet LNET unconfigure error 22: (null) unloading modules on: 'oleg133-server' oleg133-server: oleg133-server.virtnet: executing unload_modules_local oleg133-server: LNET unconfigure error 22: (null) modules unloaded. Loading LNet and configuring DLC Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --nid 192.168.201.33@tcp /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --nid 192.168.201.33@tcp Cleaning up LNet Writer error: failed to resolve Netlink family id unloading modules on: 'oleg133-server' oleg133-server: oleg133-server.virtnet: executing unload_modules_local oleg133-server: LNET unconfigure error 22: (null) modules unloaded. ip netns exec test_ns ip addr add 10.1.2.3/31 dev test1pg ip netns exec test_ns ip link set test1pg up pdsh@oleg133-client: oleg133-client: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-client: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 PASS 110 (5s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg133-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 200: load lnet w/o module option, configure in a non-default namespace ========================================================== 20:08:33 (1712275713) Cleaning up LNet LNET unconfigure error 22: (null) unloading modules on: 'oleg133-server' oleg133-server: oleg133-server.virtnet: executing unload_modules_local oleg133-server: LNET unconfigure error 22: (null) modules unloaded. Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ../lnet/lnet/lnet options: 'networks=""' ip netns exec test_ns /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 PASS 200 (4s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg133-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 201: load lnet using networks module options in a non-default namespace ========================================================== 20:08:38 (1712275718) Cleaning up LNet Writer error: failed to resolve Netlink family id unloading modules on: 'oleg133-server' oleg133-server: oleg133-server.virtnet: executing unload_modules_local oleg133-server: LNET unconfigure error 22: (null) modules unloaded. Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ../lnet/lnet/lnet options: 'networks=tcp(test1pg)' ip netns exec test_ns /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 PASS 201 (4s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg133-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 202: load lnet using ip2nets in a non-default namespace ========================================================== 20:08:44 (1712275724) Cleaning up LNet Writer error: failed to resolve Netlink family id unloading modules on: 'oleg133-server' oleg133-server: oleg133-server.virtnet: executing unload_modules_local oleg133-server: LNET unconfigure error 22: (null) modules unloaded. Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ../lnet/lnet/lnet options: 'networks="" ip2nets="tcp0(test1pg) 10.1.2.3"' ip netns exec test_ns /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 PASS 202 (4s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg133-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 203: add a network using an interface in the non-default namespace ========================================================== 20:08:50 (1712275730) Cleaning up LNet Writer error: failed to resolve Netlink family id unloading modules on: 'oleg133-server' oleg133-server: oleg133-server.virtnet: executing unload_modules_local oleg133-server: LNET unconfigure error 22: (null) modules unloaded. Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure ip netns exec test_ns /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp0 --if test1pg pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 PASS 203 (3s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg133-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 204: Check no health or resends for single-rail local failures ========================================================== 20:08:55 (1712275735) Cleaning up LNet Writer error: failed to resolve Netlink family id unloading modules on: 'oleg133-server' oleg133-server: oleg133-server.virtnet: executing unload_modules_local oleg133-server: LNET unconfigure error 22: (null) modules unloaded. Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all oleg133-server: Writer error: failed to resolve Netlink family id oleg133-server: opening /dev/lnet failed: No such file or directory oleg133-server: hint: the kernel modules may not be loaded oleg133-server: IOC_LIBCFS_GET_NI error 2: No such file or directory pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 oleg133-server: oleg133-server.virtnet: executing load_lnet config_on_load=1 oleg133-server: Loading modules from /home/green/git/lustre-release/lustre oleg133-server: detected 4 online CPUs by sysfs oleg133-server: Force libcfs to create 2 CPU partitions oleg133-server: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.133@tcp discover: - primary nid: 192.168.201.133@tcp Multi-Rail: true peer_ni: - nid: 192.168.201.133@tcp net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.201.33@tcp status: up interfaces: 0: eth0 - primary nid: 192.168.201.133@tcp - nid: 192.168.201.133@tcp health stats: health value: 1000 debug=+net Simulate local_interrupt Added drop rule 192.168.201.33@tcp->192.168.201.133@tcp (1/1) Added drop rule 192.168.201.33@tcp->192.168.201.33@tcp (1/1) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.133@tcp manage: - ping: errno: -5 descr: ! 'failed to ping 192.168.201.133@tcp: Input/output error' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all --- set: - net: errno: 3 descr: ! "Object not found" ... Removed 2 drop rules Check that no resends took place Check that local NI health is unchanged Simulate local_dropped Added drop rule 192.168.201.33@tcp->192.168.201.133@tcp (1/1) Added drop rule 192.168.201.33@tcp->192.168.201.33@tcp (1/1) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.133@tcp manage: - ping: errno: -5 descr: ! 'failed to ping 192.168.201.133@tcp: Input/output error' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all --- set: - net: errno: 3 descr: ! "Object not found" ... Removed 2 drop rules Check that no resends took place Check that local NI health is unchanged Simulate local_aborted Added drop rule 192.168.201.33@tcp->192.168.201.133@tcp (1/1) Added drop rule 192.168.201.33@tcp->192.168.201.33@tcp (1/1) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.133@tcp manage: - ping: errno: -5 descr: ! 'failed to ping 192.168.201.133@tcp: Input/output error' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all --- set: - net: errno: 3 descr: ! "Object not found" ... Removed 2 drop rules Check that no resends took place Check that local NI health is unchanged Simulate local_no_route Added drop rule 192.168.201.33@tcp->192.168.201.133@tcp (1/1) Added drop rule 192.168.201.33@tcp->192.168.201.33@tcp (1/1) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.133@tcp manage: - ping: errno: -5 descr: ! 'failed to ping 192.168.201.133@tcp: Input/output error' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all --- set: - net: errno: 3 descr: ! "Object not found" ... Removed 2 drop rules Check that no resends took place Check that local NI health is unchanged Simulate local_timeout Added drop rule 192.168.201.33@tcp->192.168.201.133@tcp (1/1) Added drop rule 192.168.201.33@tcp->192.168.201.33@tcp (1/1) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.133@tcp manage: - ping: errno: -5 descr: ! 'failed to ping 192.168.201.133@tcp: Input/output error' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all --- set: - net: errno: 3 descr: ! "Object not found" ... Removed 2 drop rules Check that no resends took place Check that local NI health is unchanged Simulate local_error Added drop rule 192.168.201.33@tcp->192.168.201.133@tcp (1/1) Added drop rule 192.168.201.33@tcp->192.168.201.33@tcp (1/1) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.133@tcp manage: - ping: errno: -5 descr: ! 'failed to ping 192.168.201.133@tcp: Input/output error' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all --- set: - net: errno: 3 descr: ! "Object not found" ... Removed 2 drop rules Check that no resends took place Check that local NI health is unchanged Writer error: failed to resolve Netlink family id unloading modules on: 'oleg133-server' oleg133-server: oleg133-server.virtnet: executing unload_modules_local modules unloaded. oleg133-server: oleg133-server.virtnet: executing unload_modules_local oleg133-server: LNET unconfigure error 22: (null) pdsh@oleg133-client: oleg133-client: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-client: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 PASS 204 (8s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg133-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 205: Check health and resends for multi-rail local failures ========================================================== 20:09:05 (1712275745) Cleaning up LNet LNET unconfigure error 22: (null) unloading modules on: 'oleg133-server' oleg133-server: oleg133-server.virtnet: executing unload_modules_local oleg133-server: LNET unconfigure error 22: (null) modules unloaded. Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all oleg133-server: Writer error: failed to resolve Netlink family id oleg133-server: opening /dev/lnet failed: No such file or directory oleg133-server: hint: the kernel modules may not be loaded oleg133-server: IOC_LIBCFS_GET_NI error 2: No such file or directory pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 oleg133-server: oleg133-server.virtnet: executing load_lnet config_on_load=1 oleg133-server: Loading modules from /home/green/git/lustre-release/lustre oleg133-server: detected 4 online CPUs by sysfs oleg133-server: Force libcfs to create 2 CPU partitions oleg133-server: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.133@tcp discover: - primary nid: 192.168.201.133@tcp Multi-Rail: true peer_ni: - nid: 192.168.201.133@tcp oleg133-server: oleg133-server.virtnet: executing lnet_if_list oleg133-server: oleg133-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure oleg133-server: oleg133-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if eth0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if eth0 net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.201.33@tcp status: up interfaces: 0: eth0 - net type: tcp1 local NI(s): - nid: 192.168.201.33@tcp1 status: up interfaces: 0: eth0 - primary nid: 192.168.201.133@tcp - nid: 192.168.201.133@tcp health stats: health value: 1000 - nid: 192.168.201.133@tcp1 health stats: health value: 1000 debug=+net Simulate local_interrupt Added drop rule 192.168.201.33@tcp->192.168.201.133@tcp (1/1) Added drop rule 192.168.201.33@tcp->192.168.201.133@tcp1 (1/1) Added drop rule 192.168.201.33@tcp->192.168.201.33@tcp (1/1) Added drop rule 192.168.201.33@tcp->192.168.201.33@tcp1 (1/1) Added drop rule 192.168.201.33@tcp1->192.168.201.133@tcp (1/1) Added drop rule 192.168.201.33@tcp1->192.168.201.133@tcp1 (1/1) Added drop rule 192.168.201.33@tcp1->192.168.201.33@tcp (1/1) Added drop rule 192.168.201.33@tcp1->192.168.201.33@tcp1 (1/1) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.133@tcp manage: - ping: errno: -5 descr: ! 'failed to ping 192.168.201.133@tcp: Input/output error' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all --- set: - net: errno: 3 descr: ! "Object not found" ... Removed 8 drop rules Check that 2 resends took place Check that local NI health has been changed Simulate local_dropped Added drop rule 192.168.201.33@tcp->192.168.201.133@tcp (1/1) Added drop rule 192.168.201.33@tcp->192.168.201.133@tcp1 (1/1) Added drop rule 192.168.201.33@tcp->192.168.201.33@tcp (1/1) Added drop rule 192.168.201.33@tcp->192.168.201.33@tcp1 (1/1) Added drop rule 192.168.201.33@tcp1->192.168.201.133@tcp (1/1) Added drop rule 192.168.201.33@tcp1->192.168.201.133@tcp1 (1/1) Added drop rule 192.168.201.33@tcp1->192.168.201.33@tcp (1/1) Added drop rule 192.168.201.33@tcp1->192.168.201.33@tcp1 (1/1) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.133@tcp manage: - ping: errno: -5 descr: ! 'failed to ping 192.168.201.133@tcp: Input/output error' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all --- set: - net: errno: 3 descr: ! "Object not found" ... Removed 8 drop rules Check that 2 resends took place Check that local NI health has been changed Simulate local_aborted Added drop rule 192.168.201.33@tcp->192.168.201.133@tcp (1/1) Added drop rule 192.168.201.33@tcp->192.168.201.133@tcp1 (1/1) Added drop rule 192.168.201.33@tcp->192.168.201.33@tcp (1/1) Added drop rule 192.168.201.33@tcp->192.168.201.33@tcp1 (1/1) Added drop rule 192.168.201.33@tcp1->192.168.201.133@tcp (1/1) Added drop rule 192.168.201.33@tcp1->192.168.201.133@tcp1 (1/1) Added drop rule 192.168.201.33@tcp1->192.168.201.33@tcp (1/1) Added drop rule 192.168.201.33@tcp1->192.168.201.33@tcp1 (1/1) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.133@tcp manage: - ping: errno: -5 descr: ! 'failed to ping 192.168.201.133@tcp: Input/output error' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all --- set: - net: errno: 3 descr: ! "Object not found" ... Removed 8 drop rules Check that 2 resends took place Check that local NI health has been changed Simulate local_no_route Added drop rule 192.168.201.33@tcp->192.168.201.133@tcp (1/1) Added drop rule 192.168.201.33@tcp->192.168.201.133@tcp1 (1/1) Added drop rule 192.168.201.33@tcp->192.168.201.33@tcp (1/1) Added drop rule 192.168.201.33@tcp->192.168.201.33@tcp1 (1/1) Added drop rule 192.168.201.33@tcp1->192.168.201.133@tcp (1/1) Added drop rule 192.168.201.33@tcp1->192.168.201.133@tcp1 (1/1) Added drop rule 192.168.201.33@tcp1->192.168.201.33@tcp (1/1) Added drop rule 192.168.201.33@tcp1->192.168.201.33@tcp1 (1/1) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.133@tcp manage: - ping: errno: -5 descr: ! 'failed to ping 192.168.201.133@tcp: Input/output error' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all --- set: - net: errno: 3 descr: ! "Object not found" ... Removed 8 drop rules Check that 2 resends took place Check that local NI health has been changed Simulate local_timeout Added drop rule 192.168.201.33@tcp->192.168.201.133@tcp (1/1) Added drop rule 192.168.201.33@tcp->192.168.201.133@tcp1 (1/1) Added drop rule 192.168.201.33@tcp->192.168.201.33@tcp (1/1) Added drop rule 192.168.201.33@tcp->192.168.201.33@tcp1 (1/1) Added drop rule 192.168.201.33@tcp1->192.168.201.133@tcp (1/1) Added drop rule 192.168.201.33@tcp1->192.168.201.133@tcp1 (1/1) Added drop rule 192.168.201.33@tcp1->192.168.201.33@tcp (1/1) Added drop rule 192.168.201.33@tcp1->192.168.201.33@tcp1 (1/1) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.133@tcp manage: - ping: errno: -5 descr: ! 'failed to ping 192.168.201.133@tcp: Input/output error' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all --- set: - net: errno: 3 descr: ! "Object not found" ... Removed 8 drop rules Check that 2 resends took place Check that local NI health has been changed Simulate local_error Added drop rule 192.168.201.33@tcp->192.168.201.133@tcp (1/1) Added drop rule 192.168.201.33@tcp->192.168.201.133@tcp1 (1/1) Added drop rule 192.168.201.33@tcp->192.168.201.33@tcp (1/1) Added drop rule 192.168.201.33@tcp->192.168.201.33@tcp1 (1/1) Added drop rule 192.168.201.33@tcp1->192.168.201.133@tcp (1/1) Added drop rule 192.168.201.33@tcp1->192.168.201.133@tcp1 (1/1) Added drop rule 192.168.201.33@tcp1->192.168.201.33@tcp (1/1) Added drop rule 192.168.201.33@tcp1->192.168.201.33@tcp1 (1/1) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.133@tcp manage: - ping: errno: -5 descr: ! 'failed to ping 192.168.201.133@tcp: Input/output error' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all --- set: - net: errno: 3 descr: ! "Object not found" ... Removed 8 drop rules Check that no resends took place Check that local NI health has been changed oleg133-server: oleg133-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp1 --if eth0 Writer error: failed to resolve Netlink family id unloading modules on: 'oleg133-server' oleg133-server: oleg133-server.virtnet: executing unload_modules_local modules unloaded. oleg133-server: oleg133-server.virtnet: executing unload_modules_local oleg133-server: LNET unconfigure error 22: (null) pdsh@oleg133-client: oleg133-client: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-client: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 PASS 205 (12s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg133-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 206: Check no health or resends for single-rail remote failures ========================================================== 20:09:18 (1712275758) Cleaning up LNet LNET unconfigure error 22: (null) unloading modules on: 'oleg133-server' oleg133-server: oleg133-server.virtnet: executing unload_modules_local oleg133-server: LNET unconfigure error 22: (null) modules unloaded. Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all oleg133-server: Writer error: failed to resolve Netlink family id oleg133-server: opening /dev/lnet failed: No such file or directory oleg133-server: hint: the kernel modules may not be loaded oleg133-server: IOC_LIBCFS_GET_NI error 2: No such file or directory pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 oleg133-server: oleg133-server.virtnet: executing load_lnet config_on_load=1 oleg133-server: Loading modules from /home/green/git/lustre-release/lustre oleg133-server: detected 4 online CPUs by sysfs oleg133-server: Force libcfs to create 2 CPU partitions oleg133-server: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.133@tcp discover: - primary nid: 192.168.201.133@tcp Multi-Rail: true peer_ni: - nid: 192.168.201.133@tcp net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.201.33@tcp status: up interfaces: 0: eth0 - primary nid: 192.168.201.133@tcp - nid: 192.168.201.133@tcp health stats: health value: 1000 debug=+net Simulate remote_dropped Added drop rule 192.168.201.33@tcp->192.168.201.133@tcp (1/1) Added drop rule 192.168.201.33@tcp->192.168.201.33@tcp (1/1) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.133@tcp manage: - ping: errno: -5 descr: ! 'failed to ping 192.168.201.133@tcp: Input/output error' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all --- set: - net: errno: 3 descr: ! "Object not found" ... Removed 2 drop rules Check that no resends took place Check that local NI health is unchanged Check that remote NI health is unchanged Simulate remote_error Added drop rule 192.168.201.33@tcp->192.168.201.133@tcp (1/1) Added drop rule 192.168.201.33@tcp->192.168.201.33@tcp (1/1) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.133@tcp manage: - ping: errno: -5 descr: ! 'failed to ping 192.168.201.133@tcp: Input/output error' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all --- set: - net: errno: 3 descr: ! "Object not found" ... Removed 2 drop rules Check that no resends took place Check that local NI health is unchanged Check that remote NI health is unchanged Simulate remote_timeout Added drop rule 192.168.201.33@tcp->192.168.201.133@tcp (1/1) Added drop rule 192.168.201.33@tcp->192.168.201.33@tcp (1/1) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.133@tcp manage: - ping: errno: -5 descr: ! 'failed to ping 192.168.201.133@tcp: Input/output error' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all --- set: - net: errno: 3 descr: ! "Object not found" ... Removed 2 drop rules Check that no resends took place Check that local NI health is unchanged Check that remote NI health is unchanged Writer error: failed to resolve Netlink family id unloading modules on: 'oleg133-server' oleg133-server: oleg133-server.virtnet: executing unload_modules_local modules unloaded. oleg133-server: oleg133-server.virtnet: executing unload_modules_local oleg133-server: LNET unconfigure error 22: (null) pdsh@oleg133-client: oleg133-client: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-client: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 PASS 206 (7s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg133-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 207: Check health and resends for multi-rail remote errors ========================================================== 20:09:27 (1712275767) Cleaning up LNet LNET unconfigure error 22: (null) unloading modules on: 'oleg133-server' oleg133-server: oleg133-server.virtnet: executing unload_modules_local oleg133-server: LNET unconfigure error 22: (null) modules unloaded. Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all oleg133-server: Writer error: failed to resolve Netlink family id oleg133-server: opening /dev/lnet failed: No such file or directory oleg133-server: hint: the kernel modules may not be loaded oleg133-server: IOC_LIBCFS_GET_NI error 2: No such file or directory pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 oleg133-server: oleg133-server.virtnet: executing load_lnet config_on_load=1 oleg133-server: Loading modules from /home/green/git/lustre-release/lustre oleg133-server: detected 4 online CPUs by sysfs oleg133-server: Force libcfs to create 2 CPU partitions oleg133-server: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.133@tcp discover: - primary nid: 192.168.201.133@tcp Multi-Rail: true peer_ni: - nid: 192.168.201.133@tcp oleg133-server: oleg133-server.virtnet: executing lnet_if_list oleg133-server: oleg133-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure oleg133-server: oleg133-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if eth0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if eth0 net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.201.33@tcp status: up interfaces: 0: eth0 - net type: tcp1 local NI(s): - nid: 192.168.201.33@tcp1 status: up interfaces: 0: eth0 - primary nid: 192.168.201.133@tcp - nid: 192.168.201.133@tcp health stats: health value: 1000 - nid: 192.168.201.133@tcp1 health stats: health value: 1000 debug=+net Simulate remote_dropped Added drop rule 192.168.201.33@tcp->192.168.201.133@tcp (1/1) Added drop rule 192.168.201.33@tcp->192.168.201.133@tcp1 (1/1) Added drop rule 192.168.201.33@tcp->192.168.201.33@tcp (1/1) Added drop rule 192.168.201.33@tcp->192.168.201.33@tcp1 (1/1) Added drop rule 192.168.201.33@tcp1->192.168.201.133@tcp (1/1) Added drop rule 192.168.201.33@tcp1->192.168.201.133@tcp1 (1/1) Added drop rule 192.168.201.33@tcp1->192.168.201.33@tcp (1/1) Added drop rule 192.168.201.33@tcp1->192.168.201.33@tcp1 (1/1) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.133@tcp manage: - ping: errno: -5 descr: ! 'failed to ping 192.168.201.133@tcp: Input/output error' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all --- set: - net: errno: 3 descr: ! "Object not found" ... Removed 8 drop rules Check that 2 resends took place Check that local NI health is unchanged Check that remote NI health has been changed /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all Simulate remote_error Added drop rule 192.168.201.33@tcp->192.168.201.133@tcp (1/1) Added drop rule 192.168.201.33@tcp->192.168.201.133@tcp1 (1/1) Added drop rule 192.168.201.33@tcp->192.168.201.33@tcp (1/1) Added drop rule 192.168.201.33@tcp->192.168.201.33@tcp1 (1/1) Added drop rule 192.168.201.33@tcp1->192.168.201.133@tcp (1/1) Added drop rule 192.168.201.33@tcp1->192.168.201.133@tcp1 (1/1) Added drop rule 192.168.201.33@tcp1->192.168.201.33@tcp (1/1) Added drop rule 192.168.201.33@tcp1->192.168.201.33@tcp1 (1/1) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.133@tcp manage: - ping: errno: -5 descr: ! 'failed to ping 192.168.201.133@tcp: Input/output error' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all --- set: - net: errno: 3 descr: ! "Object not found" ... Removed 8 drop rules Check that no resends took place Check that local NI health is unchanged Check that remote NI health has been changed /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all Simulate remote_timeout Added drop rule 192.168.201.33@tcp->192.168.201.133@tcp (1/1) Added drop rule 192.168.201.33@tcp->192.168.201.133@tcp1 (1/1) Added drop rule 192.168.201.33@tcp->192.168.201.33@tcp (1/1) Added drop rule 192.168.201.33@tcp->192.168.201.33@tcp1 (1/1) Added drop rule 192.168.201.33@tcp1->192.168.201.133@tcp (1/1) Added drop rule 192.168.201.33@tcp1->192.168.201.133@tcp1 (1/1) Added drop rule 192.168.201.33@tcp1->192.168.201.33@tcp (1/1) Added drop rule 192.168.201.33@tcp1->192.168.201.33@tcp1 (1/1) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.133@tcp manage: - ping: errno: -5 descr: ! 'failed to ping 192.168.201.133@tcp: Input/output error' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all --- set: - net: errno: 3 descr: ! "Object not found" ... Removed 8 drop rules Check that no resends took place Check that local NI health is unchanged Check that remote NI health has been changed /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all oleg133-server: oleg133-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp1 --if eth0 Writer error: failed to resolve Netlink family id unloading modules on: 'oleg133-server' oleg133-server: oleg133-server.virtnet: executing unload_modules_local modules unloaded. oleg133-server: oleg133-server.virtnet: executing unload_modules_local oleg133-server: LNET unconfigure error 22: (null) pdsh@oleg133-client: oleg133-client: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-client: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 PASS 207 (12s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg133-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 208: Test various kernel ip2nets configurations ========================================================== 20:09:40 (1712275780) Cleaning up LNet LNET unconfigure error 22: (null) unloading modules on: 'oleg133-server' oleg133-server: oleg133-server.virtnet: executing unload_modules_local oleg133-server: LNET unconfigure error 22: (null) modules unloaded. Configure single NID "tcp(eth0) 192.168.201.33" Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ../lnet/lnet/lnet options: 'networks="" ip2nets="tcp(eth0) 192.168.201.33"' LNET configured Cleaning up LNet Writer error: failed to resolve Netlink family id unloading modules on: 'oleg133-server' oleg133-server: oleg133-server.virtnet: executing unload_modules_local oleg133-server: LNET unconfigure error 22: (null) modules unloaded. Configure two NIDs; two NETs "tcp(eth0) 192.168.201.33; tcp1(test1pg) 10.1.2.3" Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ../lnet/lnet/lnet options: 'networks="" ip2nets="tcp(eth0) 192.168.201.33; tcp1(test1pg) 10.1.2.3"' LNET configured Cleaning up LNet Writer error: failed to resolve Netlink family id unloading modules on: 'oleg133-server' oleg133-server: oleg133-server.virtnet: executing unload_modules_local oleg133-server: LNET unconfigure error 22: (null) modules unloaded. Configure two NIDs; one NET "tcp(eth0) 192.168.201.33; tcp(test1pg) 10.1.2.3" Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ../lnet/lnet/lnet options: 'networks="" ip2nets="tcp(eth0) 192.168.201.33; tcp(test1pg) 10.1.2.3"' LNET configured Cleaning up LNet Writer error: failed to resolve Netlink family id unloading modules on: 'oleg133-server' oleg133-server: oleg133-server.virtnet: executing unload_modules_local oleg133-server: LNET unconfigure error 22: (null) modules unloaded. Configured two NIDs; one NET alt syntax "tcp(eth0,test1pg) [192,10].[168,1].[201,2].[33,3]" Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ../lnet/lnet/lnet options: 'networks="" ip2nets="tcp(eth0,test1pg) [192,10].[168,1].[201,2].[33,3]"' LNET configured Cleaning up LNet Writer error: failed to resolve Netlink family id unloading modules on: 'oleg133-server' oleg133-server: oleg133-server.virtnet: executing unload_modules_local oleg133-server: LNET unconfigure error 22: (null) modules unloaded. alt syntax with missing IF "tcp(eth0,test1pg) [192,10].[168,1].[201,2].[33,3]" Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ../lnet/lnet/lnet options: 'networks="" ip2nets="tcp(eth0,test1pg) [192,10].[168,1].[201,2].[33,3]"' /home/green/git/lustre-release/lustre/utils/lctl net up should fail LNET configure error 100: Unspecific failure Cleaning up LNet Writer error: failed to resolve Netlink family id unloading modules on: 'oleg133-server' oleg133-server: oleg133-server.virtnet: executing unload_modules_local oleg133-server: LNET unconfigure error 22: (null) modules unloaded. pdsh@oleg133-client: oleg133-client: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-client: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 PASS 208 (12s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg133-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 209: Check health, but not resends, for network timeout ========================================================== 20:09:53 (1712275793) Cleaning up LNet LNET unconfigure error 22: (null) unloading modules on: 'oleg133-server' oleg133-server: oleg133-server.virtnet: executing unload_modules_local oleg133-server: LNET unconfigure error 22: (null) modules unloaded. Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all oleg133-server: Writer error: failed to resolve Netlink family id oleg133-server: opening /dev/lnet failed: No such file or directory oleg133-server: hint: the kernel modules may not be loaded oleg133-server: IOC_LIBCFS_GET_NI error 2: No such file or directory pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 oleg133-server: oleg133-server.virtnet: executing load_lnet config_on_load=1 oleg133-server: Loading modules from /home/green/git/lustre-release/lustre oleg133-server: detected 4 online CPUs by sysfs oleg133-server: Force libcfs to create 2 CPU partitions oleg133-server: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.133@tcp discover: - primary nid: 192.168.201.133@tcp Multi-Rail: true peer_ni: - nid: 192.168.201.133@tcp net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.201.33@tcp status: up interfaces: 0: eth0 - primary nid: 192.168.201.133@tcp - nid: 192.168.201.133@tcp health stats: health value: 1000 debug=+net Simulate network_timeout w/SR config Added drop rule 192.168.201.33@tcp->192.168.201.133@tcp (1/1) Added drop rule 192.168.201.33@tcp->192.168.201.33@tcp (1/1) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.133@tcp manage: - discover: errno: -5 descr: ! 'failed to discover 192.168.201.133@tcp: Input/output error' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all --- set: - net: errno: 3 descr: ! "Object not found" ... Check that no resends took place Check that local NI health is unchanged Check that remote NI health is unchanged Writer error: failed to resolve Netlink family id unloading modules on: 'oleg133-server' oleg133-server: oleg133-server.virtnet: executing unload_modules_local modules unloaded. oleg133-server: oleg133-server.virtnet: executing unload_modules_local oleg133-server: LNET unconfigure error 22: (null) Cleaning up LNet LNET unconfigure error 22: (null) unloading modules on: 'oleg133-server' oleg133-server: oleg133-server.virtnet: executing unload_modules_local oleg133-server: LNET unconfigure error 22: (null) modules unloaded. Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all oleg133-server: Writer error: failed to resolve Netlink family id oleg133-server: opening /dev/lnet failed: No such file or directory oleg133-server: hint: the kernel modules may not be loaded oleg133-server: IOC_LIBCFS_GET_NI error 2: No such file or directory pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 oleg133-server: oleg133-server.virtnet: executing load_lnet config_on_load=1 oleg133-server: Loading modules from /home/green/git/lustre-release/lustre oleg133-server: detected 4 online CPUs by sysfs oleg133-server: Force libcfs to create 2 CPU partitions oleg133-server: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.133@tcp discover: - primary nid: 192.168.201.133@tcp Multi-Rail: true peer_ni: - nid: 192.168.201.133@tcp oleg133-server: oleg133-server.virtnet: executing lnet_if_list oleg133-server: oleg133-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure oleg133-server: oleg133-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if eth0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if eth0 net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.201.33@tcp status: up interfaces: 0: eth0 - net type: tcp1 local NI(s): - nid: 192.168.201.33@tcp1 status: up interfaces: 0: eth0 - primary nid: 192.168.201.133@tcp - nid: 192.168.201.133@tcp health stats: health value: 1000 - nid: 192.168.201.133@tcp1 health stats: health value: 1000 debug=+net Simulate network_timeout w/MR config Added drop rule 192.168.201.33@tcp->192.168.201.133@tcp (1/1) Added drop rule 192.168.201.33@tcp->192.168.201.133@tcp1 (1/1) Added drop rule 192.168.201.33@tcp->192.168.201.33@tcp (1/1) Added drop rule 192.168.201.33@tcp->192.168.201.33@tcp1 (1/1) Added drop rule 192.168.201.33@tcp1->192.168.201.133@tcp (1/1) Added drop rule 192.168.201.33@tcp1->192.168.201.133@tcp1 (1/1) Added drop rule 192.168.201.33@tcp1->192.168.201.33@tcp (1/1) Added drop rule 192.168.201.33@tcp1->192.168.201.33@tcp1 (1/1) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.133@tcp manage: - discover: errno: -5 descr: ! 'failed to discover 192.168.201.133@tcp: Input/output error' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all --- set: - net: errno: 3 descr: ! "Object not found" ... Check that no resends took place Check that local NI health has been changed Check that remote NI health has been changed oleg133-server: oleg133-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp1 --if eth0 Writer error: failed to resolve Netlink family id unloading modules on: 'oleg133-server' oleg133-server: oleg133-server.virtnet: executing unload_modules_local modules unloaded. oleg133-server: oleg133-server.virtnet: executing unload_modules_local oleg133-server: LNET unconfigure error 22: (null) pdsh@oleg133-client: oleg133-client: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-client: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 PASS 209 (17s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg133-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 210: Local NI recovery checks ======== 20:10:12 (1712275812) Loading LNet and configuring DLC Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if eth0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if eth0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.33@tcp discover: - primary nid: 192.168.201.33@tcp Multi-Rail: true peer_ni: - nid: 192.168.201.33@tcp - nid: 192.168.201.33@tcp1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 debug=+net Added drop rule 255.255.255.255@tcp->255.255.255.255@tcp (1/1) Added drop rule 255.255.255.255@tcp1->255.255.255.255@tcp1 (1/1) Added drop rule 255.255.255.255@tcp->255.255.255.255@tcp (1/1) Added drop rule 255.255.255.255@tcp1->255.255.255.255@tcp1 (1/1) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.33@tcp Check ping counts: Waiting 10s for '2' Updated after 3s: want '2' got '2' ping_count: 0 - nid: 192.168.201.33@tcp ping_count: 2 - nid: 192.168.201.33@tcp1 ping_count: 0 -l recovery queue should have 192.168.201.33@tcp /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l local NI recovery: nid-0: 192.168.201.33@tcp Check ping counts: Waiting 10s for '3' Updated after 4s: want '3' got '3' ping_count: 0 - nid: 192.168.201.33@tcp ping_count: 3 - nid: 192.168.201.33@tcp1 ping_count: 0 -l recovery queue should have 192.168.201.33@tcp /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l local NI recovery: nid-0: 192.168.201.33@tcp Removed 4 drop rules /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if eth0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if eth0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.33@tcp discover: - primary nid: 192.168.201.33@tcp Multi-Rail: true peer_ni: - nid: 192.168.201.33@tcp - nid: 192.168.201.33@tcp1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 debug=+net Added drop rule 255.255.255.255@tcp->255.255.255.255@tcp (1/1) Added drop rule 255.255.255.255@tcp1->255.255.255.255@tcp1 (1/1) Added drop rule 255.255.255.255@tcp->255.255.255.255@tcp (1/1) Added drop rule 255.255.255.255@tcp1->255.255.255.255@tcp1 (1/1) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 0 --nid 192.168.201.33@tcp Check ping counts: Waiting 10s for '2' Updated after 3s: want '2' got '2' ping_count: 0 - nid: 192.168.201.33@tcp ping_count: 2 - nid: 192.168.201.33@tcp1 ping_count: 0 -l recovery queue should have 192.168.201.33@tcp /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l local NI recovery: nid-0: 192.168.201.33@tcp Check ping counts: Waiting 10s for '4' Updated after 8s: want '4' got '4' ping_count: 0 - nid: 192.168.201.33@tcp ping_count: 4 - nid: 192.168.201.33@tcp1 ping_count: 0 -l recovery queue should have 192.168.201.33@tcp /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l local NI recovery: nid-0: 192.168.201.33@tcp Removed 4 drop rules /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 PASS 210 (22s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg133-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 211: Remote NI recovery checks ======= 20:10:36 (1712275836) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if eth0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if eth0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.33@tcp discover: - primary nid: 192.168.201.33@tcp Multi-Rail: true peer_ni: - nid: 192.168.201.33@tcp - nid: 192.168.201.33@tcp1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 10 Added drop rule 255.255.255.255@tcp->255.255.255.255@tcp (1/1) Added drop rule 255.255.255.255@tcp1->255.255.255.255@tcp1 (1/1) Added drop rule 255.255.255.255@tcp->255.255.255.255@tcp (1/1) Added drop rule 255.255.255.255@tcp1->255.255.255.255@tcp1 (1/1) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.33@tcp --health 0 -p recovery queue should have 192.168.201.33@tcp /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p peer NI recovery: nid-0: 192.168.201.33@tcp -p recovery queue should be empty Waiting 20s for '0' Waiting 10s for '0' Updated after 15s: want '0' got '0' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p Check ping counts: - nid: 192.168.201.33@tcp ping_count: 0 - nid: 192.168.201.33@tcp1 ping_count: 0 Removed 4 drop rules /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.33@tcp --health 500 -p recovery queue should have 192.168.201.33@tcp /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p peer NI recovery: nid-0: 192.168.201.33@tcp -p recovery queue should be empty Waiting 20s for '0' Updated after 7s: want '0' got '0' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if eth0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if eth0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.33@tcp discover: - primary nid: 192.168.201.33@tcp Multi-Rail: true peer_ni: - nid: 192.168.201.33@tcp - nid: 192.168.201.33@tcp1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set recovery_limit 0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 4 Added drop rule 255.255.255.255@tcp->255.255.255.255@tcp (1/1) Added drop rule 255.255.255.255@tcp1->255.255.255.255@tcp1 (1/1) Added drop rule 255.255.255.255@tcp->255.255.255.255@tcp (1/1) Added drop rule 255.255.255.255@tcp1->255.255.255.255@tcp1 (1/1) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 192.168.201.33@tcp --health 0 Check ping counts: Waiting 4s for '1' - nid: 192.168.201.33@tcp ping_count: 1 - nid: 192.168.201.33@tcp1 ping_count: 0 -p recovery queue should have 192.168.201.33@tcp /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p peer NI recovery: nid-0: 192.168.201.33@tcp Check ping counts: Waiting 14s for '4' Updated after 10s: want '4' got '4' - nid: 192.168.201.33@tcp ping_count: 4 - nid: 192.168.201.33@tcp1 ping_count: 0 -p recovery queue should have 192.168.201.33@tcp /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p peer NI recovery: nid-0: 192.168.201.33@tcp Removed 4 drop rules /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl set max_recovery_ping_interval 900 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 PASS 211 (39s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg133-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 212: Check discovery refcount loss bug (LU-14627) ========================================================== 20:11:17 (1712275877) Cleaning up LNet Writer error: failed to resolve Netlink family id unloading modules on: 'oleg133-server' oleg133-server: oleg133-server.virtnet: executing unload_modules_local oleg133-server: LNET unconfigure error 22: (null) modules unloaded. Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all oleg133-server: Writer error: failed to resolve Netlink family id oleg133-server: opening /dev/lnet failed: No such file or directory oleg133-server: hint: the kernel modules may not be loaded oleg133-server: IOC_LIBCFS_GET_NI error 2: No such file or directory pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 oleg133-server: oleg133-server.virtnet: executing load_lnet config_on_load=1 oleg133-server: Loading modules from /home/green/git/lustre-release/lustre oleg133-server: detected 4 online CPUs by sysfs oleg133-server: Force libcfs to create 2 CPU partitions oleg133-server: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all Initial discovery /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.133@tcp discover: - primary nid: 192.168.201.133@tcp Multi-Rail: true peer_ni: - nid: 192.168.201.133@tcp discover: - primary nid: 192.168.201.33@tcp Multi-Rail: true peer_ni: - nid: 192.168.201.33@tcp Fail local discover ping to set LNET_PEER_REDISCOVER flag Added drop rule 255.255.255.255@tcp->255.255.255.255@tcp (1/1) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.133@tcp manage: - discover: errno: -5 descr: ! 'failed to discover 192.168.201.133@tcp: Input/output error' Removed 1 drop rules Added delay rule 255.255.255.255@tcp->192.168.201.133@tcp (1/1) Added drop rule 255.255.255.255@tcp->192.168.201.133@tcp (1/1) Added delay rule 255.255.255.255@tcp->192.168.201.133@tcp (1/1) Force oleg133-client.virtnet to discover 192.168.201.133@tcp (in background) Force oleg133-server to discover 192.168.201.33@tcp /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover --force 192.168.201.133@tcp discover: - primary nid: 192.168.201.33@tcp Multi-Rail: true peer_ni: - nid: 192.168.201.33@tcp Wait for 30893 manage: - discover: errno: -5 descr: ! 'failed to discover 192.168.201.133@tcp: Input/output error' Finished wait on 30893 Removed 2 delay rules Removed 1 drop rules Writer error: failed to resolve Netlink family id unloading modules on: 'oleg133-server' oleg133-server: oleg133-server.virtnet: executing unload_modules_local modules unloaded. oleg133-server: oleg133-server.virtnet: executing unload_modules_local oleg133-server: LNET unconfigure error 22: (null) pdsh@oleg133-client: oleg133-client: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-client: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 PASS 212 (15s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg133-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 213: Check LNetDist calculation for multiple local NIDs ========================================================== 20:11:33 (1712275893) Cleaning up LNet LNET unconfigure error 22: (null) unloading modules on: 'oleg133-server' oleg133-server: oleg133-server.virtnet: executing unload_modules_local oleg133-server: LNET unconfigure error 22: (null) modules unloaded. Loading LNet and configuring DLC Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if eth0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 PASS 213 (4s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg133-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 214: Check local NI status when link is downed ========================================================== 20:11:38 (1712275898) Cleaning up LNet Writer error: failed to resolve Netlink family id unloading modules on: 'oleg133-server' oleg133-server: oleg133-server.virtnet: executing unload_modules_local oleg133-server: LNET unconfigure error 22: (null) modules unloaded. Loading LNet and configuring DLC Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if eth0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg NI 0@lo expect status "up" found "up" NI 192.168.201.33@tcp expect status "up" found "up" NI 10.1.2.3@tcp expect status "up" found "up" /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 10.1.2.3@tcp 192.168.201.33@tcp ping: - primary nid: 192.168.201.33@tcp Multi-Rail: false peer_ni: - nid: 192.168.201.33@tcp - nid: 10.1.2.3@tcp Set test1pg down ip link set dev test1pg down NI 0@lo expect status "up" found "up" NI 192.168.201.33@tcp expect status "up" found "up" NI 10.1.2.3@tcp expect status "down" found "down" pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 PASS 214 (4s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg133-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 215: Test lnetctl ping --source option ========================================================== 20:11:44 (1712275904) Cleaning up LNet Writer error: failed to resolve Netlink family id unloading modules on: 'oleg133-server' oleg133-server: oleg133-server.virtnet: executing unload_modules_local oleg133-server: LNET unconfigure error 22: (null) modules unloaded. Loading LNet and configuring DLC Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if eth0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp2 --if eth0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 192.168.201.33@tcp1 --nid 192.168.201.33@tcp2 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.33@tcp1 x 25 192.168.201.33@tcp1 pre send_count 0 post send_count 26 192.168.201.33@tcp1 pre recv_count 0 post recv_count 26 192.168.201.33@tcp2 pre send_count 0 post send_count 24 192.168.201.33@tcp2 pre recv_count 0 post recv_count 24 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.33@tcp1 192.168.201.33@tcp1 x 25 192.168.201.33@tcp1 pre send_count 26 post send_count 76 192.168.201.33@tcp1 pre recv_count 26 post recv_count 76 192.168.201.33@tcp2 pre send_count 24 post send_count 24 192.168.201.33@tcp2 pre recv_count 24 post recv_count 24 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.33@tcp2 x 50 192.168.201.33@tcp1 pre send_count 76 post send_count 126 192.168.201.33@tcp1 pre recv_count 76 post recv_count 126 192.168.201.33@tcp2 pre send_count 24 post send_count 74 192.168.201.33@tcp2 pre recv_count 24 post recv_count 74 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.33@tcp2 192.168.201.33@tcp2 x 50 192.168.201.33@tcp1 pre send_count 126 post send_count 126 192.168.201.33@tcp1 pre recv_count 126 post recv_count 126 192.168.201.33@tcp2 pre send_count 74 post send_count 174 192.168.201.33@tcp2 pre recv_count 74 post recv_count 174 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.33@tcp1 192.168.201.33@tcp2 manage: - ping: errno: -113 descr: ! 'failed to ping 192.168.201.33@tcp2: No route to host' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.33@tcp2 192.168.201.33@tcp1 manage: - ping: errno: -113 descr: ! 'failed to ping 192.168.201.33@tcp1: No route to host' pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 PASS 215 (6s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg133-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 216: Failed send to peer NI owned by local host should not trigger peer NI recovery ========================================================== 20:11:52 (1712275912) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if eth0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if eth0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.33@tcp discover: - primary nid: 192.168.201.33@tcp Multi-Rail: true peer_ni: - nid: 192.168.201.33@tcp - nid: 192.168.201.33@tcp1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.33@tcp 192.168.201.33@tcp ping: - primary nid: 192.168.201.33@tcp Multi-Rail: true peer_ni: - nid: 192.168.201.33@tcp - nid: 192.168.201.33@tcp1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --source 192.168.201.33@tcp1 192.168.201.33@tcp1 ping: - primary nid: 192.168.201.33@tcp Multi-Rail: true peer_ni: - nid: 192.168.201.33@tcp - nid: 192.168.201.33@tcp1 Added drop rule 192.168.201.33@tcp->192.168.201.33@tcp (1/1) Added drop rule 192.168.201.33@tcp->192.168.201.33@tcp1 (1/1) Added drop rule 192.168.201.33@tcp1->192.168.201.33@tcp (1/1) Added drop rule 192.168.201.33@tcp1->192.168.201.33@tcp1 (1/1) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.33@tcp manage: - ping: errno: -5 descr: ! 'failed to ping 192.168.201.33@tcp: Input/output error' Removed 4 drop rules -p recovery queue should be empty /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -p -l recovery queue should have 192.168.201.33@tcp /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug recovery -l local NI recovery: nid-0: 192.168.201.33@tcp pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 PASS 216 (3s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg133-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 217: Don't leak memory when discovering peer with nnis <= 1 ========================================================== 20:11:56 (1712275916) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 0@lo discover: - primary nid: 0@lo Multi-Rail: true peer_ni: Writer error: failed to resolve Netlink family id unloading modules on: 'oleg133-server' oleg133-server: oleg133-server.virtnet: executing unload_modules_local oleg133-server: LNET unconfigure error 22: (null) modules unloaded. pdsh@oleg133-client: oleg133-client: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-client: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 PASS 217 (5s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg133-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 218: Local recovery pings should exercise all available paths ========================================================== 20:12:02 (1712275922) Loading LNet and configuring DLC Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure SKIP: sanity-lnet test_218 Need two LNet interfaces SKIP 218 (2s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg133-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 219: Consolidate peer entries ======== 20:12:05 (1712275925) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if eth0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if eth0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.33@tcp ping: - primary nid: 192.168.201.33@tcp Multi-Rail: false peer_ni: - nid: 192.168.201.33@tcp - nid: 192.168.201.33@tcp1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.33@tcp1 ping: - primary nid: 192.168.201.33@tcp1 Multi-Rail: false peer_ni: - nid: 192.168.201.33@tcp - nid: 192.168.201.33@tcp1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.33@tcp1 discover: - primary nid: 192.168.201.33@tcp Multi-Rail: true peer_ni: - nid: 192.168.201.33@tcp - nid: 192.168.201.33@tcp1 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 PASS 219 (2s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg133-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 220: Add routes w/default options - check aliveness ========================================================== 20:12:09 (1712275929) SKIP: sanity-lnet test_220 Need at least 2 remote nodes found " oleg133-server" SKIP 220 (0s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg133-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 221: Add routes w/DD disabled - check aliveness ========================================================== 20:12:11 (1712275931) SKIP: sanity-lnet test_221 Need at least 2 remote nodes found " oleg133-server" SKIP 221 (1s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg133-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 222: Check avoid_asym_router_failure=1 ========================================================== 20:12:14 (1712275934) SKIP: sanity-lnet test_222 Need at least 2 remote nodes found " oleg133-server" SKIP 222 (1s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg133-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 223: Check avoid_asym_router_failure=1 w/DD disabled ========================================================== 20:12:16 (1712275936) SKIP: sanity-lnet test_223 Need at least 2 remote nodes found " oleg133-server" SKIP 223 (1s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg133-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 224: Check avoid_asym_router_failure=0 ========================================================== 20:12:19 (1712275939) SKIP: sanity-lnet test_224 Need at least 2 remote nodes found " oleg133-server" SKIP 224 (1s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg133-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 225: Check avoid_asym_router_failure=0 w/DD disabled ========================================================== 20:12:21 (1712275941) SKIP: sanity-lnet test_225 Need at least 2 remote nodes found " oleg133-server" SKIP 225 (1s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg133-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 230: Test setting conns-per-peer ===== 20:12:24 (1712275944) Check valid values; Should succeed /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if eth0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 4 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.33@tcp ping: - primary nid: 192.168.201.33@tcp Multi-Rail: false peer_ni: - nid: 192.168.201.33@tcp /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if eth0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 5 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.33@tcp ping: - primary nid: 192.168.201.33@tcp Multi-Rail: false peer_ni: - nid: 192.168.201.33@tcp /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if eth0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 6 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.33@tcp ping: - primary nid: 192.168.201.33@tcp Multi-Rail: false peer_ni: - nid: 192.168.201.33@tcp /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if eth0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 7 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.33@tcp ping: - primary nid: 192.168.201.33@tcp Multi-Rail: false peer_ni: - nid: 192.168.201.33@tcp /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if eth0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 8 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.33@tcp ping: - primary nid: 192.168.201.33@tcp Multi-Rail: false peer_ni: - nid: 192.168.201.33@tcp /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if eth0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 9 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.33@tcp ping: - primary nid: 192.168.201.33@tcp Multi-Rail: false peer_ni: - nid: 192.168.201.33@tcp /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if eth0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 10 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.33@tcp ping: - primary nid: 192.168.201.33@tcp Multi-Rail: false peer_ni: - nid: 192.168.201.33@tcp /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if eth0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 11 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.33@tcp ping: - primary nid: 192.168.201.33@tcp Multi-Rail: false peer_ni: - nid: 192.168.201.33@tcp /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if eth0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 12 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.33@tcp ping: - primary nid: 192.168.201.33@tcp Multi-Rail: false peer_ni: - nid: 192.168.201.33@tcp /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if eth0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 13 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.33@tcp ping: - primary nid: 192.168.201.33@tcp Multi-Rail: false peer_ni: - nid: 192.168.201.33@tcp /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if eth0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 14 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.33@tcp ping: - primary nid: 192.168.201.33@tcp Multi-Rail: false peer_ni: - nid: 192.168.201.33@tcp /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if eth0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 15 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.33@tcp ping: - primary nid: 192.168.201.33@tcp Multi-Rail: false peer_ni: - nid: 192.168.201.33@tcp /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if eth0 Set > 127; Should fail /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer 128 --- set: - net: errno: -34 descr: ! "Input data out of range" ... /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if eth0 Set < 0; Should be ignored /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --all --conns-per-peer -1 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 PASS 230 (20s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg133-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 231: Check DLC handling of peer_timeout parameter ========================================================== 20:12:45 (1712275965) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if eth0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-expected.yaml Expected: net: - net type: tcp231 local NI(s): - interfaces: 0: eth0 tunables: peer_timeout: 0 peer_credits: 8 peer_buffer_credits: 0 credits: 256 lnd tunables: conns_per_peer: 1 timeout: 49 CPT: "[0]" global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: net: - net type: tcp231 local NI(s): - interfaces: 0: eth0 tunables: peer_timeout: 0 peer_credits: 8 peer_buffer_credits: 0 credits: 256 lnd tunables: conns_per_peer: 1 timeout: 49 CPT: "[0]" global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp231 --if eth0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp231 --if eth0 --peer-timeout=0 Expected: net: - net type: tcp231 local NI(s): - interfaces: 0: eth0 tunables: peer_timeout: 0 peer_credits: 8 peer_buffer_credits: 0 credits: 256 lnd tunables: conns_per_peer: 1 timeout: 49 CPT: "[0]" global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: net: - net type: tcp231 local NI(s): - interfaces: 0: eth0 tunables: peer_timeout: 0 peer_credits: 8 peer_buffer_credits: 0 credits: 256 lnd tunables: conns_per_peer: 1 timeout: 49 CPT: "[0]" global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl import /tmp/sanity-lnet-231-actual.yaml Expected: net: - net type: tcp231 local NI(s): - interfaces: 0: eth0 tunables: peer_timeout: 180 peer_credits: 8 peer_buffer_credits: 0 credits: 256 lnd tunables: conns_per_peer: 1 timeout: 49 CPT: "[0]" global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 Actual: net: - net type: tcp231 local NI(s): - interfaces: 0: eth0 tunables: peer_timeout: 180 peer_credits: 8 peer_buffer_credits: 0 credits: 256 lnd tunables: conns_per_peer: 1 timeout: 49 CPT: "[0]" global: numa_range: 0 max_interfaces: 200 discovery: 1 drop_asym_route: 0 retry_count: 2 transaction_timeout: 150 health_sensitivity: 100 recovery_interval: 1 router_sensitivity: 100 lnd_timeout: 49 response_tracking: 3 recovery_limit: 0 max_recovery_ping_interval: 900 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 PASS 231 (3s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg133-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 250: test that linux routes are added ========================================================== 20:12:50 (1712275970) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if eth0 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 PASS 250 (2s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg133-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 251: Define multiple kfi networks on single interface ========================================================== 20:12:55 (1712275975) SKIP: sanity-lnet test_251 Need kfi NETTYPE SKIP 251 (1s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg133-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 252: Ping to down peer should unlink quickly ========================================================== 20:12:58 (1712275978) Cleaning up LNet Writer error: failed to resolve Netlink family id unloading modules on: 'oleg133-server' oleg133-server: oleg133-server.virtnet: executing unload_modules_local oleg133-server: LNET unconfigure error 22: (null) modules unloaded. Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all oleg133-server: Writer error: failed to resolve Netlink family id oleg133-server: opening /dev/lnet failed: No such file or directory oleg133-server: hint: the kernel modules may not be loaded oleg133-server: IOC_LIBCFS_GET_NI error 2: No such file or directory pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 oleg133-server: oleg133-server.virtnet: executing load_lnet config_on_load=1 oleg133-server: Loading modules from /home/green/git/lustre-release/lustre oleg133-server: detected 4 online CPUs by sysfs oleg133-server: Force libcfs to create 2 CPU partitions oleg133-server: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.133@tcp discover: - primary nid: 192.168.201.133@tcp Multi-Rail: true peer_ni: - nid: 192.168.201.133@tcp net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.201.33@tcp status: up interfaces: 0: eth0 - primary nid: 192.168.201.133@tcp - nid: 192.168.201.133@tcp health stats: health value: 1000 debug=+net oleg133-server: oleg133-server.virtnet: executing unload_modules_local /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping --timeout 15 192.168.201.133@tcp manage: - ping: errno: -111 descr: ! 'failed to ping 192.168.201.133@tcp: Connection refused' Writer error: failed to resolve Netlink family id unloading modules on: 'oleg133-server' oleg133-server: oleg133-server.virtnet: executing unload_modules_local oleg133-server: LNET unconfigure error 22: (null) modules unloaded. pdsh@oleg133-client: oleg133-client: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-client: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 PASS 252 (9s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory SKIP: sanity-lnet test_253 skipping ALWAYS excluded test 253 SKIP: sanity-lnet test_254 skipping ALWAYS excluded test 254 error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg133-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 255: Use lnet routes param with pdsh syntax ========================================================== 20:13:10 (1712275990) Loading LNet and configuring DLC Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure Cleaning up LNet Writer error: failed to resolve Netlink family id unloading modules on: 'oleg133-server' oleg133-server: oleg133-server.virtnet: executing unload_modules_local oleg133-server: LNET unconfigure error 22: (null) modules unloaded. Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ../lnet/lnet/lnet options: 'networks="tcp(eth0)" routes="o2ib 192.168.201.[34-38]"' LNET configured pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 PASS 255 (4s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg133-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 300: packaged LNet UAPI headers can be compiled ========================================================== 20:13:16 (1712275996) Cleaning up LNet Writer error: failed to resolve Netlink family id unloading modules on: 'oleg133-server' oleg133-server: oleg133-server.virtnet: executing unload_modules_local oleg133-server: LNET unconfigure error 22: (null) modules unloaded. Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 PASS 300 (12s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg133-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 301: Check for dynamic adds of same/wrong interface (memory leak) ========================================================== 20:13:30 (1712276010) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if eth0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if eth0 --- add: - net: errno: -17 descr: ! "Object exists" ... /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp --if eth0 Writer error: failed to resolve Netlink family id unloading modules on: 'oleg133-server' oleg133-server: oleg133-server.virtnet: executing unload_modules_local oleg133-server: LNET unconfigure error 22: (null) modules unloaded. pdsh@oleg133-client: oleg133-client: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-client: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 PASS 301 (4s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg133-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 302: Check that peer debug info can be dumped ========================================================== 20:13:36 (1712276016) Loading LNet and configuring DLC Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if eth0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl ping 192.168.201.33@tcp ping: - primary nid: 192.168.201.33@tcp Multi-Rail: false peer_ni: - nid: 192.168.201.33@tcp /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl debug peer --nid 192.168.201.33@tcp pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 PASS 302 (3s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg133-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 303: Check peer NI health after link down ========================================================== 20:13:42 (1712276022) Cleaning up LNet Writer error: failed to resolve Netlink family id unloading modules on: 'oleg133-server' oleg133-server: oleg133-server.virtnet: executing unload_modules_local oleg133-server: LNET unconfigure error 22: (null) modules unloaded. Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all oleg133-server: Writer error: failed to resolve Netlink family id oleg133-server: opening /dev/lnet failed: No such file or directory oleg133-server: hint: the kernel modules may not be loaded oleg133-server: IOC_LIBCFS_GET_NI error 2: No such file or directory pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 oleg133-server: oleg133-server.virtnet: executing load_lnet config_on_load=1 oleg133-server: Loading modules from /home/green/git/lustre-release/lustre oleg133-server: detected 4 online CPUs by sysfs oleg133-server: Force libcfs to create 2 CPU partitions oleg133-server: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.133@tcp discover: - primary nid: 192.168.201.133@tcp Multi-Rail: true peer_ni: - nid: 192.168.201.133@tcp oleg133-server: oleg133-server.virtnet: executing lnet_if_list oleg133-server: oleg133-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure oleg133-server: oleg133-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if eth0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if eth0 net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.201.33@tcp status: up interfaces: 0: eth0 - net type: tcp1 local NI(s): - nid: 192.168.201.33@tcp1 status: up interfaces: 0: eth0 - primary nid: 192.168.201.133@tcp - nid: 192.168.201.133@tcp health stats: health value: 1000 - nid: 192.168.201.133@tcp1 health stats: health value: 1000 debug=+net /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp99 --if test1pg Set test1pg down ip link set dev test1pg down NI 10.1.2.3@tcp99 expect status "down" found "down" PASS 303 (10s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg133-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 304: Check locked primary peer nid consolidation ========================================================== 20:13:54 (1712276034) Cleaning up LNet Writer error: failed to resolve Netlink family id unloading modules on: 'oleg133-server' oleg133-server: oleg133-server.virtnet: executing unload_modules_local modules unloaded. Loading LNet and configuring DLC Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if eth0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg NI 192.168.201.33@tcp expect status "up" found "up" NI 10.1.2.3@tcp expect status "up" found "up" /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim_nid 10.1.2.3@tcp --lock_prim /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --nid 10.1.2.3@tcp --state 1048577 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.33@tcp discover: - primary nid: 10.1.2.3@tcp Multi-Rail: true peer_ni: - nid: 192.168.201.33@tcp - nid: 10.1.2.3@tcp Expected: peer: - primary nid: 10.1.2.3@tcp Multi-Rail: True peer ni: - nid: 192.168.201.33@tcp state: NA - nid: 10.1.2.3@tcp state: NA Actual: peer: - primary nid: 10.1.2.3@tcp Multi-Rail: true peer ni: - nid: 192.168.201.33@tcp state: NA - nid: 10.1.2.3@tcp state: NA pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 PASS 304 (8s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg133-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 310: Set timeout and verify ========== 20:14:04 (1712276044) Cleaning up LNet Writer error: failed to resolve Netlink family id unloading modules on: 'oleg133-server' oleg133-server: oleg133-server.virtnet: executing unload_modules_local oleg133-server: LNET unconfigure error 22: (null) modules unloaded. loading ../lnet/klnds/socklnd/ksocklnd sock_timeout=65 type tcp ../lnet/klnds/socklnd/ksocklnd options: 'sock_timeout=65' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all check parameter timeout value 65 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net show -v net: - net type: lo local NI(s): - nid: 0@lo status: up statistics: send_count: 0 recv_count: 0 drop_count: 0 tunables: peer_timeout: 0 peer_credits: 0 peer_buffer_credits: 0 credits: 0 dev cpt: 0 CPT: "[0]" - net type: tcp local NI(s): - nid: 192.168.201.33@tcp status: up interfaces: 0: eth0 statistics: send_count: 0 recv_count: 0 drop_count: 0 tunables: peer_timeout: 180 peer_credits: 8 peer_buffer_credits: 0 credits: 256 lnd tunables: conns_per_peer: 1 timeout: 65 dev cpt: -1 CPT: "[0]" pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 PASS 310 (5s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg133-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 311: Fail bulk put in send wait completion ========================================================== 20:14:11 (1712276051) SKIP: sanity-lnet test_311 Need kfi network type SKIP 311 (1s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg133-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 312: TAG_RX_OK is possible after TX_FAIL ========================================================== 20:14:15 (1712276055) SKIP: sanity-lnet test_312 Need kfi network type SKIP 312 (1s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg133-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 400: Check for udsp add/delete net rule without net num ========================================================== 20:14:18 (1712276058) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --src tcp --priority 0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp del --idx 0 Writer error: failed to resolve Netlink family id unloading modules on: 'oleg133-server' oleg133-server: oleg133-server.virtnet: executing unload_modules_local oleg133-server: LNET unconfigure error 22: (null) modules unloaded. pdsh@oleg133-client: oleg133-client: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-client: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 PASS 400 (3s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg133-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 401: Discover peer after adding peer net UDSP rule ========================================================== 20:14:23 (1712276063) Loading LNet and configuring DLC Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if eth0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst tcp --prio 1 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.33@tcp discover: - primary nid: 192.168.201.33@tcp Multi-Rail: true peer_ni: - nid: 192.168.201.33@tcp 192.168.201.33@tcp: net_prio 1 expect 1 192.168.201.33@tcp: nid_prio -1 expect -1 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 PASS 401 (3s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg133-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 402: Destination net rule should not panic ========================================================== 20:14:28 (1712276068) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl udsp add --dst kfi --priority 0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer add --prim 402@kfi pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 PASS 402 (3s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg133-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 500: Check deadlock on ping target update ========================================================== 20:14:33 (1712276073) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet unconfigure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if eth0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp --if test1pg /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.33@tcp discover: - primary nid: 192.168.201.33@tcp Multi-Rail: true peer_ni: - nid: 192.168.201.33@tcp - nid: 10.1.2.3@tcp Added delay rule 255.255.255.255@tcp->255.255.255.255@tcp (1/1) Added drop rule 255.255.255.255@tcp->192.168.201.33@tcp (1/1) Added drop rule 255.255.255.255@tcp->10.1.2.3@tcp (1/1) Writer error: failed to resolve Netlink family id unloading modules on: 'oleg133-server' oleg133-server: oleg133-server.virtnet: executing unload_modules_local oleg133-server: LNET unconfigure error 22: (null) modules unloaded. pdsh@oleg133-client: oleg133-client: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-client: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 PASS 500 (6s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg133-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 501: Verify lnetctl peer set --health (SR) ========================================================== 20:14:41 (1712276081) Loading LNet and configuring DLC Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure Cleaning up LNet Writer error: failed to resolve Netlink family id unloading modules on: 'oleg133-server' oleg133-server: oleg133-server.virtnet: executing unload_modules_local oleg133-server: LNET unconfigure error 22: (null) modules unloaded. Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all oleg133-server: Writer error: failed to resolve Netlink family id oleg133-server: opening /dev/lnet failed: No such file or directory oleg133-server: hint: the kernel modules may not be loaded oleg133-server: IOC_LIBCFS_GET_NI error 2: No such file or directory pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 oleg133-server: oleg133-server.virtnet: executing load_lnet config_on_load=1 oleg133-server: Loading modules from /home/green/git/lustre-release/lustre oleg133-server: detected 4 online CPUs by sysfs oleg133-server: Force libcfs to create 2 CPU partitions oleg133-server: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.133@tcp discover: - primary nid: 192.168.201.133@tcp Multi-Rail: true peer_ni: - nid: 192.168.201.133@tcp net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.201.33@tcp status: up interfaces: 0: eth0 - primary nid: 192.168.201.133@tcp - nid: 192.168.201.133@tcp health stats: health value: 1000 debug=+net debug=-1 Added drop rule 255.255.255.255@tcp->255.255.255.255@tcp (1/1) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.201.133@tcp /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all --- set: - net: errno: 3 descr: ! "Object not found" ... Check that remote NI health has been changed hval: 1000 max_hval: 1000 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all --- set: - net: errno: 3 descr: ! "Object not found" ... Check that remote NI health has been changed hval: 1000 max_hval: 1000 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.201.133@tcp /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all --- set: - net: errno: 3 descr: ! "Object not found" ... Check that remote NI health has been changed hval: 1000 max_hval: 1000 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all --- set: - net: errno: 3 descr: ! "Object not found" ... Check that remote NI health has been changed hval: 1000 max_hval: 1000 Writer error: failed to resolve Netlink family id unloading modules on: 'oleg133-server' oleg133-server: oleg133-server.virtnet: executing unload_modules_local modules unloaded. oleg133-server: oleg133-server.virtnet: executing unload_modules_local oleg133-server: LNET unconfigure error 22: (null) pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-client: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-client: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 PASS 501 (11s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory error: get_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory oleg133-server: error: get_param: param_path 'debug_raw_pointers': No such file or directory pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='Y': No such file or directory == sanity-lnet test 502: Verify lnetctl peer set --health (MR) ========================================================== 20:14:54 (1712276094) Loading LNet and configuring DLC Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure Cleaning up LNet Writer error: failed to resolve Netlink family id unloading modules on: 'oleg133-server' oleg133-server: oleg133-server.virtnet: executing unload_modules_local oleg133-server: LNET unconfigure error 22: (null) modules unloaded. Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all oleg133-server: Writer error: failed to resolve Netlink family id oleg133-server: opening /dev/lnet failed: No such file or directory oleg133-server: hint: the kernel modules may not be loaded oleg133-server: IOC_LIBCFS_GET_NI error 2: No such file or directory pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 oleg133-server: oleg133-server.virtnet: executing load_lnet config_on_load=1 oleg133-server: Loading modules from /home/green/git/lustre-release/lustre oleg133-server: detected 4 online CPUs by sysfs oleg133-server: Force libcfs to create 2 CPU partitions oleg133-server: /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure --all /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl discover 192.168.201.133@tcp discover: - primary nid: 192.168.201.133@tcp Multi-Rail: true peer_ni: - nid: 192.168.201.133@tcp oleg133-server: oleg133-server.virtnet: executing lnet_if_list oleg133-server: oleg133-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure oleg133-server: oleg133-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if eth0 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl lnet configure /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp1 --if eth0 net: - net type: lo local NI(s): - nid: 0@lo status: up - net type: tcp local NI(s): - nid: 192.168.201.33@tcp status: up interfaces: 0: eth0 - net type: tcp1 local NI(s): - nid: 192.168.201.33@tcp1 status: up interfaces: 0: eth0 - primary nid: 192.168.201.133@tcp - nid: 192.168.201.133@tcp health stats: health value: 1000 - nid: 192.168.201.133@tcp1 health stats: health value: 1000 debug=+net debug=-1 Added drop rule 255.255.255.255@tcp->255.255.255.255@tcp (1/1) /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --nid 192.168.201.133@tcp /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all --- set: - net: errno: 3 descr: ! "Object not found" ... Check that remote NI health has been changed hval: 2000 max_hval: 2000 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 0 --all /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all --- set: - net: errno: 3 descr: ! "Object not found" ... Check that remote NI health has been changed hval: 2000 max_hval: 2000 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --nid 192.168.201.133@tcp /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all --- set: - net: errno: 3 descr: ! "Object not found" ... Check that remote NI health has been changed hval: 2000 max_hval: 2000 /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 500 --all /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl peer set --health 1000 --all /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net set --health 1000 --all --- set: - net: errno: 3 descr: ! "Object not found" ... Check that remote NI health has been changed hval: 2000 max_hval: 2000 oleg133-server: oleg133-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net del --net tcp1 --if eth0 Writer error: failed to resolve Netlink family id unloading modules on: 'oleg133-server' oleg133-server: oleg133-server.virtnet: executing unload_modules_local modules unloaded. oleg133-server: oleg133-server.virtnet: executing unload_modules_local oleg133-server: LNET unconfigure error 22: (null) pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-client: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-client: ssh exited with exit code 2 PASS 502 (15s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg133-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg133-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory == sanity-lnet test complete, duration 588 sec =========== 20:15:09 (1712276109) Cleaning up LNet LNET unconfigure error 22: (null) unloading modules on: 'oleg133-server' oleg133-server: oleg133-server.virtnet: executing unload_modules_local oleg133-server: LNET unconfigure error 22: (null) modules unloaded. Checking servers environments Checking clients oleg133-client.virtnet environments Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg133-server' oleg133-server: oleg133-server.virtnet: executing load_modules_local oleg133-server: Loading modules from /home/green/git/lustre-release/lustre oleg133-server: detected 4 online CPUs by sysfs oleg133-server: Force libcfs to create 2 CPU partitions oleg133-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg133-server: quota/lquota options: 'hash_lqs_cur_bits=3' Setup mgs, mdt, osts Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg133-server: oleg133-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 Started lustre-MDT0000 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg133-server: oleg133-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 Started lustre-OST0000 Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg133-server: oleg133-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 Started lustre-OST0001 Starting client: oleg133-client.virtnet: -o user_xattr,flock oleg133-server@tcp:/lustre /mnt/lustre Starting client oleg133-client.virtnet: -o user_xattr,flock oleg133-server@tcp:/lustre /mnt/lustre Started clients oleg133-client.virtnet: 192.168.201.133@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800b5c37000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800b5c37000.idle_timeout=debug disable quota as required