-----============= acceptance-small: conf-sanity ============----- Thu May 1 16:51:27 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg155-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32b 32c 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a Stopping clients: oleg155-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg155-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg155-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg155-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg155-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg155-server oleg155-server: oleg155-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory loading modules on: 'oleg155-server' oleg155-server: oleg155-server.virtnet: executing load_modules_local oleg155-server: Loading modules from /home/green/git/lustre-release/lustre oleg155-server: detected 4 online CPUs by sysfs oleg155-server: MODOPTS_LIBCFS= oleg155-server: Force libcfs to create 2 CPU partitions oleg155-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg155-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg155-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg155-server: oleg155-server.virtnet: executing set_default_debug -1 all pdsh@oleg155-client: oleg155-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg155-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg155-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg155-server: oleg155-server.virtnet: executing set_default_debug -1 all pdsh@oleg155-client: oleg155-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg155-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg155-server: oleg155-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg155-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg155-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg155-server stop mds service on oleg155-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg155-server == conf-sanity test 45: long unlink handling in ptlrpcd == 16:53:30 (1746132810) start mds service on oleg155-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg155-server: oleg155-server.virtnet: executing set_default_debug -1 all pdsh@oleg155-client: oleg155-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg155-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg155-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg155-server: oleg155-server.virtnet: executing set_default_debug -1 all pdsh@oleg155-client: oleg155-server: ssh exited with exit code 1 Started lustre-OST0000 oleg155-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg155-client.virtnet: -o user_xattr,flock 192.168.201.155@tcp:/lustre /mnt/lustre setup single mount lustre success stop mds service on oleg155-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg155-server sleep 60 sec fail_loc=0x8000050f sleep 10 sec manual umount lustre on /mnt/lustre.... df: '/mnt/lustre': Cannot send after transport endpoint shutdown fail_loc=0x0 start mds service on oleg155-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg155-server: oleg155-server.virtnet: executing set_default_debug -1 all pdsh@oleg155-client: oleg155-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg155-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg155-client.virtnet: -o user_xattr,flock 192.168.201.155@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg155-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg155-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg155-server stop mds service on oleg155-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg155-server unloading modules via unload_modules_local on: 'oleg155-server' oleg155-server: oleg155-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg155-client: oleg155-client: ssh exited with exit code 2 pdsh@oleg155-client: oleg155-server: ssh exited with exit code 2 pdsh@oleg155-client: oleg155-client: ssh exited with exit code 2 pdsh@oleg155-client: oleg155-server: ssh exited with exit code 2 PASS 45 (151s) == conf-sanity test 69: replace an OST with the same index ========================================================== 16:56:02 (1746132962) start mds service on oleg155-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg155-server' oleg155-server: oleg155-server.virtnet: executing load_modules_local oleg155-server: Loading modules from /home/green/git/lustre-release/lustre oleg155-server: detected 4 online CPUs by sysfs oleg155-server: MODOPTS_LIBCFS= oleg155-server: Force libcfs to create 2 CPU partitions oleg155-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg155-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg155-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg155-server: oleg155-server.virtnet: executing set_default_debug -1 all pdsh@oleg155-client: oleg155-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg155-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg155-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg155-server: oleg155-server.virtnet: executing set_default_debug -1 all pdsh@oleg155-client: oleg155-server: ssh exited with exit code 1 Started lustre-OST0000 oleg155-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg155-client.virtnet: -o user_xattr,flock 192.168.201.155@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 1106 (time 1746133008.75 total 10.00 last 110.55) - open/close 2218 (time 1746133018.75 total 20.01 last 111.16) - open/close 3519 (time 1746133028.76 total 30.02 last 130.01) - open/close 4664 (time 1746133038.76 total 40.02 last 114.48) - open/close 5704 (time 1746133048.77 total 50.03 last 103.91) - open/close 6864 (time 1746133058.78 total 60.03 last 115.91) - open/close 8204 (time 1746133068.78 total 70.04 last 133.90) - open/close 9548 (time 1746133078.79 total 80.05 last 134.31) total: 10000 open/close in 83.66 seconds: 119.53 ops/second - unlinked 0 (time 1746133084 ; total 0 ; last 0) total: 10000 unlinks in 54 seconds: 185.185181 unlinks/second - open/close 1057 (time 1746133150.21 total 10.00 last 105.65) - open/close 2310 (time 1746133160.21 total 20.01 last 125.24) - open/close 3423 (time 1746133170.22 total 30.02 last 111.22) - open/close 4457 (time 1746133180.23 total 40.03 last 103.27) - open/close 5612 (time 1746133190.25 total 50.05 last 115.31) - open/close 6831 (time 1746133200.25 total 60.05 last 121.87) - open/close 7972 (time 1746133210.26 total 70.05 last 114.05) - open/close 9084 (time 1746133220.27 total 80.06 last 111.08) total: 10000 open/close in 87.88 seconds: 113.79 ops/second - unlinked 0 (time 1746133229 ; total 0 ; last 0) total: 10000 unlinks in 60 seconds: 166.666672 unlinks/second - open/close 994 (time 1746133301.41 total 10.01 last 99.30) - open/close 1964 (time 1746133311.41 total 20.01 last 96.95) - open/close 3010 (time 1746133321.43 total 30.03 last 104.42) - open/close 4102 (time 1746133331.43 total 40.03 last 109.17) - open/close 5080 (time 1746133341.44 total 50.04 last 97.70) - open/close 6080 (time 1746133351.47 total 60.07 last 99.73) - open/close 6975 (time 1746133361.47 total 70.07 last 89.49) - open/close 8084 (time 1746133371.48 total 80.08 last 110.87) - open/close 8972 (time 1746133381.49 total 90.09 last 88.68) - open/close 9976 (time 1746133391.49 total 100.09 last 100.38) total: 10000 open/close in 100.26 seconds: 99.74 ops/second - unlinked 0 (time 1746133393 ; total 0 ; last 0) total: 10000 unlinks in 63 seconds: 158.730164 unlinks/second - open/close 1157 (time 1746133468.18 total 10.02 last 115.52) - open/close 2182 (time 1746133478.19 total 20.03 last 102.38) - open/close 3238 (time 1746133488.20 total 30.03 last 105.57) - open/close 4285 (time 1746133498.20 total 40.03 last 104.70) - open/close 5268 (time 1746133508.21 total 50.04 last 98.20) - open/close 6152 (time 1746133518.22 total 60.05 last 88.33) - open/close 7089 (time 1746133528.22 total 70.06 last 93.64) - open/close 8156 (time 1746133538.24 total 80.07 last 106.54) - open/close 9270 (time 1746133548.24 total 90.07 last 111.40) total: 10000 open/close in 96.96 seconds: 103.13 ops/second - unlinked 0 (time 1746133557 ; total 0 ; last 0) total: 10000 unlinks in 63 seconds: 158.730164 unlinks/second - open/close 992 (time 1746133632.65 total 10.00 last 99.20) - open/close 2032 (time 1746133642.65 total 20.00 last 103.96) - open/close 3111 (time 1746133652.66 total 30.01 last 107.86) - open/close 4067 (time 1746133662.68 total 40.02 last 95.44) - open/close 5010 (time 1746133672.68 total 50.03 last 94.25) - open/close 5986 (time 1746133682.69 total 60.04 last 97.55) - open/close 6936 (time 1746133692.69 total 70.04 last 94.93) - open/close 7737 (time 1746133702.70 total 80.05 last 80.07) - open/close 8699 (time 1746133712.70 total 90.05 last 96.13) - open/close 9667 (time 1746133722.72 total 100.07 last 96.69) total: 10000 open/close in 103.20 seconds: 96.89 ops/second - unlinked 0 (time 1746133728 ; total 0 ; last 0) total: 10000 unlinks in 53 seconds: 188.679245 unlinks/second - open/close 1197 (time 1746133793.70 total 10.01 last 119.59) - open/close 2363 (time 1746133803.70 total 20.01 last 116.56) - open/close 3495 (time 1746133813.70 total 30.01 last 113.18) - open/close 4551 (time 1746133823.71 total 40.02 last 105.50) - open/close 5559 (time 1746133833.72 total 50.03 last 100.73) - open/close 6625 (time 1746133843.72 total 60.03 last 106.58) - open/close 7851 (time 1746133853.73 total 70.04 last 122.48) - open/close 8975 (time 1746133863.74 total 80.05 last 112.33) total: 10000 open/close in 88.42 seconds: 113.10 ops/second - unlinked 0 (time 1746133874 ; total 0 ; last 0) total: 10000 unlinks in 47 seconds: 212.765961 unlinks/second - open/close 1464 (time 1746133933.97 total 10.01 last 146.24) - open/close 2679 (time 1746133943.97 total 20.02 last 121.43) - open/close 3720 (time 1746133953.98 total 30.02 last 104.05) - open/close 4907 (time 1746133963.98 total 40.02 last 118.67) - open/close 6057 (time 1746133973.98 total 50.03 last 114.93) - open/close 7178 (time 1746133984.02 total 60.06 last 111.72) - open/close 8225 (time 1746133994.02 total 70.06 last 104.70) - open/close 9484 (time 1746134004.02 total 80.07 last 125.82) total: 10000 open/close in 85.40 seconds: 117.09 ops/second - unlinked 0 (time 1746134011 ; total 0 ; last 0) total: 10000 unlinks in 42 seconds: 238.095245 unlinks/second - open/close 1242 (time 1746134066.05 total 10.00 last 124.16) - open/close 2467 (time 1746134076.05 total 20.00 last 122.48) - open/close 3637 (time 1746134086.06 total 30.01 last 116.89) - open/close 4746 (time 1746134096.07 total 40.02 last 110.80) - open/close 5809 (time 1746134106.08 total 50.03 last 106.22) - open/close 6811 (time 1746134116.08 total 60.03 last 100.16) - open/close 7982 (time 1746134126.09 total 70.04 last 117.03) - open/close 9272 (time 1746134136.09 total 80.05 last 128.90) total: 10000 open/close in 86.41 seconds: 115.73 ops/second - unlinked 0 (time 1746134144 ; total 0 ; last 0) total: 10000 unlinks in 49 seconds: 204.081635 unlinks/second - open/close 1278 (time 1746134205.59 total 10.01 last 127.73) - open/close 2509 (time 1746134215.59 total 20.01 last 123.05) - open/close 3568 (time 1746134225.60 total 30.02 last 105.78) - open/close 4653 (time 1746134235.61 total 40.03 last 108.40) - open/close 5819 (time 1746134245.61 total 50.03 last 116.57) - open/close 6926 (time 1746134255.62 total 60.03 last 110.68) - open/close 8099 (time 1746134265.62 total 70.03 last 117.30) - open/close 9107 (time 1746134275.62 total 80.04 last 100.72) total: 10000 open/close in 87.29 seconds: 114.56 ops/second - unlinked 0 (time 1746134284 ; total 0 ; last 0) total: 10000 unlinks in 62 seconds: 161.290329 unlinks/second - open/close 1056 (time 1746134359.16 total 10.00 last 105.56) - open/close 2118 (time 1746134369.16 total 20.00 last 106.19) - open/close 3229 (time 1746134379.16 total 30.01 last 111.04) - open/close 4427 (time 1746134389.17 total 40.02 last 119.71) - open/close 5607 (time 1746134399.18 total 50.02 last 117.93) - open/close 6746 (time 1746134409.19 total 60.04 last 113.74) - open/close 7613 (time 1746134419.20 total 70.05 last 86.63) - open/close 8508 (time 1746134429.21 total 80.06 last 89.41) - open/close 9352 (time 1746134439.21 total 90.06 last 84.35) total: 10000 open/close in 96.93 seconds: 103.16 ops/second - unlinked 0 (time 1746134448 ; total 0 ; last 0) total: 10000 unlinks in 57 seconds: 175.438599 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg155-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg155-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg155-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.201.155@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.201.155@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg155-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg155-server: oleg155-server.virtnet: executing set_default_debug -1 all pdsh@oleg155-client: oleg155-server: ssh exited with exit code 1 Started lustre-OST0000 oleg155-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg155-server: oleg155-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg155-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec mount lustre on /mnt/lustre..... Starting client: oleg155-client.virtnet: -o user_xattr,flock 192.168.201.155@tcp:/lustre /mnt/lustre On OST0, 13229 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg155-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg155-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg155-server stop mds service on oleg155-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg155-server unloading modules via unload_modules_local on: 'oleg155-server' oleg155-server: oleg155-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg155-client: oleg155-server: ssh exited with exit code 2 pdsh@oleg155-client: oleg155-client: ssh exited with exit code 2 pdsh@oleg155-client: oleg155-client: ssh exited with exit code 2 pdsh@oleg155-client: oleg155-server: ssh exited with exit code 2 PASS 69 (1608s) == conf-sanity test 111: Adding large_dir with over 2GB directory ========================================================== 17:22:51 (1746134571) SKIP: conf-sanity test_111 Only applicable to ldiskfs-based MDTs SKIP 111 (3s) Stopping clients: oleg155-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg155-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg155-client: oleg155-server: ssh exited with exit code 2 oleg155-server: oleg155-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg155-server' oleg155-server: oleg155-server.virtnet: executing load_modules_local oleg155-server: Loading modules from /home/green/git/lustre-release/lustre oleg155-server: detected 4 online CPUs by sysfs oleg155-server: MODOPTS_LIBCFS= oleg155-server: Force libcfs to create 2 CPU partitions oleg155-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg155-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 1928 sec ========== 17:23:36 (1746134616) === conf-sanity: start cleanup 17:23:37 (1746134617) === === conf-sanity: finish cleanup 17:23:40 (1746134620) ===