-----============= acceptance-small: conf-sanity ============----- Thu May 1 00:01:22 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg242-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32b 32c 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a Stopping clients: oleg242-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg242-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg242-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg242-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg242-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg242-server oleg242-server: oleg242-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg242-server' oleg242-server: oleg242-server.virtnet: executing load_modules_local oleg242-server: Loading modules from /home/green/git/lustre-release/lustre oleg242-server: detected 4 online CPUs by sysfs oleg242-server: MODOPTS_LIBCFS= oleg242-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg242-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg242-server: oleg242-server.virtnet: executing set_default_debug -1 all pdsh@oleg242-client: oleg242-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg242-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg242-server: oleg242-server.virtnet: executing set_default_debug -1 all pdsh@oleg242-client: oleg242-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg242-server: oleg242-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg242-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg242-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg242-server stop mds service on oleg242-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg242-server == conf-sanity test 45: long unlink handling in ptlrpcd == 00:02:46 (1746072166) start mds service on oleg242-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg242-server: oleg242-server.virtnet: executing set_default_debug -1 all pdsh@oleg242-client: oleg242-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg242-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg242-server: oleg242-server.virtnet: executing set_default_debug -1 all pdsh@oleg242-client: oleg242-server: ssh exited with exit code 1 Started lustre-OST0000 oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg242-client.virtnet: -o user_xattr,flock 192.168.202.142@tcp:/lustre /mnt/lustre setup single mount lustre success stop mds service on oleg242-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg242-server sleep 60 sec fail_loc=0x8000050f sleep 10 sec manual umount lustre on /mnt/lustre.... df: '/mnt/lustre': Cannot send after transport endpoint shutdown fail_loc=0x0 start mds service on oleg242-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg242-server: oleg242-server.virtnet: executing set_default_debug -1 all pdsh@oleg242-client: oleg242-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg242-client.virtnet: -o user_xattr,flock 192.168.202.142@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg242-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg242-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg242-server stop mds service on oleg242-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg242-server unloading modules via unload_modules_local on: 'oleg242-server' oleg242-server: oleg242-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg242-client: oleg242-client: ssh exited with exit code 2 pdsh@oleg242-client: oleg242-server: ssh exited with exit code 2 pdsh@oleg242-client: oleg242-client: ssh exited with exit code 2 pdsh@oleg242-client: oleg242-server: ssh exited with exit code 2 PASS 45 (140s) == conf-sanity test 69: replace an OST with the same index ========================================================== 00:05:06 (1746072306) start mds service on oleg242-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg242-server' oleg242-server: oleg242-server.virtnet: executing load_modules_local oleg242-server: Loading modules from /home/green/git/lustre-release/lustre oleg242-server: detected 4 online CPUs by sysfs oleg242-server: MODOPTS_LIBCFS= oleg242-server: Force libcfs to create 2 CPU partitions oleg242-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg242-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg242-server: oleg242-server.virtnet: executing set_default_debug -1 all pdsh@oleg242-client: oleg242-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg242-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg242-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg242-server: oleg242-server.virtnet: executing set_default_debug -1 all pdsh@oleg242-client: oleg242-server: ssh exited with exit code 1 Started lustre-OST0000 oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg242-client.virtnet: -o user_xattr,flock 192.168.202.142@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 1782 (time 1746072347.68 total 10.00 last 178.16) - open/close 3988 (time 1746072357.68 total 20.00 last 220.60) - open/close 6227 (time 1746072367.68 total 30.00 last 223.90) - open/close 8408 (time 1746072377.68 total 40.00 last 218.09) total: 10000 open/close in 46.39 seconds: 215.55 ops/second - unlinked 0 (time 1746072385 ; total 0 ; last 0) total: 10000 unlinks in 25 seconds: 400.000000 unlinks/second - open/close 2087 (time 1746072421.74 total 10.00 last 208.67) - open/close 4391 (time 1746072431.75 total 20.00 last 230.34) - open/close 7388 (time 1746072441.75 total 30.01 last 299.64) - open/close 9428 (time 1746072451.75 total 40.01 last 203.99) total: 10000 open/close in 42.85 seconds: 233.39 ops/second - unlinked 0 (time 1746072455 ; total 0 ; last 0) total: 10000 unlinks in 22 seconds: 454.545441 unlinks/second - open/close 3393 (time 1746072488.43 total 10.00 last 339.24) - open/close 6189 (time 1746072498.43 total 20.00 last 279.56) - open/close 9462 (time 1746072508.43 total 30.00 last 327.27) total: 10000 open/close in 31.60 seconds: 316.46 ops/second - unlinked 0 (time 1746072511 ; total 1 ; last 1) total: 10000 unlinks in 15 seconds: 666.666687 unlinks/second - open/close 2865 (time 1746072536.29 total 10.00 last 286.47) - open/close 5358 (time 1746072546.29 total 20.00 last 249.28) - open/close 8453 (time 1746072556.29 total 30.00 last 309.49) total: 10000 open/close in 34.80 seconds: 287.33 ops/second - unlinked 0 (time 1746072562 ; total 0 ; last 0) total: 10000 unlinks in 20 seconds: 500.000000 unlinks/second - open/close 2346 (time 1746072593.98 total 10.00 last 234.60) - open/close 5241 (time 1746072603.98 total 20.00 last 289.40) - open/close 6775 (time 1746072613.98 total 30.00 last 153.39) - open/close 8178 (time 1746072623.98 total 40.01 last 140.23) total: 10000 open/close in 48.73 seconds: 205.20 ops/second - unlinked 0 (time 1746072633 ; total 0 ; last 0) total: 10000 unlinks in 22 seconds: 454.545441 unlinks/second - open/close 2241 (time 1746072666.30 total 10.00 last 224.09) - open/close 5030 (time 1746072676.30 total 20.00 last 278.88) - open/close 8022 (time 1746072686.30 total 30.00 last 299.15) total: 10000 open/close in 34.65 seconds: 288.58 ops/second - unlinked 0 (time 1746072691 ; total 0 ; last 0) total: 10000 unlinks in 25 seconds: 400.000000 unlinks/second - open/close 1824 (time 1746072727.60 total 10.01 last 182.28) - open/close 3156 (time 1746072737.60 total 20.01 last 133.15) - open/close 4385 (time 1746072747.61 total 30.02 last 122.80) - open/close 5861 (time 1746072757.61 total 40.02 last 147.57) - open/close 7647 (time 1746072767.62 total 50.02 last 178.55) - open/close 9354 (time 1746072777.62 total 60.03 last 170.63) total: 10000 open/close in 63.69 seconds: 157.01 ops/second - unlinked 0 (time 1746072782 ; total 0 ; last 0) total: 10000 unlinks in 34 seconds: 294.117645 unlinks/second - open/close 1810 (time 1746072828.17 total 10.00 last 180.95) - open/close 3367 (time 1746072838.17 total 20.00 last 155.70) - open/close 4975 (time 1746072848.17 total 30.00 last 160.77) - open/close 6684 (time 1746072858.17 total 40.01 last 170.86) - open/close 9151 (time 1746072868.18 total 50.02 last 246.43) total: 10000 open/close in 54.06 seconds: 184.99 ops/second - unlinked 0 (time 1746072873 ; total 0 ; last 0) total: 10000 unlinks in 33 seconds: 303.030304 unlinks/second - open/close 1859 (time 1746072917.20 total 10.00 last 185.87) - open/close 3356 (time 1746072927.20 total 20.01 last 149.65) - open/close 4784 (time 1746072937.21 total 30.01 last 142.73) - open/close 6182 (time 1746072947.21 total 40.02 last 139.69) - open/close 7890 (time 1746072957.21 total 50.02 last 170.79) - open/close 9809 (time 1746072967.21 total 60.02 last 191.88) total: 10000 open/close in 61.03 seconds: 163.86 ops/second - unlinked 0 (time 1746072969 ; total 0 ; last 0) total: 10000 unlinks in 33 seconds: 303.030304 unlinks/second - open/close 1516 (time 1746073013.92 total 10.00 last 151.57) - open/close 3080 (time 1746073023.93 total 20.01 last 156.34) - open/close 4553 (time 1746073033.93 total 30.01 last 147.23) - open/close 6316 (time 1746073043.93 total 40.01 last 176.30) - open/close 7609 (time 1746073053.93 total 50.01 last 129.26) - open/close 9247 (time 1746073063.94 total 60.01 last 163.79) total: 10000 open/close in 64.23 seconds: 155.68 ops/second - unlinked 0 (time 1746073069 ; total 0 ; last 0) total: 10000 unlinks in 37 seconds: 270.270264 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg242-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg242-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg242-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.202.142@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.202.142@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg242-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg242-server: oleg242-server.virtnet: executing set_default_debug -1 all pdsh@oleg242-client: oleg242-server: ssh exited with exit code 1 Started lustre-OST0000 oleg242-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg242-server: oleg242-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg242-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec mount lustre on /mnt/lustre..... Starting client: oleg242-client.virtnet: -o user_xattr,flock 192.168.202.142@tcp:/lustre /mnt/lustre On OST0, 12693 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg242-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg242-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg242-server stop mds service on oleg242-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg242-server unloading modules via unload_modules_local on: 'oleg242-server' oleg242-server: oleg242-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg242-client: oleg242-client: ssh exited with exit code 2 pdsh@oleg242-client: oleg242-server: ssh exited with exit code 2 pdsh@oleg242-client: oleg242-client: ssh exited with exit code 2 pdsh@oleg242-client: oleg242-server: ssh exited with exit code 2 PASS 69 (845s) == conf-sanity test 111: Adding large_dir with over 2GB directory ========================================================== 00:19:11 (1746073151) SKIP: conf-sanity test_111 Only applicable to ldiskfs-based MDTs SKIP 111 (1s) Stopping clients: oleg242-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg242-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg242-client: oleg242-server: ssh exited with exit code 2 oleg242-server: oleg242-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg242-server' oleg242-server: oleg242-server.virtnet: executing load_modules_local oleg242-server: Loading modules from /home/green/git/lustre-release/lustre oleg242-server: detected 4 online CPUs by sysfs oleg242-server: MODOPTS_LIBCFS= oleg242-server: Force libcfs to create 2 CPU partitions oleg242-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory oleg242-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg242-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 1098 sec ========== 00:19:40 (1746073180) === conf-sanity: start cleanup 00:19:41 (1746073181) === === conf-sanity: finish cleanup 00:19:42 (1746073182) ===