-----============= acceptance-small: conf-sanity ============----- Thu May 1 14:59:26 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg210-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32b 32c 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a Stopping clients: oleg210-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg210-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg210-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg210-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg210-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg210-server oleg210-server: oleg210-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory loading modules on: 'oleg210-server' oleg210-server: oleg210-server.virtnet: executing load_modules_local oleg210-server: Loading modules from /home/green/git/lustre-release/lustre oleg210-server: detected 4 online CPUs by sysfs oleg210-server: MODOPTS_LIBCFS= oleg210-server: Force libcfs to create 2 CPU partitions oleg210-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory oleg210-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg210-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg210-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg210-server: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg210-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg210-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg210-server stop mds service on oleg210-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg210-server == conf-sanity test 45: long unlink handling in ptlrpcd == 15:00:14 (1746126014) start mds service on oleg210-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg210-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-OST0000 oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg210-client.virtnet: -o user_xattr,flock 192.168.202.110@tcp:/lustre /mnt/lustre setup single mount lustre success stop mds service on oleg210-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg210-server sleep 60 sec fail_loc=0x8000050f sleep 10 sec manual umount lustre on /mnt/lustre.... df: '/mnt/lustre': Cannot send after transport endpoint shutdown fail_loc=0x0 start mds service on oleg210-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg210-client.virtnet: -o user_xattr,flock 192.168.202.110@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg210-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg210-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg210-server stop mds service on oleg210-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg210-server unloading modules via unload_modules_local on: 'oleg210-server' oleg210-server: oleg210-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg210-client: oleg210-client: ssh exited with exit code 2 pdsh@oleg210-client: oleg210-server: ssh exited with exit code 2 pdsh@oleg210-client: oleg210-client: ssh exited with exit code 2 pdsh@oleg210-client: oleg210-server: ssh exited with exit code 2 PASS 45 (133s) == conf-sanity test 69: replace an OST with the same index ========================================================== 15:02:28 (1746126148) start mds service on oleg210-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg210-server' oleg210-server: oleg210-server.virtnet: executing load_modules_local oleg210-server: Loading modules from /home/green/git/lustre-release/lustre oleg210-server: detected 4 online CPUs by sysfs oleg210-server: MODOPTS_LIBCFS= oleg210-server: Force libcfs to create 2 CPU partitions oleg210-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory oleg210-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg210-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg210-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg210-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-OST0000 oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg210-client.virtnet: -o user_xattr,flock 192.168.202.110@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 1148 (time 1746126191.81 total 10.00 last 114.77) - open/close 2673 (time 1746126201.81 total 20.00 last 152.49) - open/close 4194 (time 1746126211.81 total 30.00 last 152.10) - open/close 5686 (time 1746126221.81 total 40.01 last 149.14) - open/close 6859 (time 1746126231.82 total 50.01 last 117.26) - open/close 8276 (time 1746126241.82 total 60.01 last 141.66) - open/close 9544 (time 1746126251.82 total 70.02 last 126.72) total: 10000 open/close in 74.07 seconds: 135.02 ops/second - unlinked 0 (time 1746126257 ; total 0 ; last 0) total: 10000 unlinks in 57 seconds: 175.438599 unlinks/second - open/close 1203 (time 1746126326.12 total 10.01 last 120.21) - open/close 2301 (time 1746126336.12 total 20.01 last 109.78) - open/close 3546 (time 1746126346.13 total 30.01 last 124.47) - open/close 4751 (time 1746126356.13 total 40.02 last 120.40) - open/close 6178 (time 1746126366.14 total 50.02 last 142.66) - open/close 7603 (time 1746126376.14 total 60.02 last 142.45) - open/close 8873 (time 1746126386.14 total 70.03 last 126.97) total: 10000 open/close in 79.14 seconds: 126.36 ops/second - unlinked 0 (time 1746126396 ; total 0 ; last 0) total: 10000 unlinks in 38 seconds: 263.157898 unlinks/second - open/close 1561 (time 1746126445.48 total 10.00 last 156.05) - open/close 3189 (time 1746126455.49 total 20.02 last 162.58) - open/close 4568 (time 1746126465.50 total 30.02 last 137.84) - open/close 5825 (time 1746126475.50 total 40.03 last 125.60) - open/close 7056 (time 1746126485.50 total 50.03 last 123.09) - open/close 8370 (time 1746126495.52 total 60.04 last 131.24) - open/close 9454 (time 1746126505.53 total 70.05 last 108.29) total: 10000 open/close in 75.57 seconds: 132.34 ops/second - unlinked 0 (time 1746126513 ; total 0 ; last 0) total: 10000 unlinks in 53 seconds: 188.679245 unlinks/second - open/close 1440 (time 1746126578.45 total 10.00 last 143.95) - open/close 2637 (time 1746126588.45 total 20.01 last 119.65) - open/close 4154 (time 1746126598.45 total 30.01 last 151.64) - open/close 5508 (time 1746126608.45 total 40.01 last 135.40) - open/close 7040 (time 1746126618.46 total 50.02 last 153.14) - open/close 8441 (time 1746126628.48 total 60.03 last 139.85) - open/close 9815 (time 1746126638.48 total 70.04 last 137.35) total: 10000 open/close in 71.28 seconds: 140.30 ops/second - unlinked 0 (time 1746126641 ; total 0 ; last 0) total: 10000 unlinks in 35 seconds: 285.714294 unlinks/second - open/close 1344 (time 1746126688.46 total 10.00 last 134.35) - open/close 2648 (time 1746126698.46 total 20.00 last 130.39) - open/close 4296 (time 1746126708.47 total 30.01 last 164.77) - open/close 5860 (time 1746126718.47 total 40.01 last 156.33) - open/close 7246 (time 1746126728.48 total 50.02 last 138.52) - open/close 8652 (time 1746126738.48 total 60.02 last 140.58) total: 10000 open/close in 69.89 seconds: 143.09 ops/second - unlinked 0 (time 1746126749 ; total 0 ; last 0) total: 10000 unlinks in 49 seconds: 204.081635 unlinks/second - open/close 1204 (time 1746126810.48 total 10.00 last 120.35) - open/close 2293 (time 1746126820.49 total 20.01 last 108.82) - open/close 3485 (time 1746126830.50 total 30.02 last 119.13) - open/close 5001 (time 1746126840.50 total 40.02 last 151.58) - open/close 6395 (time 1746126850.50 total 50.02 last 139.32) - open/close 7603 (time 1746126860.51 total 60.02 last 120.79) - open/close 8833 (time 1746126870.51 total 70.03 last 123.00) total: 10000 open/close in 79.89 seconds: 125.18 ops/second - unlinked 0 (time 1746126882 ; total 0 ; last 0) total: 10000 unlinks in 39 seconds: 256.410248 unlinks/second - open/close 1610 (time 1746126932.61 total 10.01 last 160.89) - open/close 3256 (time 1746126942.61 total 20.01 last 164.59) - open/close 4777 (time 1746126952.62 total 30.01 last 152.00) - open/close 6190 (time 1746126962.62 total 40.02 last 141.24) - open/close 7819 (time 1746126972.62 total 50.02 last 162.88) - open/close 9258 (time 1746126982.63 total 60.02 last 143.87) total: 10000 open/close in 66.04 seconds: 151.43 ops/second - unlinked 0 (time 1746126990 ; total 0 ; last 0) total: 10000 unlinks in 46 seconds: 217.391312 unlinks/second - open/close 1357 (time 1746127048.48 total 10.01 last 135.63) - open/close 2536 (time 1746127058.49 total 20.01 last 117.78) - open/close 3689 (time 1746127068.49 total 30.02 last 115.28) - open/close 5079 (time 1746127078.49 total 40.02 last 138.99) - open/close 6582 (time 1746127088.49 total 50.02 last 150.30) - open/close 7815 (time 1746127098.50 total 60.02 last 123.25) - open/close 9060 (time 1746127108.50 total 70.03 last 124.43) total: 10000 open/close in 75.96 seconds: 131.65 ops/second - unlinked 0 (time 1746127116 ; total 0 ; last 0) total: 10000 unlinks in 42 seconds: 238.095245 unlinks/second - open/close 1491 (time 1746127170.02 total 10.01 last 148.99) - open/close 2799 (time 1746127180.02 total 20.01 last 130.80) - open/close 4152 (time 1746127190.02 total 30.01 last 135.28) - open/close 5678 (time 1746127200.02 total 40.01 last 152.58) - open/close 7137 (time 1746127210.03 total 50.01 last 145.85) - open/close 8531 (time 1746127220.03 total 60.02 last 139.34) - open/close 9882 (time 1746127230.03 total 70.02 last 135.08) total: 10000 open/close in 70.85 seconds: 141.14 ops/second - unlinked 0 (time 1746127232 ; total 0 ; last 0) total: 10000 unlinks in 36 seconds: 277.777771 unlinks/second - open/close 1672 (time 1746127279.36 total 10.01 last 167.11) - open/close 3368 (time 1746127289.37 total 20.01 last 169.51) - open/close 4675 (time 1746127299.38 total 30.02 last 130.58) - open/close 5821 (time 1746127309.38 total 40.02 last 114.59) - open/close 7127 (time 1746127319.38 total 50.03 last 130.54) - open/close 8395 (time 1746127329.38 total 60.03 last 126.80) - open/close 9901 (time 1746127339.39 total 70.03 last 150.56) total: 10000 open/close in 70.77 seconds: 141.31 ops/second - unlinked 0 (time 1746127341 ; total 0 ; last 0) total: 10000 unlinks in 45 seconds: 222.222229 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg210-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg210-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg210-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.202.110@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.202.110@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg210-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg210-server: oleg210-server.virtnet: executing set_default_debug -1 all pdsh@oleg210-client: oleg210-server: ssh exited with exit code 1 Started lustre-OST0000 oleg210-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg210-server: oleg210-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg210-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 9 sec mount lustre on /mnt/lustre..... Starting client: oleg210-client.virtnet: -o user_xattr,flock 192.168.202.110@tcp:/lustre /mnt/lustre On OST0, 17701 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg210-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg210-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg210-server stop mds service on oleg210-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg210-server unloading modules via unload_modules_local on: 'oleg210-server' oleg210-server: oleg210-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg210-client: oleg210-client: ssh exited with exit code 2 pdsh@oleg210-client: oleg210-server: ssh exited with exit code 2 pdsh@oleg210-client: oleg210-server: ssh exited with exit code 2 pdsh@oleg210-client: oleg210-client: ssh exited with exit code 2 PASS 69 (1292s) == conf-sanity test 111: Adding large_dir with over 2GB directory ========================================================== 15:24:01 (1746127441) SKIP: conf-sanity test_111 Only applicable to ldiskfs-based MDTs SKIP 111 (2s) Stopping clients: oleg210-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg210-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg210-client: oleg210-server: ssh exited with exit code 2 oleg210-server: oleg210-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg210-server' oleg210-server: oleg210-server.virtnet: executing load_modules_local oleg210-server: Loading modules from /home/green/git/lustre-release/lustre oleg210-server: detected 4 online CPUs by sysfs oleg210-server: MODOPTS_LIBCFS= oleg210-server: Force libcfs to create 2 CPU partitions oleg210-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg210-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 1510 sec ========== 15:24:36 (1746127476) === conf-sanity: start cleanup 15:24:37 (1746127477) === === conf-sanity: finish cleanup 15:24:39 (1746127479) ===