-----============= acceptance-small: conf-sanity ============----- Thu May 1 16:17:25 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg440-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32b 32c 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a Stopping clients: oleg440-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg440-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg440-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg440-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg440-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg440-server oleg440-server: oleg440-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg440-server' oleg440-server: oleg440-server.virtnet: executing load_modules_local oleg440-server: Loading modules from /home/green/git/lustre-release/lustre oleg440-server: detected 4 online CPUs by sysfs oleg440-server: MODOPTS_LIBCFS= oleg440-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg440-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg440-server: oleg440-server.virtnet: executing set_default_debug -1 all pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg440-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg440-server: oleg440-server.virtnet: executing set_default_debug -1 all pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg440-server: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg440-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg440-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg440-server stop mds service on oleg440-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg440-server == conf-sanity test 45: long unlink handling in ptlrpcd == 16:18:26 (1746130706) start mds service on oleg440-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg440-server: oleg440-server.virtnet: executing set_default_debug -1 all pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg440-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg440-server: oleg440-server.virtnet: executing set_default_debug -1 all pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 Started lustre-OST0000 oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg440-client.virtnet: -o user_xattr,flock 192.168.204.140@tcp:/lustre /mnt/lustre setup single mount lustre success stop mds service on oleg440-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg440-server sleep 60 sec fail_loc=0x8000050f sleep 10 sec manual umount lustre on /mnt/lustre.... df: '/mnt/lustre': Cannot send after transport endpoint shutdown fail_loc=0x0 start mds service on oleg440-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg440-server: oleg440-server.virtnet: executing set_default_debug -1 all pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg440-client.virtnet: -o user_xattr,flock 192.168.204.140@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg440-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg440-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg440-server stop mds service on oleg440-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg440-server unloading modules via unload_modules_local on: 'oleg440-server' oleg440-server: oleg440-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg440-client: oleg440-server: ssh exited with exit code 2 pdsh@oleg440-client: oleg440-client: ssh exited with exit code 2 pdsh@oleg440-client: oleg440-server: ssh exited with exit code 2 pdsh@oleg440-client: oleg440-client: ssh exited with exit code 2 PASS 45 (150s) == conf-sanity test 69: replace an OST with the same index ========================================================== 16:20:56 (1746130856) start mds service on oleg440-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg440-server' oleg440-server: oleg440-server.virtnet: executing load_modules_local oleg440-server: Loading modules from /home/green/git/lustre-release/lustre oleg440-server: detected 4 online CPUs by sysfs oleg440-server: MODOPTS_LIBCFS= oleg440-server: Force libcfs to create 2 CPU partitions oleg440-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg440-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg440-server: oleg440-server.virtnet: executing set_default_debug -1 all pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg440-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg440-server: oleg440-server.virtnet: executing set_default_debug -1 all pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 Started lustre-OST0000 oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg440-client.virtnet: -o user_xattr,flock 192.168.204.140@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 1968 (time 1746130898.94 total 10.01 last 196.69) - open/close 3804 (time 1746130908.94 total 20.01 last 183.45) - open/close 5797 (time 1746130918.95 total 30.02 last 199.17) - open/close 7881 (time 1746130928.95 total 40.02 last 208.33) - open/close 9835 (time 1746130938.96 total 50.03 last 195.34) total: 10000 open/close in 50.82 seconds: 196.75 ops/second - unlinked 0 (time 1746130941 ; total 0 ; last 0) total: 10000 unlinks in 38 seconds: 263.157898 unlinks/second - open/close 1557 (time 1746130990.98 total 10.00 last 155.65) - open/close 3540 (time 1746131000.99 total 20.01 last 198.09) - open/close 5587 (time 1746131010.99 total 30.02 last 204.60) - open/close 8185 (time 1746131021.00 total 40.02 last 259.68) total: 10000 open/close in 49.47 seconds: 202.13 ops/second - unlinked 0 (time 1746131031 ; total 0 ; last 0) total: 10000 unlinks in 23 seconds: 434.782623 unlinks/second - open/close 2505 (time 1746131065.67 total 10.00 last 250.45) - open/close 4759 (time 1746131075.67 total 20.00 last 225.37) - open/close 6725 (time 1746131085.67 total 30.01 last 196.51) - open/close 8968 (time 1746131095.67 total 40.01 last 224.27) total: 10000 open/close in 46.08 seconds: 216.99 ops/second - unlinked 0 (time 1746131103 ; total 0 ; last 0) total: 10000 unlinks in 36 seconds: 277.777771 unlinks/second - open/close 1905 (time 1746131150.97 total 10.01 last 190.28) - open/close 3370 (time 1746131160.97 total 20.01 last 146.48) - open/close 5476 (time 1746131170.97 total 30.02 last 210.55) - open/close 7473 (time 1746131180.97 total 40.02 last 199.64) - open/close 8679 (time 1746131190.98 total 50.02 last 120.57) total: 10000 open/close in 59.43 seconds: 168.27 ops/second - unlinked 0 (time 1746131201 ; total 0 ; last 0) total: 10000 unlinks in 34 seconds: 294.117645 unlinks/second - open/close 2284 (time 1746131247.00 total 10.00 last 228.31) - open/close 4259 (time 1746131257.00 total 20.01 last 197.48) - open/close 6542 (time 1746131267.01 total 30.01 last 228.24) - open/close 9698 (time 1746131277.01 total 40.01 last 315.43) total: 10000 open/close in 41.43 seconds: 241.36 ops/second - unlinked 0 (time 1746131279 ; total 0 ; last 0) total: 10000 unlinks in 35 seconds: 285.714294 unlinks/second - open/close 1545 (time 1746131325.90 total 10.00 last 154.47) - open/close 3405 (time 1746131335.90 total 20.00 last 185.96) - open/close 5091 (time 1746131345.90 total 30.00 last 168.58) - open/close 7131 (time 1746131355.90 total 40.01 last 204.00) - open/close 9415 (time 1746131365.90 total 50.01 last 228.38) total: 10000 open/close in 51.61 seconds: 193.75 ops/second - unlinked 0 (time 1746131368 ; total 0 ; last 0) total: 10000 unlinks in 33 seconds: 303.030304 unlinks/second - open/close 2135 (time 1746131412.17 total 10.01 last 213.31) - open/close 4815 (time 1746131422.18 total 20.02 last 267.81) - open/close 6330 (time 1746131432.18 total 30.02 last 151.46) - open/close 8544 (time 1746131442.18 total 40.02 last 221.34) total: 10000 open/close in 47.44 seconds: 210.79 ops/second - unlinked 0 (time 1746131450 ; total 0 ; last 0) total: 10000 unlinks in 35 seconds: 285.714294 unlinks/second - open/close 1195 (time 1746131496.73 total 10.00 last 119.47) - open/close 2182 (time 1746131506.74 total 20.01 last 98.65) - open/close 3248 (time 1746131516.74 total 30.01 last 106.56) - open/close 4358 (time 1746131526.74 total 40.01 last 110.99) - open/close 5329 (time 1746131536.75 total 50.02 last 97.03) - open/close 6498 (time 1746131546.76 total 60.02 last 116.85) - open/close 8110 (time 1746131556.76 total 70.03 last 161.18) - open/close 9332 (time 1746131566.76 total 80.03 last 122.18) total: 10000 open/close in 84.42 seconds: 118.45 ops/second - unlinked 0 (time 1746131572 ; total 0 ; last 0) total: 10000 unlinks in 30 seconds: 333.333344 unlinks/second - open/close 1148 (time 1746131614.69 total 10.00 last 114.76) - open/close 2403 (time 1746131624.70 total 20.01 last 125.45) - open/close 3570 (time 1746131634.70 total 30.01 last 116.61) - open/close 4818 (time 1746131644.71 total 40.02 last 124.77) - open/close 6242 (time 1746131654.72 total 50.03 last 142.28) - open/close 7599 (time 1746131664.72 total 60.03 last 135.69) - open/close 8808 (time 1746131674.73 total 70.04 last 120.72) total: 10000 open/close in 79.81 seconds: 125.29 ops/second - unlinked 0 (time 1746131686 ; total 0 ; last 0) total: 10000 unlinks in 52 seconds: 192.307693 unlinks/second - open/close 928 (time 1746131750.48 total 10.02 last 92.65) - open/close 1951 (time 1746131760.48 total 20.02 last 102.27) - open/close 3059 (time 1746131770.49 total 30.03 last 110.72) - open/close 4102 (time 1746131780.50 total 40.04 last 104.15) - open/close 5022 (time 1746131790.51 total 50.05 last 91.92) - open/close 5993 (time 1746131800.52 total 60.05 last 97.08) - open/close 6888 (time 1746131810.52 total 70.06 last 89.48) - open/close 7926 (time 1746131820.52 total 80.06 last 103.75) - open/close 9118 (time 1746131830.53 total 90.07 last 119.13) total: 10000 open/close in 98.86 seconds: 101.15 ops/second - unlinked 0 (time 1746131841 ; total 0 ; last 0) total: 10000 unlinks in 44 seconds: 227.272720 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg440-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg440-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg440-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.204.140@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.204.140@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg440-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg440-server: oleg440-server.virtnet: executing set_default_debug -1 all pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 Started lustre-OST0000 oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg440-server: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg440-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 5 sec mount lustre on /mnt/lustre..... Starting client: oleg440-client.virtnet: -o user_xattr,flock 192.168.204.140@tcp:/lustre /mnt/lustre On OST0, 17701 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg440-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg440-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg440-server stop mds service on oleg440-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg440-server unloading modules via unload_modules_local on: 'oleg440-server' oleg440-server: oleg440-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg440-client: oleg440-client: ssh exited with exit code 2 pdsh@oleg440-client: oleg440-server: ssh exited with exit code 2 pdsh@oleg440-client: oleg440-client: ssh exited with exit code 2 pdsh@oleg440-client: oleg440-server: ssh exited with exit code 2 PASS 69 (1084s) == conf-sanity test 111: Adding large_dir with over 2GB directory ========================================================== 16:39:00 (1746131940) SKIP: conf-sanity test_111 Only applicable to ldiskfs-based MDTs SKIP 111 (2s) Stopping clients: oleg440-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg440-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg440-client: oleg440-server: ssh exited with exit code 2 oleg440-server: oleg440-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_refcnt: could not read integer from '/sys/module/acpi_cpufreq/refcnt': 'No such device' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg440-server' oleg440-server: oleg440-server.virtnet: executing load_modules_local oleg440-server: Loading modules from /home/green/git/lustre-release/lustre oleg440-server: detected 4 online CPUs by sysfs oleg440-server: MODOPTS_LIBCFS= oleg440-server: Force libcfs to create 2 CPU partitions oleg440-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory oleg440-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg440-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 1326 sec ========== 16:39:31 (1746131971) === conf-sanity: start cleanup 16:39:32 (1746131972) === === conf-sanity: finish cleanup 16:39:33 (1746131973) ===