-----============= acceptance-small: conf-sanity ============----- Thu May 1 15:50:39 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg235-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32b 32c 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a Stopping clients: oleg235-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg235-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg235-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg235-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg235-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg235-server oleg235-server: oleg235-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory loading modules on: 'oleg235-server' oleg235-server: oleg235-server.virtnet: executing load_modules_local oleg235-server: Loading modules from /home/green/git/lustre-release/lustre oleg235-server: detected 4 online CPUs by sysfs oleg235-server: MODOPTS_LIBCFS= oleg235-server: Force libcfs to create 2 CPU partitions oleg235-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg235-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg235-server: oleg235-server.virtnet: executing set_default_debug -1 all pdsh@oleg235-client: oleg235-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg235-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg235-server: oleg235-server.virtnet: executing set_default_debug -1 all pdsh@oleg235-client: oleg235-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg235-server: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg235-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg235-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg235-server stop mds service on oleg235-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg235-server == conf-sanity test 45: long unlink handling in ptlrpcd == 15:52:26 (1746129146) start mds service on oleg235-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg235-server: oleg235-server.virtnet: executing set_default_debug -1 all pdsh@oleg235-client: oleg235-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg235-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg235-server: oleg235-server.virtnet: executing set_default_debug -1 all pdsh@oleg235-client: oleg235-server: ssh exited with exit code 1 Started lustre-OST0000 oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg235-client.virtnet: -o user_xattr,flock 192.168.202.135@tcp:/lustre /mnt/lustre setup single mount lustre success stop mds service on oleg235-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg235-server sleep 60 sec fail_loc=0x8000050f sleep 10 sec manual umount lustre on /mnt/lustre.... df: '/mnt/lustre': Cannot send after transport endpoint shutdown fail_loc=0x0 start mds service on oleg235-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg235-server: oleg235-server.virtnet: executing set_default_debug -1 all pdsh@oleg235-client: oleg235-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg235-client.virtnet: -o user_xattr,flock 192.168.202.135@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg235-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg235-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg235-server stop mds service on oleg235-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg235-server unloading modules via unload_modules_local on: 'oleg235-server' oleg235-server: oleg235-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg235-client: oleg235-server: ssh exited with exit code 2 pdsh@oleg235-client: oleg235-client: ssh exited with exit code 2 pdsh@oleg235-client: oleg235-client: ssh exited with exit code 2 pdsh@oleg235-client: oleg235-server: ssh exited with exit code 2 PASS 45 (145s) == conf-sanity test 69: replace an OST with the same index ========================================================== 15:54:51 (1746129291) start mds service on oleg235-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg235-server' oleg235-server: oleg235-server.virtnet: executing load_modules_local oleg235-server: Loading modules from /home/green/git/lustre-release/lustre oleg235-server: detected 4 online CPUs by sysfs oleg235-server: MODOPTS_LIBCFS= oleg235-server: Force libcfs to create 2 CPU partitions oleg235-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg235-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg235-server: oleg235-server.virtnet: executing set_default_debug -1 all pdsh@oleg235-client: oleg235-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg235-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg235-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg235-server: oleg235-server.virtnet: executing set_default_debug -1 all pdsh@oleg235-client: oleg235-server: ssh exited with exit code 1 Started lustre-OST0000 oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg235-client.virtnet: -o user_xattr,flock 192.168.202.135@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 1306 (time 1746129332.84 total 10.00 last 130.58) - open/close 2785 (time 1746129342.84 total 20.00 last 147.87) - open/close 4200 (time 1746129352.84 total 30.01 last 141.41) - open/close 5404 (time 1746129362.85 total 40.02 last 120.27) - open/close 6461 (time 1746129372.87 total 50.03 last 105.57) - open/close 7617 (time 1746129382.87 total 60.03 last 115.60) - open/close 8884 (time 1746129392.87 total 70.04 last 126.63) total: 10000 open/close in 77.91 seconds: 128.36 ops/second - unlinked 0 (time 1746129402 ; total 0 ; last 0) total: 10000 unlinks in 50 seconds: 200.000000 unlinks/second - open/close 1226 (time 1746129463.99 total 10.01 last 122.53) - open/close 2188 (time 1746129473.99 total 20.01 last 96.14) - open/close 3415 (time 1746129484.00 total 30.02 last 122.66) - open/close 4682 (time 1746129494.00 total 40.02 last 126.62) - open/close 6109 (time 1746129504.01 total 50.03 last 142.62) - open/close 7177 (time 1746129514.01 total 60.03 last 106.75) - open/close 8246 (time 1746129524.01 total 70.03 last 106.89) - open/close 9439 (time 1746129534.02 total 80.03 last 119.28) total: 10000 open/close in 84.31 seconds: 118.62 ops/second - unlinked 0 (time 1746129539 ; total 0 ; last 0) total: 10000 unlinks in 55 seconds: 181.818176 unlinks/second - open/close 1127 (time 1746129606.15 total 10.00 last 112.68) - open/close 2142 (time 1746129616.15 total 20.01 last 101.46) - open/close 3392 (time 1746129626.15 total 30.01 last 124.99) - open/close 5133 (time 1746129636.16 total 40.01 last 174.00) - open/close 6584 (time 1746129646.16 total 50.02 last 145.01) - open/close 7878 (time 1746129656.17 total 60.02 last 129.31) - open/close 9247 (time 1746129666.17 total 70.03 last 136.84) total: 10000 open/close in 74.75 seconds: 133.77 ops/second - unlinked 0 (time 1746129672 ; total 0 ; last 0) total: 10000 unlinks in 38 seconds: 263.157898 unlinks/second - open/close 1643 (time 1746129721.86 total 10.00 last 164.28) - open/close 3301 (time 1746129731.87 total 20.01 last 165.70) - open/close 4907 (time 1746129741.87 total 30.01 last 160.53) - open/close 6374 (time 1746129751.87 total 40.01 last 146.67) - open/close 7666 (time 1746129761.88 total 50.02 last 129.12) - open/close 9216 (time 1746129771.89 total 60.03 last 154.86) total: 10000 open/close in 66.47 seconds: 150.44 ops/second - unlinked 0 (time 1746129780 ; total 0 ; last 0) total: 10000 unlinks in 44 seconds: 227.272720 unlinks/second - open/close 1596 (time 1746129836.35 total 10.01 last 159.48) - open/close 2975 (time 1746129846.35 total 20.01 last 137.83) - open/close 4228 (time 1746129856.36 total 30.02 last 125.19) - open/close 5911 (time 1746129866.37 total 40.03 last 168.22) - open/close 7468 (time 1746129876.37 total 50.03 last 155.61) - open/close 9122 (time 1746129886.38 total 60.04 last 165.34) total: 10000 open/close in 65.09 seconds: 153.63 ops/second - unlinked 0 (time 1746129892 ; total 0 ; last 0) total: 10000 unlinks in 40 seconds: 250.000000 unlinks/second - open/close 1045 (time 1746129945.16 total 10.01 last 104.41) - open/close 2308 (time 1746129955.17 total 20.01 last 126.25) - open/close 3528 (time 1746129965.18 total 30.03 last 121.81) - open/close 4713 (time 1746129975.19 total 40.03 last 118.45) - open/close 5997 (time 1746129985.19 total 50.03 last 128.38) - open/close 7305 (time 1746129995.20 total 60.04 last 130.73) - open/close 8697 (time 1746130005.20 total 70.04 last 139.17) total: 10000 open/close in 78.80 seconds: 126.90 ops/second - unlinked 0 (time 1746130015 ; total 0 ; last 0) total: 10000 unlinks in 46 seconds: 217.391312 unlinks/second - open/close 1516 (time 1746130072.55 total 10.00 last 151.60) - open/close 3223 (time 1746130082.56 total 20.00 last 170.62) - open/close 4709 (time 1746130092.56 total 30.01 last 148.50) - open/close 6106 (time 1746130102.57 total 40.02 last 139.64) - open/close 7601 (time 1746130112.57 total 50.02 last 149.41) - open/close 9053 (time 1746130122.58 total 60.03 last 145.12) total: 10000 open/close in 66.15 seconds: 151.16 ops/second - unlinked 0 (time 1746130130 ; total 0 ; last 0) total: 10000 unlinks in 46 seconds: 217.391312 unlinks/second - open/close 1327 (time 1746130187.90 total 10.00 last 132.66) - open/close 3066 (time 1746130197.90 total 20.01 last 173.80) - open/close 4816 (time 1746130207.91 total 30.01 last 174.97) - open/close 6346 (time 1746130217.91 total 40.01 last 152.94) - open/close 7996 (time 1746130227.91 total 50.02 last 164.93) - open/close 9501 (time 1746130237.92 total 60.02 last 150.48) total: 10000 open/close in 63.61 seconds: 157.20 ops/second - unlinked 0 (time 1746130243 ; total 0 ; last 0) total: 10000 unlinks in 36 seconds: 277.777771 unlinks/second - open/close 1654 (time 1746130290.80 total 10.00 last 165.32) - open/close 3420 (time 1746130300.80 total 20.01 last 176.53) - open/close 4929 (time 1746130310.81 total 30.01 last 150.84) - open/close 6253 (time 1746130320.82 total 40.02 last 132.30) - open/close 7374 (time 1746130330.82 total 50.03 last 112.04) - open/close 8726 (time 1746130340.82 total 60.03 last 135.16) - open/close 9995 (time 1746130350.83 total 70.04 last 126.82) total: 10000 open/close in 70.07 seconds: 142.72 ops/second - unlinked 0 (time 1746130352 ; total 0 ; last 0) total: 10000 unlinks in 43 seconds: 232.558136 unlinks/second - open/close 1405 (time 1746130407.33 total 10.00 last 140.48) - open/close 2560 (time 1746130417.34 total 20.00 last 115.46) - open/close 3439 (time 1746130427.34 total 30.01 last 87.88) - open/close 4615 (time 1746130437.34 total 40.01 last 117.60) - open/close 5554 (time 1746130447.35 total 50.02 last 93.80) - open/close 6593 (time 1746130457.35 total 60.02 last 103.89) - open/close 7850 (time 1746130467.36 total 70.02 last 125.64) - open/close 8942 (time 1746130477.36 total 80.02 last 109.19) - open/close 9997 (time 1746130487.36 total 90.03 last 105.43) total: 10000 open/close in 90.05 seconds: 111.05 ops/second - unlinked 0 (time 1746130489 ; total 0 ; last 0) total: 10000 unlinks in 58 seconds: 172.413788 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg235-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg235-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg235-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.202.135@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.202.135@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg235-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg235-server: oleg235-server.virtnet: executing set_default_debug -1 all pdsh@oleg235-client: oleg235-server: ssh exited with exit code 1 Started lustre-OST0000 oleg235-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg235-server: oleg235-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg235-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec mount lustre on /mnt/lustre..... Starting client: oleg235-client.virtnet: -o user_xattr,flock 192.168.202.135@tcp:/lustre /mnt/lustre On OST0, 18069 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg235-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg235-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg235-server stop mds service on oleg235-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg235-server unloading modules via unload_modules_local on: 'oleg235-server' oleg235-server: oleg235-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg235-client: oleg235-client: ssh exited with exit code 2 pdsh@oleg235-client: oleg235-server: ssh exited with exit code 2 pdsh@oleg235-client: oleg235-client: ssh exited with exit code 2 pdsh@oleg235-client: oleg235-server: ssh exited with exit code 2 PASS 69 (1314s) == conf-sanity test 111: Adding large_dir with over 2GB directory ========================================================== 16:16:45 (1746130605) SKIP: conf-sanity test_111 Only applicable to ldiskfs-based MDTs SKIP 111 (1s) Stopping clients: oleg235-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg235-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg235-client: oleg235-server: ssh exited with exit code 2 oleg235-server: oleg235-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg235-server' oleg235-server: oleg235-server.virtnet: executing load_modules_local oleg235-server: Loading modules from /home/green/git/lustre-release/lustre oleg235-server: detected 4 online CPUs by sysfs oleg235-server: MODOPTS_LIBCFS= oleg235-server: Force libcfs to create 2 CPU partitions oleg235-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg235-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 1597 sec ========== 16:17:18 (1746130638) === conf-sanity: start cleanup 16:17:18 (1746130638) === === conf-sanity: finish cleanup 16:17:20 (1746130640) ===