-----============= acceptance-small: conf-sanity ============----- Thu May 1 19:34:29 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg239-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32b 32c 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a Stopping clients: oleg239-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg239-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg239-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg239-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg239-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg239-server oleg239-server: oleg239-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory loading modules on: 'oleg239-server' oleg239-server: oleg239-server.virtnet: executing load_modules_local oleg239-server: Loading modules from /home/green/git/lustre-release/lustre oleg239-server: detected 4 online CPUs by sysfs oleg239-server: MODOPTS_LIBCFS= oleg239-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg239-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg239-server: oleg239-server.virtnet: executing set_default_debug -1 all pdsh@oleg239-client: oleg239-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg239-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg239-server: oleg239-server.virtnet: executing set_default_debug -1 all pdsh@oleg239-client: oleg239-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg239-server: oleg239-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg239-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg239-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg239-server stop mds service on oleg239-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg239-server == conf-sanity test 45: long unlink handling in ptlrpcd == 19:35:41 (1746142541) start mds service on oleg239-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg239-server: oleg239-server.virtnet: executing set_default_debug -1 all pdsh@oleg239-client: oleg239-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg239-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg239-server: oleg239-server.virtnet: executing set_default_debug -1 all pdsh@oleg239-client: oleg239-server: ssh exited with exit code 1 Started lustre-OST0000 oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg239-client.virtnet: -o user_xattr,flock 192.168.202.139@tcp:/lustre /mnt/lustre setup single mount lustre success stop mds service on oleg239-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg239-server sleep 60 sec fail_loc=0x8000050f sleep 10 sec manual umount lustre on /mnt/lustre.... df: '/mnt/lustre': Cannot send after transport endpoint shutdown fail_loc=0x0 start mds service on oleg239-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg239-server: oleg239-server.virtnet: executing set_default_debug -1 all pdsh@oleg239-client: oleg239-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg239-client.virtnet: -o user_xattr,flock 192.168.202.139@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg239-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg239-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg239-server stop mds service on oleg239-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg239-server unloading modules via unload_modules_local on: 'oleg239-server' oleg239-server: oleg239-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg239-client: oleg239-server: ssh exited with exit code 2 pdsh@oleg239-client: oleg239-client: ssh exited with exit code 2 pdsh@oleg239-client: oleg239-client: ssh exited with exit code 2 pdsh@oleg239-client: oleg239-server: ssh exited with exit code 2 PASS 45 (147s) == conf-sanity test 69: replace an OST with the same index ========================================================== 19:38:08 (1746142688) start mds service on oleg239-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg239-server' oleg239-server: oleg239-server.virtnet: executing load_modules_local oleg239-server: Loading modules from /home/green/git/lustre-release/lustre oleg239-server: detected 4 online CPUs by sysfs oleg239-server: MODOPTS_LIBCFS= oleg239-server: Force libcfs to create 2 CPU partitions oleg239-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg239-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg239-server: oleg239-server.virtnet: executing set_default_debug -1 all pdsh@oleg239-client: oleg239-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg239-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg239-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg239-server: oleg239-server.virtnet: executing set_default_debug -1 all pdsh@oleg239-client: oleg239-server: ssh exited with exit code 1 Started lustre-OST0000 oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg239-client.virtnet: -o user_xattr,flock 192.168.202.139@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 1250 (time 1746142732.69 total 10.00 last 124.97) - open/close 2368 (time 1746142742.69 total 20.01 last 111.76) - open/close 3260 (time 1746142752.70 total 30.01 last 89.16) - open/close 4181 (time 1746142762.70 total 40.01 last 92.09) - open/close 5094 (time 1746142772.70 total 50.02 last 91.26) - open/close 6173 (time 1746142782.71 total 60.02 last 107.87) - open/close 7351 (time 1746142792.71 total 70.02 last 117.74) - open/close 8358 (time 1746142802.71 total 80.03 last 100.69) - open/close 9450 (time 1746142812.72 total 90.03 last 109.13) total: 10000 open/close in 93.69 seconds: 106.73 ops/second - unlinked 0 (time 1746142818 ; total 0 ; last 0) total: 10000 unlinks in 50 seconds: 200.000000 unlinks/second - open/close 1242 (time 1746142880.18 total 10.00 last 124.17) - open/close 2467 (time 1746142890.18 total 20.00 last 122.48) - open/close 3746 (time 1746142900.18 total 30.01 last 127.85) - open/close 4968 (time 1746142910.19 total 40.02 last 122.10) - open/close 6232 (time 1746142920.20 total 50.02 last 126.37) - open/close 7188 (time 1746142930.22 total 60.04 last 95.40) - open/close 8178 (time 1746142940.23 total 70.05 last 98.90) - open/close 9220 (time 1746142950.23 total 80.05 last 104.15) total: 10000 open/close in 86.47 seconds: 115.64 ops/second - unlinked 0 (time 1746142958 ; total 0 ; last 0) total: 10000 unlinks in 53 seconds: 188.679245 unlinks/second - open/close 1055 (time 1746143023.81 total 10.01 last 105.40) - open/close 2071 (time 1746143033.81 total 20.01 last 101.59) - open/close 3198 (time 1746143043.81 total 30.01 last 112.69) - open/close 4283 (time 1746143053.81 total 40.02 last 108.45) - open/close 5306 (time 1746143063.82 total 50.02 last 102.22) - open/close 6331 (time 1746143073.83 total 60.03 last 102.42) - open/close 7275 (time 1746143083.84 total 70.04 last 94.29) - open/close 8178 (time 1746143093.85 total 80.05 last 90.24) - open/close 9186 (time 1746143103.85 total 90.05 last 100.78) total: 10000 open/close in 98.18 seconds: 101.86 ops/second - unlinked 0 (time 1746143114 ; total 0 ; last 0) total: 10000 unlinks in 64 seconds: 156.250000 unlinks/second - open/close 964 (time 1746143190.67 total 10.01 last 96.34) - open/close 2111 (time 1746143200.67 total 20.01 last 114.64) - open/close 3059 (time 1746143210.69 total 30.02 last 94.68) - open/close 4060 (time 1746143220.69 total 40.03 last 100.02) - open/close 5157 (time 1746143230.70 total 50.04 last 109.65) - open/close 6149 (time 1746143240.70 total 60.04 last 99.16) - open/close 7105 (time 1746143250.71 total 70.05 last 95.55) - open/close 8099 (time 1746143260.71 total 80.05 last 99.36) - open/close 9120 (time 1746143270.73 total 90.07 last 101.94) total: 10000 open/close in 99.41 seconds: 100.60 ops/second - unlinked 0 (time 1746143282 ; total 0 ; last 0) total: 10000 unlinks in 62 seconds: 161.290329 unlinks/second - open/close 1002 (time 1746143356.60 total 10.01 last 100.13) - open/close 2023 (time 1746143366.61 total 20.01 last 102.05) - open/close 3044 (time 1746143376.62 total 30.02 last 101.98) - open/close 4042 (time 1746143386.63 total 40.03 last 99.75) - open/close 5017 (time 1746143396.64 total 50.04 last 97.40) - open/close 6005 (time 1746143406.64 total 60.04 last 98.76) - open/close 7005 (time 1746143416.65 total 70.05 last 99.94) - open/close 8005 (time 1746143426.67 total 80.07 last 99.78) - open/close 9010 (time 1746143436.67 total 90.08 last 100.44) total: 10000 open/close in 99.24 seconds: 100.77 ops/second - unlinked 0 (time 1746143447 ; total 0 ; last 0) total: 10000 unlinks in 54 seconds: 185.185181 unlinks/second - open/close 1065 (time 1746143513.09 total 10.00 last 106.47) - open/close 2096 (time 1746143523.09 total 20.00 last 103.10) - open/close 3200 (time 1746143533.10 total 30.01 last 110.34) - open/close 4311 (time 1746143543.11 total 40.02 last 111.01) - open/close 5310 (time 1746143553.11 total 50.02 last 99.82) - open/close 6332 (time 1746143563.12 total 60.03 last 102.11) - open/close 7317 (time 1746143573.12 total 70.03 last 98.50) - open/close 8249 (time 1746143583.13 total 80.04 last 93.13) - open/close 9291 (time 1746143593.13 total 90.05 last 104.15) total: 10000 open/close in 97.05 seconds: 103.04 ops/second - unlinked 0 (time 1746143602 ; total 0 ; last 0) total: 10000 unlinks in 55 seconds: 181.818176 unlinks/second - open/close 988 (time 1746143669.58 total 10.00 last 98.77) - open/close 2032 (time 1746143679.58 total 20.00 last 104.38) - open/close 3044 (time 1746143689.60 total 30.02 last 101.08) - open/close 4028 (time 1746143699.60 total 40.02 last 98.39) - open/close 5080 (time 1746143709.61 total 50.03 last 105.06) - open/close 6228 (time 1746143719.62 total 60.04 last 114.71) - open/close 7190 (time 1746143729.63 total 70.05 last 96.08) - open/close 8234 (time 1746143739.64 total 80.06 last 104.32) - open/close 9366 (time 1746143749.64 total 90.06 last 113.15) total: 10000 open/close in 95.58 seconds: 104.62 ops/second - unlinked 0 (time 1746143756 ; total 0 ; last 0) total: 10000 unlinks in 45 seconds: 222.222229 unlinks/second - open/close 1478 (time 1746143812.62 total 10.00 last 147.77) - open/close 2848 (time 1746143822.62 total 20.00 last 136.97) - open/close 4317 (time 1746143832.62 total 30.01 last 146.88) - open/close 5510 (time 1746143842.62 total 40.01 last 119.29) - open/close 6683 (time 1746143852.62 total 50.01 last 117.28) - open/close 7845 (time 1746143862.62 total 60.01 last 116.19) - open/close 8982 (time 1746143872.63 total 70.02 last 113.60) total: 10000 open/close in 77.69 seconds: 128.72 ops/second - unlinked 0 (time 1746143882 ; total 0 ; last 0) total: 10000 unlinks in 43 seconds: 232.558136 unlinks/second - open/close 1231 (time 1746143937.28 total 10.01 last 123.04) - open/close 2467 (time 1746143947.28 total 20.01 last 123.55) - open/close 3652 (time 1746143957.29 total 30.01 last 118.45) - open/close 5105 (time 1746143967.29 total 40.02 last 145.27) - open/close 6243 (time 1746143977.30 total 50.03 last 113.67) - open/close 7598 (time 1746143987.30 total 60.03 last 135.49) - open/close 9007 (time 1746143997.31 total 70.03 last 140.82) total: 10000 open/close in 77.15 seconds: 129.62 ops/second - unlinked 0 (time 1746144006 ; total 0 ; last 0) total: 10000 unlinks in 51 seconds: 196.078430 unlinks/second - open/close 1074 (time 1746144069.76 total 10.00 last 107.35) - open/close 2189 (time 1746144079.76 total 20.01 last 111.47) - open/close 3357 (time 1746144089.77 total 30.01 last 116.73) - open/close 4433 (time 1746144099.77 total 40.01 last 107.59) - open/close 5562 (time 1746144109.78 total 50.03 last 112.74) - open/close 6775 (time 1746144119.79 total 60.03 last 121.25) - open/close 8012 (time 1746144129.79 total 70.04 last 123.62) - open/close 9138 (time 1746144139.80 total 80.04 last 112.56) total: 10000 open/close in 86.84 seconds: 115.15 ops/second - unlinked 0 (time 1746144148 ; total 0 ; last 0) total: 10000 unlinks in 48 seconds: 208.333328 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg239-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg239-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg239-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.202.139@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.202.139@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg239-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg239-server: oleg239-server.virtnet: executing set_default_debug -1 all pdsh@oleg239-client: oleg239-server: ssh exited with exit code 1 Started lustre-OST0000 oleg239-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg239-server: oleg239-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg239-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg239-client.virtnet: -o user_xattr,flock 192.168.202.139@tcp:/lustre /mnt/lustre On OST0, 15117 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg239-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg239-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg239-server stop mds service on oleg239-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg239-server unloading modules via unload_modules_local on: 'oleg239-server' oleg239-server: oleg239-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg239-client: oleg239-client: ssh exited with exit code 2 pdsh@oleg239-client: oleg239-server: ssh exited with exit code 2 pdsh@oleg239-client: oleg239-client: ssh exited with exit code 2 pdsh@oleg239-client: oleg239-server: ssh exited with exit code 2 PASS 69 (1563s) == conf-sanity test 111: Adding large_dir with over 2GB directory ========================================================== 20:04:12 (1746144252) SKIP: conf-sanity test_111 Only applicable to ldiskfs-based MDTs SKIP 111 (1s) Stopping clients: oleg239-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg239-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg239-client: oleg239-server: ssh exited with exit code 2 oleg239-server: oleg239-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg239-server' oleg239-server: oleg239-server.virtnet: executing load_modules_local oleg239-server: Loading modules from /home/green/git/lustre-release/lustre oleg239-server: detected 4 online CPUs by sysfs oleg239-server: MODOPTS_LIBCFS= oleg239-server: Force libcfs to create 2 CPU partitions oleg239-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg239-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg239-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 1816 sec ========== 20:04:45 (1746144285) === conf-sanity: start cleanup 20:04:46 (1746144286) === === conf-sanity: finish cleanup 20:04:47 (1746144287) ===