-----============= acceptance-small: conf-sanity ============----- Thu May 1 15:00:42 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg417-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32b 32c 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a Stopping clients: oleg417-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg417-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg417-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg417-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg417-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg417-server oleg417-server: oleg417-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg417-server' oleg417-server: oleg417-server.virtnet: executing load_modules_local oleg417-server: Loading modules from /home/green/git/lustre-release/lustre oleg417-server: detected 4 online CPUs by sysfs oleg417-server: MODOPTS_LIBCFS= oleg417-server: Force libcfs to create 2 CPU partitions oleg417-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg417-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg417-server: oleg417-server.virtnet: executing set_default_debug -1 all pdsh@oleg417-client: oleg417-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg417-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg417-server: oleg417-server.virtnet: executing set_default_debug -1 all pdsh@oleg417-client: oleg417-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg417-server: oleg417-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg417-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg417-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg417-server stop mds service on oleg417-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg417-server == conf-sanity test 45: long unlink handling in ptlrpcd == 15:02:24 (1746126144) start mds service on oleg417-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg417-server: oleg417-server.virtnet: executing set_default_debug -1 all pdsh@oleg417-client: oleg417-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg417-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg417-server: oleg417-server.virtnet: executing set_default_debug -1 all pdsh@oleg417-client: oleg417-server: ssh exited with exit code 1 Started lustre-OST0000 oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg417-client.virtnet: -o user_xattr,flock 192.168.204.117@tcp:/lustre /mnt/lustre setup single mount lustre success stop mds service on oleg417-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg417-server sleep 60 sec fail_loc=0x8000050f sleep 10 sec manual umount lustre on /mnt/lustre.... df: '/mnt/lustre': Cannot send after transport endpoint shutdown fail_loc=0x0 start mds service on oleg417-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg417-server: oleg417-server.virtnet: executing set_default_debug -1 all pdsh@oleg417-client: oleg417-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg417-client.virtnet: -o user_xattr,flock 192.168.204.117@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg417-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg417-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg417-server stop mds service on oleg417-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg417-server unloading modules via unload_modules_local on: 'oleg417-server' oleg417-server: oleg417-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg417-client: oleg417-client: ssh exited with exit code 2 pdsh@oleg417-client: oleg417-server: ssh exited with exit code 2 pdsh@oleg417-client: oleg417-client: ssh exited with exit code 2 pdsh@oleg417-client: oleg417-server: ssh exited with exit code 2 PASS 45 (150s) == conf-sanity test 69: replace an OST with the same index ========================================================== 15:04:54 (1746126294) start mds service on oleg417-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg417-server' oleg417-server: oleg417-server.virtnet: executing load_modules_local oleg417-server: Loading modules from /home/green/git/lustre-release/lustre oleg417-server: detected 4 online CPUs by sysfs oleg417-server: MODOPTS_LIBCFS= oleg417-server: Force libcfs to create 2 CPU partitions oleg417-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory oleg417-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg417-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg417-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg417-server: oleg417-server.virtnet: executing set_default_debug -1 all pdsh@oleg417-client: oleg417-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg417-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg417-server: oleg417-server.virtnet: executing set_default_debug -1 all pdsh@oleg417-client: oleg417-server: ssh exited with exit code 1 Started lustre-OST0000 oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg417-client.virtnet: -o user_xattr,flock 192.168.204.117@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 987 (time 1746126342.84 total 10.01 last 98.57) - open/close 2049 (time 1746126352.85 total 20.02 last 106.17) - open/close 3027 (time 1746126362.85 total 30.02 last 97.77) - open/close 3966 (time 1746126372.85 total 40.02 last 93.89) - open/close 4993 (time 1746126382.85 total 50.02 last 102.70) - open/close 5984 (time 1746126392.86 total 60.03 last 99.01) - open/close 6973 (time 1746126402.87 total 70.04 last 98.81) - open/close 7981 (time 1746126412.88 total 80.05 last 100.73) - open/close 8889 (time 1746126422.88 total 90.05 last 90.78) - open/close 9871 (time 1746126432.88 total 100.05 last 98.17) total: 10000 open/close in 101.38 seconds: 98.64 ops/second - unlinked 0 (time 1746126436 ; total 0 ; last 0) total: 10000 unlinks in 55 seconds: 181.818176 unlinks/second - open/close 1207 (time 1746126504.14 total 10.00 last 120.67) - open/close 2355 (time 1746126514.14 total 20.01 last 114.76) - open/close 3576 (time 1746126524.15 total 30.02 last 121.93) - open/close 4646 (time 1746126534.16 total 40.02 last 106.96) - open/close 5818 (time 1746126544.16 total 50.03 last 117.13) - open/close 6884 (time 1746126554.18 total 60.04 last 106.48) - open/close 8020 (time 1746126564.18 total 70.04 last 113.55) - open/close 9081 (time 1746126574.19 total 80.05 last 106.04) total: 10000 open/close in 88.50 seconds: 113.00 ops/second - unlinked 0 (time 1746126584 ; total 0 ; last 0) total: 10000 unlinks in 54 seconds: 185.185181 unlinks/second - open/close 1356 (time 1746126650.85 total 10.01 last 135.53) - open/close 2914 (time 1746126660.86 total 20.01 last 155.75) - open/close 4312 (time 1746126670.86 total 30.01 last 139.74) - open/close 5459 (time 1746126680.87 total 40.02 last 114.63) - open/close 6862 (time 1746126690.87 total 50.02 last 140.30) - open/close 8252 (time 1746126700.87 total 60.02 last 138.96) - open/close 9735 (time 1746126710.87 total 70.02 last 148.25) total: 10000 open/close in 71.72 seconds: 139.44 ops/second - unlinked 0 (time 1746126713 ; total 0 ; last 0) total: 10000 unlinks in 36 seconds: 277.777771 unlinks/second - open/close 1613 (time 1746126760.72 total 10.00 last 161.29) - open/close 3700 (time 1746126770.72 total 20.00 last 208.66) - open/close 5481 (time 1746126780.73 total 30.01 last 178.00) - open/close 7867 (time 1746126790.73 total 40.01 last 238.54) - open/close 9311 (time 1746126800.73 total 50.01 last 144.37) total: 10000 open/close in 54.76 seconds: 182.63 ops/second - unlinked 0 (time 1746126807 ; total 0 ; last 0) total: 10000 unlinks in 30 seconds: 333.333344 unlinks/second - open/close 1620 (time 1746126848.86 total 10.00 last 161.98) - open/close 3717 (time 1746126858.86 total 20.00 last 209.66) - open/close 5894 (time 1746126868.86 total 30.00 last 217.69) - open/close 7840 (time 1746126878.86 total 40.01 last 194.55) - open/close 9410 (time 1746126888.87 total 50.01 last 156.96) total: 10000 open/close in 53.61 seconds: 186.52 ops/second - unlinked 0 (time 1746126893 ; total 0 ; last 0) total: 10000 unlinks in 35 seconds: 285.714294 unlinks/second - open/close 2430 (time 1746126939.34 total 10.00 last 242.95) - open/close 4929 (time 1746126949.34 total 20.00 last 249.83) - open/close 7007 (time 1746126959.35 total 30.01 last 207.64) - open/close 8773 (time 1746126969.35 total 40.02 last 176.50) total: 10000 open/close in 47.09 seconds: 212.34 ops/second - unlinked 0 (time 1746126977 ; total 0 ; last 0) total: 10000 unlinks in 32 seconds: 312.500000 unlinks/second - open/close 2098 (time 1746127020.81 total 10.00 last 209.70) - open/close 4398 (time 1746127030.81 total 20.01 last 229.96) - open/close 6322 (time 1746127040.82 total 30.01 last 192.34) - open/close 8217 (time 1746127050.82 total 40.01 last 189.41) total: 10000 open/close in 49.32 seconds: 202.75 ops/second - unlinked 0 (time 1746127061 ; total 0 ; last 0) total: 10000 unlinks in 26 seconds: 384.615387 unlinks/second - open/close 1723 (time 1746127098.19 total 10.00 last 172.29) - open/close 4120 (time 1746127108.19 total 20.00 last 239.68) - open/close 6487 (time 1746127118.20 total 30.00 last 236.61) - open/close 8549 (time 1746127128.20 total 40.01 last 206.15) total: 10000 open/close in 46.63 seconds: 214.45 ops/second - unlinked 0 (time 1746127136 ; total 0 ; last 0) total: 10000 unlinks in 24 seconds: 416.666656 unlinks/second - open/close 2225 (time 1746127171.51 total 10.00 last 222.48) - open/close 4421 (time 1746127181.51 total 20.00 last 219.53) - open/close 6458 (time 1746127191.51 total 30.00 last 203.69) - open/close 8335 (time 1746127201.51 total 40.01 last 187.63) total: 10000 open/close in 48.85 seconds: 204.72 ops/second - unlinked 0 (time 1746127211 ; total 0 ; last 0) total: 10000 unlinks in 31 seconds: 322.580658 unlinks/second - open/close 2568 (time 1746127253.98 total 10.00 last 256.74) - open/close 4969 (time 1746127263.98 total 20.00 last 240.04) - open/close 7182 (time 1746127273.98 total 30.01 last 221.24) - open/close 9252 (time 1746127283.99 total 40.01 last 206.96) total: 10000 open/close in 43.42 seconds: 230.31 ops/second - unlinked 0 (time 1746127288 ; total 0 ; last 0) total: 10000 unlinks in 27 seconds: 370.370361 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg417-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg417-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg417-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.204.117@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.204.117@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg417-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg417-server: oleg417-server.virtnet: executing set_default_debug -1 all pdsh@oleg417-client: oleg417-server: ssh exited with exit code 1 Started lustre-OST0000 oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg417-server: oleg417-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg417-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 10 sec mount lustre on /mnt/lustre..... Starting client: oleg417-client.virtnet: -o user_xattr,flock 192.168.204.117@tcp:/lustre /mnt/lustre On OST0, 18413 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg417-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg417-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg417-server stop mds service on oleg417-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg417-server unloading modules via unload_modules_local on: 'oleg417-server' oleg417-server: oleg417-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg417-client: oleg417-client: ssh exited with exit code 2 pdsh@oleg417-client: oleg417-server: ssh exited with exit code 2 pdsh@oleg417-client: oleg417-client: ssh exited with exit code 2 pdsh@oleg417-client: oleg417-server: ssh exited with exit code 2 PASS 69 (1069s) == conf-sanity test 111: Adding large_dir with over 2GB directory ========================================================== 15:22:44 (1746127364) SKIP: conf-sanity test_111 Only applicable to ldiskfs-based MDTs SKIP 111 (1s) Stopping clients: oleg417-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg417-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg417-client: oleg417-server: ssh exited with exit code 2 oleg417-server: oleg417-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg417-server' oleg417-server: oleg417-server.virtnet: executing load_modules_local oleg417-server: Loading modules from /home/green/git/lustre-release/lustre oleg417-server: detected 4 online CPUs by sysfs oleg417-server: MODOPTS_LIBCFS= oleg417-server: Force libcfs to create 2 CPU partitions oleg417-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg417-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 1346 sec ========== 15:23:08 (1746127388) === conf-sanity: start cleanup 15:23:09 (1746127389) === === conf-sanity: finish cleanup 15:23:10 (1746127390) ===