-----============= acceptance-small: conf-sanity ============----- Thu May 1 20:35:38 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg332-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32b 32c 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a Stopping clients: oleg332-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg332-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg332-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg332-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg332-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg332-server oleg332-server: oleg332-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory loading modules on: 'oleg332-server' oleg332-server: oleg332-server.virtnet: executing load_modules_local oleg332-server: Loading modules from /home/green/git/lustre-release/lustre oleg332-server: detected 4 online CPUs by sysfs oleg332-server: MODOPTS_LIBCFS= oleg332-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg332-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg332-server: oleg332-server.virtnet: executing set_default_debug -1 all pdsh@oleg332-client: oleg332-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg332-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg332-server: oleg332-server.virtnet: executing set_default_debug -1 all pdsh@oleg332-client: oleg332-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg332-server: oleg332-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg332-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg332-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg332-server stop mds service on oleg332-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg332-server == conf-sanity test 45: long unlink handling in ptlrpcd == 20:36:59 (1746146219) start mds service on oleg332-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg332-server: oleg332-server.virtnet: executing set_default_debug -1 all pdsh@oleg332-client: oleg332-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg332-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg332-server: oleg332-server.virtnet: executing set_default_debug -1 all pdsh@oleg332-client: oleg332-server: ssh exited with exit code 1 Started lustre-OST0000 oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg332-client.virtnet: -o user_xattr,flock 192.168.203.132@tcp:/lustre /mnt/lustre setup single mount lustre success stop mds service on oleg332-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg332-server sleep 60 sec fail_loc=0x8000050f sleep 10 sec manual umount lustre on /mnt/lustre.... df: '/mnt/lustre': Cannot send after transport endpoint shutdown fail_loc=0x0 start mds service on oleg332-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg332-server: oleg332-server.virtnet: executing set_default_debug -1 all pdsh@oleg332-client: oleg332-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg332-client.virtnet: -o user_xattr,flock 192.168.203.132@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg332-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg332-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg332-server stop mds service on oleg332-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg332-server unloading modules via unload_modules_local on: 'oleg332-server' oleg332-server: oleg332-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg332-client: oleg332-client: ssh exited with exit code 2 pdsh@oleg332-client: oleg332-server: ssh exited with exit code 2 pdsh@oleg332-client: oleg332-client: ssh exited with exit code 2 pdsh@oleg332-client: oleg332-server: ssh exited with exit code 2 PASS 45 (138s) == conf-sanity test 69: replace an OST with the same index ========================================================== 20:39:17 (1746146357) start mds service on oleg332-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg332-server' oleg332-server: oleg332-server.virtnet: executing load_modules_local oleg332-server: Loading modules from /home/green/git/lustre-release/lustre oleg332-server: detected 4 online CPUs by sysfs oleg332-server: MODOPTS_LIBCFS= oleg332-server: Force libcfs to create 2 CPU partitions oleg332-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg332-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg332-server: oleg332-server.virtnet: executing set_default_debug -1 all pdsh@oleg332-client: oleg332-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg332-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg332-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg332-server: oleg332-server.virtnet: executing set_default_debug -1 all pdsh@oleg332-client: oleg332-server: ssh exited with exit code 1 Started lustre-OST0000 oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg332-client.virtnet: -o user_xattr,flock 192.168.203.132@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 1831 (time 1746146396.14 total 10.00 last 183.06) - open/close 3401 (time 1746146406.15 total 20.01 last 156.94) - open/close 5237 (time 1746146416.15 total 30.01 last 183.53) - open/close 7022 (time 1746146426.15 total 40.01 last 178.46) - open/close 8818 (time 1746146436.15 total 50.01 last 179.57) total: 10000 open/close in 55.13 seconds: 181.39 ops/second - unlinked 0 (time 1746146442 ; total 0 ; last 0) total: 10000 unlinks in 26 seconds: 384.615387 unlinks/second - open/close 3054 (time 1746146479.49 total 10.00 last 305.34) - open/close 5538 (time 1746146489.49 total 20.00 last 248.39) - open/close 7592 (time 1746146499.49 total 30.00 last 205.37) total: 10000 open/close in 39.89 seconds: 250.67 ops/second - unlinked 0 (time 1746146510 ; total 0 ; last 0) total: 10000 unlinks in 31 seconds: 322.580658 unlinks/second - open/close 2271 (time 1746146552.85 total 10.00 last 227.08) - open/close 4987 (time 1746146562.85 total 20.00 last 271.50) - open/close 7608 (time 1746146572.85 total 30.01 last 262.05) total: 10000 open/close in 38.86 seconds: 257.36 ops/second - unlinked 0 (time 1746146582 ; total 0 ; last 0) total: 10000 unlinks in 16 seconds: 625.000000 unlinks/second - open/close 2781 (time 1746146609.36 total 10.00 last 278.06) - open/close 6417 (time 1746146619.36 total 20.00 last 363.58) - open/close 8635 (time 1746146629.36 total 30.00 last 221.75) total: 10000 open/close in 36.43 seconds: 274.49 ops/second - unlinked 0 (time 1746146637 ; total 0 ; last 0) total: 10000 unlinks in 30 seconds: 333.333344 unlinks/second - open/close 1765 (time 1746146679.12 total 10.01 last 176.39) - open/close 4566 (time 1746146689.12 total 20.01 last 280.09) - open/close 7043 (time 1746146699.12 total 30.01 last 247.68) total: 10000 open/close in 39.12 seconds: 255.63 ops/second - unlinked 0 (time 1746146709 ; total 0 ; last 0) total: 10000 unlinks in 20 seconds: 500.000000 unlinks/second - open/close 3403 (time 1746146740.06 total 10.00 last 340.22) - open/close 6530 (time 1746146750.06 total 20.00 last 312.68) - open/close 8657 (time 1746146760.06 total 30.00 last 212.68) total: 10000 open/close in 36.58 seconds: 273.39 ops/second - unlinked 0 (time 1746146767 ; total 0 ; last 0) total: 10000 unlinks in 20 seconds: 500.000000 unlinks/second - open/close 2949 (time 1746146798.82 total 10.00 last 294.88) - open/close 5265 (time 1746146808.82 total 20.00 last 231.58) - open/close 7845 (time 1746146818.82 total 30.00 last 257.99) - open/close 9646 (time 1746146828.83 total 40.01 last 179.94) total: 10000 open/close in 42.47 seconds: 235.49 ops/second - unlinked 0 (time 1746146832 ; total 0 ; last 0) total: 10000 unlinks in 27 seconds: 370.370361 unlinks/second - open/close 2448 (time 1746146870.62 total 10.00 last 244.78) - open/close 4472 (time 1746146880.63 total 20.00 last 202.32) - open/close 5894 (time 1746146890.63 total 30.00 last 142.20) - open/close 7693 (time 1746146900.63 total 40.01 last 179.86) - open/close 9158 (time 1746146910.63 total 50.01 last 146.41) total: 10000 open/close in 55.54 seconds: 180.06 ops/second - unlinked 0 (time 1746146917 ; total 0 ; last 0) total: 10000 unlinks in 24 seconds: 416.666656 unlinks/second - open/close 2491 (time 1746146953.12 total 10.00 last 249.07) - open/close 4746 (time 1746146963.12 total 20.01 last 225.38) - open/close 7232 (time 1746146973.13 total 30.01 last 248.57) - open/close 9417 (time 1746146983.13 total 40.01 last 218.49) total: 10000 open/close in 43.30 seconds: 230.95 ops/second - unlinked 0 (time 1746146987 ; total 0 ; last 0) total: 10000 unlinks in 33 seconds: 303.030304 unlinks/second - open/close 1637 (time 1746147032.58 total 10.00 last 163.67) - open/close 3413 (time 1746147042.59 total 20.01 last 177.53) - open/close 4961 (time 1746147052.59 total 30.01 last 154.77) - open/close 6274 (time 1746147062.59 total 40.01 last 131.23) - open/close 7714 (time 1746147072.60 total 50.02 last 143.95) - open/close 9052 (time 1746147082.60 total 60.02 last 133.73) total: 10000 open/close in 68.36 seconds: 146.28 ops/second - unlinked 0 (time 1746147092 ; total 0 ; last 0) total: 10000 unlinks in 39 seconds: 256.410248 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg332-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg332-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg332-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.203.132@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.203.132@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg332-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg332-server: oleg332-server.virtnet: executing set_default_debug -1 all pdsh@oleg332-client: oleg332-server: ssh exited with exit code 1 Started lustre-OST0000 oleg332-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg332-server: oleg332-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg332-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec mount lustre on /mnt/lustre..... Starting client: oleg332-client.virtnet: -o user_xattr,flock 192.168.203.132@tcp:/lustre /mnt/lustre On OST0, 4157 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg332-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg332-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg332-server stop mds service on oleg332-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg332-server unloading modules via unload_modules_local on: 'oleg332-server' oleg332-server: oleg332-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg332-client: oleg332-client: ssh exited with exit code 2 pdsh@oleg332-client: oleg332-server: ssh exited with exit code 2 pdsh@oleg332-client: oleg332-client: ssh exited with exit code 2 pdsh@oleg332-client: oleg332-server: ssh exited with exit code 2 PASS 69 (825s) == conf-sanity test 111: Adding large_dir with over 2GB directory ========================================================== 20:53:02 (1746147182) SKIP: conf-sanity test_111 Only applicable to ldiskfs-based MDTs SKIP 111 (1s) Stopping clients: oleg332-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg332-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg332-client: oleg332-server: ssh exited with exit code 2 oleg332-server: oleg332-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg332-server' oleg332-server: oleg332-server.virtnet: executing load_modules_local oleg332-server: Loading modules from /home/green/git/lustre-release/lustre oleg332-server: detected 4 online CPUs by sysfs oleg332-server: MODOPTS_LIBCFS= oleg332-server: Force libcfs to create 2 CPU partitions oleg332-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg332-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg332-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 1076 sec ========== 20:53:35 (1746147215) === conf-sanity: start cleanup 20:53:36 (1746147216) === === conf-sanity: finish cleanup 20:53:38 (1746147218) ===