-----============= acceptance-small: conf-sanity ============----- Mon Mar 10 13:09:17 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg226-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32b 32c 32newtarball 32b 32c 32e 32h 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a Stopping clients: oleg226-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg226-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg226-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg226-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg226-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg226-server oleg226-server: oleg226-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg226-server' oleg226-server: oleg226-server.virtnet: executing load_modules_local oleg226-server: Loading modules from /home/green/git/lustre-release/lustre oleg226-server: detected 4 online CPUs by sysfs oleg226-server: MODOPTS_LIBCFS= oleg226-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg226-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg226-server: oleg226-server.virtnet: executing set_default_debug -1 all pdsh@oleg226-client: oleg226-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg226-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg226-server: oleg226-server.virtnet: executing set_default_debug -1 all pdsh@oleg226-client: oleg226-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg226-server: oleg226-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg226-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg226-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg226-server stop mds service on oleg226-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg226-server == conf-sanity test 45: long unlink handling in ptlrpcd == 13:10:18 (1741626618) start mds service on oleg226-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg226-server: oleg226-server.virtnet: executing set_default_debug -1 all pdsh@oleg226-client: oleg226-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg226-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg226-server: oleg226-server.virtnet: executing set_default_debug -1 all pdsh@oleg226-client: oleg226-server: ssh exited with exit code 1 Started lustre-OST0000 oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg226-client.virtnet: -o user_xattr,flock 192.168.202.126@tcp:/lustre /mnt/lustre setup single mount lustre success stop mds service on oleg226-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg226-server sleep 60 sec fail_loc=0x8000050f sleep 10 sec manual umount lustre on /mnt/lustre.... df: '/mnt/lustre': Cannot send after transport endpoint shutdown fail_loc=0x0 start mds service on oleg226-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg226-server: oleg226-server.virtnet: executing set_default_debug -1 all pdsh@oleg226-client: oleg226-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg226-client.virtnet: -o user_xattr,flock 192.168.202.126@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg226-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg226-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg226-server stop mds service on oleg226-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg226-server unloading modules via unload_modules_local on: 'oleg226-server' oleg226-server: oleg226-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg226-client: oleg226-server: ssh exited with exit code 2 pdsh@oleg226-client: oleg226-client: ssh exited with exit code 2 pdsh@oleg226-client: oleg226-client: ssh exited with exit code 2 pdsh@oleg226-client: oleg226-server: ssh exited with exit code 2 PASS 45 (130s) == conf-sanity test 69: replace an OST with the same index ========================================================== 13:12:28 (1741626748) start mds service on oleg226-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg226-server' oleg226-server: oleg226-server.virtnet: executing load_modules_local oleg226-server: Loading modules from /home/green/git/lustre-release/lustre oleg226-server: detected 4 online CPUs by sysfs oleg226-server: MODOPTS_LIBCFS= oleg226-server: Force libcfs to create 2 CPU partitions oleg226-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg226-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg226-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg226-server: oleg226-server.virtnet: executing set_default_debug -1 all pdsh@oleg226-client: oleg226-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg226-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg226-server: oleg226-server.virtnet: executing set_default_debug -1 all pdsh@oleg226-client: oleg226-server: ssh exited with exit code 1 Started lustre-OST0000 oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg226-client.virtnet: -o user_xattr,flock 192.168.202.126@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 2997 (time 1741626786.79 total 10.00 last 299.69) - open/close 6199 (time 1741626796.79 total 20.00 last 320.14) - open/close 9166 (time 1741626806.79 total 30.00 last 296.68) total: 10000 open/close in 32.99 seconds: 303.10 ops/second - unlinked 0 (time 1741626810 ; total 0 ; last 0) total: 10000 unlinks in 19 seconds: 526.315796 unlinks/second - open/close 2990 (time 1741626840.07 total 10.00 last 298.96) - open/close 6205 (time 1741626850.07 total 20.00 last 321.43) - open/close 9356 (time 1741626860.07 total 30.00 last 315.08) total: 10000 open/close in 32.56 seconds: 307.16 ops/second - unlinked 0 (time 1741626863 ; total 0 ; last 0) total: 10000 unlinks in 17 seconds: 588.235291 unlinks/second - open/close 3171 (time 1741626891.09 total 10.00 last 317.07) - open/close 6713 (time 1741626901.09 total 20.01 last 354.05) - open/close 9637 (time 1741626911.09 total 30.01 last 292.30) total: 10000 open/close in 31.27 seconds: 319.84 ops/second - unlinked 0 (time 1741626913 ; total 0 ; last 0) total: 10000 unlinks in 19 seconds: 526.315796 unlinks/second - open/close 2756 (time 1741626943.59 total 10.00 last 275.58) - open/close 5433 (time 1741626953.59 total 20.00 last 267.59) - open/close 7814 (time 1741626963.59 total 30.01 last 238.03) - open/close 9765 (time 1741626973.59 total 40.01 last 195.08) total: 10000 open/close in 41.17 seconds: 242.87 ops/second - unlinked 0 (time 1741626975 ; total 0 ; last 0) total: 10000 unlinks in 25 seconds: 400.000000 unlinks/second - open/close 2486 (time 1741627011.58 total 10.00 last 248.57) - open/close 4982 (time 1741627021.59 total 20.00 last 249.54) - open/close 7527 (time 1741627031.59 total 30.00 last 254.50) total: 10000 open/close in 38.45 seconds: 260.10 ops/second - unlinked 0 (time 1741627041 ; total 1 ; last 1) total: 10000 unlinks in 20 seconds: 500.000000 unlinks/second - open/close 2958 (time 1741627071.58 total 10.00 last 295.77) - open/close 5864 (time 1741627081.58 total 20.00 last 290.52) - open/close 8442 (time 1741627091.58 total 30.00 last 257.78) total: 10000 open/close in 35.76 seconds: 279.65 ops/second - unlinked 0 (time 1741627098 ; total 0 ; last 0) total: 10000 unlinks in 20 seconds: 500.000000 unlinks/second - open/close 3036 (time 1741627129.29 total 10.00 last 303.51) - open/close 5517 (time 1741627139.29 total 20.00 last 248.06) - open/close 7919 (time 1741627149.29 total 30.01 last 240.12) total: 10000 open/close in 37.21 seconds: 268.75 ops/second - unlinked 0 (time 1741627157 ; total 0 ; last 0) total: 10000 unlinks in 17 seconds: 588.235291 unlinks/second - open/close 3336 (time 1741627185.25 total 10.00 last 333.56) - open/close 6581 (time 1741627195.26 total 20.00 last 324.42) total: 10000 open/close in 28.87 seconds: 346.36 ops/second - unlinked 0 (time 1741627204 ; total 0 ; last 0) total: 10000 unlinks in 14 seconds: 714.285706 unlinks/second - open/close 4119 (time 1741627229.59 total 10.00 last 411.90) - open/close 8611 (time 1741627239.60 total 20.00 last 449.18) total: 10000 open/close in 22.96 seconds: 435.49 ops/second - unlinked 0 (time 1741627243 ; total 0 ; last 0) total: 10000 unlinks in 12 seconds: 833.333313 unlinks/second - open/close 4648 (time 1741627266.25 total 10.00 last 464.75) - open/close 8988 (time 1741627276.25 total 20.00 last 433.99) total: 10000 open/close in 22.12 seconds: 452.15 ops/second - unlinked 0 (time 1741627279 ; total 0 ; last 0) total: 10000 unlinks in 11 seconds: 909.090881 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg226-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg226-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg226-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.202.126@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.202.126@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg226-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg226-server: oleg226-server.virtnet: executing set_default_debug -1 all pdsh@oleg226-client: oleg226-server: ssh exited with exit code 1 Started lustre-OST0000 oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg226-server: oleg226-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg226-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 10 sec mount lustre on /mnt/lustre..... Starting client: oleg226-client.virtnet: -o user_xattr,flock 192.168.202.126@tcp:/lustre /mnt/lustre On OST0, 15317 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg226-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg226-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg226-server stop mds service on oleg226-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg226-server unloading modules via unload_modules_local on: 'oleg226-server' oleg226-server: oleg226-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg226-client: oleg226-client: ssh exited with exit code 2 pdsh@oleg226-client: oleg226-server: ssh exited with exit code 2 pdsh@oleg226-client: oleg226-client: ssh exited with exit code 2 pdsh@oleg226-client: oleg226-server: ssh exited with exit code 2 PASS 69 (580s) == conf-sanity test 111: Adding large_dir with over 2GB directory ========================================================== 13:22:08 (1741627328) SKIP: conf-sanity test_111 Only applicable to ldiskfs-based MDTs SKIP 111 (1s) Stopping clients: oleg226-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg226-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg226-client: oleg226-server: ssh exited with exit code 2 oleg226-server: oleg226-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg226-server' oleg226-server: oleg226-server.virtnet: executing load_modules_local oleg226-server: Loading modules from /home/green/git/lustre-release/lustre oleg226-server: detected 4 online CPUs by sysfs oleg226-server: MODOPTS_LIBCFS= oleg226-server: Force libcfs to create 2 CPU partitions oleg226-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory oleg226-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg226-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg226-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 790 sec =========== 13:22:27 (1741627347) === conf-sanity: start cleanup 13:22:27 (1741627347) === === conf-sanity: finish cleanup 13:22:28 (1741627348) ===