-----============= acceptance-small: conf-sanity ============----- Thu May 1 16:03:51 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg417-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32b 32c 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a Stopping clients: oleg417-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg417-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg417-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg417-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg417-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg417-server oleg417-server: oleg417-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg417-server' oleg417-server: oleg417-server.virtnet: executing load_modules_local oleg417-server: Loading modules from /home/green/git/lustre-release/lustre oleg417-server: detected 4 online CPUs by sysfs oleg417-server: MODOPTS_LIBCFS= oleg417-server: Force libcfs to create 2 CPU partitions oleg417-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg417-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg417-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg417-server: oleg417-server.virtnet: executing set_default_debug -1 all pdsh@oleg417-client: oleg417-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg417-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg417-server: oleg417-server.virtnet: executing set_default_debug -1 all pdsh@oleg417-client: oleg417-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg417-server: oleg417-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg417-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg417-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg417-server stop mds service on oleg417-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg417-server == conf-sanity test 45: long unlink handling in ptlrpcd == 16:04:59 (1746129899) start mds service on oleg417-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg417-server: oleg417-server.virtnet: executing set_default_debug -1 all pdsh@oleg417-client: oleg417-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg417-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg417-server: oleg417-server.virtnet: executing set_default_debug -1 all pdsh@oleg417-client: oleg417-server: ssh exited with exit code 1 Started lustre-OST0000 oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg417-client.virtnet: -o user_xattr,flock 192.168.204.117@tcp:/lustre /mnt/lustre setup single mount lustre success stop mds service on oleg417-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg417-server sleep 60 sec fail_loc=0x8000050f sleep 10 sec manual umount lustre on /mnt/lustre.... df: '/mnt/lustre': Cannot send after transport endpoint shutdown fail_loc=0x0 start mds service on oleg417-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg417-server: oleg417-server.virtnet: executing set_default_debug -1 all pdsh@oleg417-client: oleg417-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg417-client.virtnet: -o user_xattr,flock 192.168.204.117@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg417-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg417-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg417-server stop mds service on oleg417-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg417-server unloading modules via unload_modules_local on: 'oleg417-server' oleg417-server: oleg417-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg417-client: oleg417-client: ssh exited with exit code 2 pdsh@oleg417-client: oleg417-server: ssh exited with exit code 2 pdsh@oleg417-client: oleg417-client: ssh exited with exit code 2 pdsh@oleg417-client: oleg417-server: ssh exited with exit code 2 PASS 45 (125s) == conf-sanity test 69: replace an OST with the same index ========================================================== 16:07:04 (1746130024) start mds service on oleg417-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg417-server' oleg417-server: oleg417-server.virtnet: executing load_modules_local oleg417-server: Loading modules from /home/green/git/lustre-release/lustre oleg417-server: detected 4 online CPUs by sysfs oleg417-server: MODOPTS_LIBCFS= oleg417-server: Force libcfs to create 2 CPU partitions oleg417-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg417-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg417-server: oleg417-server.virtnet: executing set_default_debug -1 all pdsh@oleg417-client: oleg417-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg417-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg417-server: oleg417-server.virtnet: executing set_default_debug -1 all pdsh@oleg417-client: oleg417-server: ssh exited with exit code 1 Started lustre-OST0000 oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg417-client.virtnet: -o user_xattr,flock 192.168.204.117@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 1631 (time 1746130063.08 total 10.01 last 163.00) - open/close 3093 (time 1746130073.09 total 20.01 last 146.08) - open/close 4480 (time 1746130083.10 total 30.02 last 138.59) - open/close 6001 (time 1746130093.10 total 40.02 last 152.08) - open/close 7823 (time 1746130103.10 total 50.03 last 182.14) - open/close 9392 (time 1746130113.11 total 60.04 last 156.74) total: 10000 open/close in 63.56 seconds: 157.33 ops/second - unlinked 0 (time 1746130117 ; total 0 ; last 0) total: 10000 unlinks in 37 seconds: 270.270264 unlinks/second - open/close 990 (time 1746130165.98 total 10.00 last 99.00) - open/close 2220 (time 1746130175.98 total 20.00 last 122.99) - open/close 3509 (time 1746130185.98 total 30.00 last 128.89) - open/close 4907 (time 1746130195.99 total 40.01 last 139.63) - open/close 6132 (time 1746130206.00 total 50.02 last 122.46) - open/close 7516 (time 1746130216.00 total 60.02 last 138.36) - open/close 9110 (time 1746130226.00 total 70.02 last 159.38) total: 10000 open/close in 76.67 seconds: 130.43 ops/second - unlinked 0 (time 1746130234 ; total 0 ; last 0) total: 10000 unlinks in 37 seconds: 270.270264 unlinks/second - open/close 1464 (time 1746130283.50 total 10.00 last 146.34) - open/close 3148 (time 1746130293.50 total 20.01 last 168.35) - open/close 4288 (time 1746130303.51 total 30.01 last 113.94) - open/close 5531 (time 1746130313.51 total 40.02 last 124.22) - open/close 6556 (time 1746130323.52 total 50.02 last 102.49) - open/close 7486 (time 1746130333.53 total 60.03 last 92.91) - open/close 8937 (time 1746130343.53 total 70.04 last 145.02) total: 10000 open/close in 78.72 seconds: 127.04 ops/second - unlinked 0 (time 1746130353 ; total 0 ; last 0) total: 10000 unlinks in 34 seconds: 294.117645 unlinks/second - open/close 1301 (time 1746130398.87 total 10.01 last 129.91) - open/close 2435 (time 1746130408.87 total 20.02 last 113.36) - open/close 3813 (time 1746130418.88 total 30.03 last 137.68) - open/close 5010 (time 1746130428.89 total 40.03 last 119.64) - open/close 6574 (time 1746130438.89 total 50.04 last 156.26) - open/close 7707 (time 1746130448.90 total 60.04 last 113.29) - open/close 9115 (time 1746130458.90 total 70.05 last 140.75) total: 10000 open/close in 74.69 seconds: 133.88 ops/second - unlinked 0 (time 1746130464 ; total 0 ; last 0) total: 10000 unlinks in 35 seconds: 285.714294 unlinks/second - open/close 1775 (time 1746130510.79 total 10.00 last 177.50) - open/close 3337 (time 1746130520.79 total 20.00 last 156.17) - open/close 5219 (time 1746130530.80 total 30.00 last 188.15) - open/close 6811 (time 1746130540.80 total 40.01 last 159.16) - open/close 8073 (time 1746130550.80 total 50.01 last 126.19) - open/close 9594 (time 1746130560.80 total 60.01 last 152.05) total: 10000 open/close in 63.29 seconds: 158.01 ops/second - unlinked 0 (time 1746130565 ; total 0 ; last 0) total: 10000 unlinks in 36 seconds: 277.777771 unlinks/second - open/close 1247 (time 1746130613.16 total 10.01 last 124.61) - open/close 2457 (time 1746130623.17 total 20.01 last 120.97) - open/close 4046 (time 1746130633.17 total 30.02 last 158.77) - open/close 5509 (time 1746130643.17 total 40.02 last 146.30) - open/close 7365 (time 1746130653.18 total 50.02 last 185.54) - open/close 9040 (time 1746130663.19 total 60.03 last 167.32) total: 10000 open/close in 66.61 seconds: 150.13 ops/second - unlinked 0 (time 1746130671 ; total 0 ; last 0) total: 10000 unlinks in 30 seconds: 333.333344 unlinks/second - open/close 1343 (time 1746130712.93 total 10.01 last 134.23) - open/close 2705 (time 1746130722.93 total 20.01 last 136.17) - open/close 4077 (time 1746130732.94 total 30.01 last 137.11) - open/close 5400 (time 1746130742.94 total 40.02 last 132.23) - open/close 6912 (time 1746130752.95 total 50.03 last 151.07) - open/close 8397 (time 1746130762.95 total 60.03 last 148.47) - open/close 9938 (time 1746130772.95 total 70.03 last 154.04) total: 10000 open/close in 70.65 seconds: 141.55 ops/second - unlinked 0 (time 1746130775 ; total 0 ; last 0) total: 10000 unlinks in 34 seconds: 294.117645 unlinks/second - open/close 1990 (time 1746130820.30 total 10.00 last 198.96) - open/close 5032 (time 1746130830.31 total 20.01 last 304.08) - open/close 6965 (time 1746130840.31 total 30.01 last 193.22) - open/close 8877 (time 1746130850.31 total 40.01 last 191.17) total: 10000 open/close in 45.84 seconds: 218.14 ops/second - unlinked 0 (time 1746130857 ; total 0 ; last 0) total: 10000 unlinks in 35 seconds: 285.714294 unlinks/second - open/close 2131 (time 1746130903.86 total 10.00 last 213.03) - open/close 4382 (time 1746130913.87 total 20.00 last 225.06) - open/close 6252 (time 1746130923.87 total 30.01 last 186.99) - open/close 8500 (time 1746130933.87 total 40.01 last 224.77) total: 10000 open/close in 47.15 seconds: 212.10 ops/second - unlinked 0 (time 1746130942 ; total 0 ; last 0) total: 10000 unlinks in 39 seconds: 256.410248 unlinks/second - open/close 1688 (time 1746130993.16 total 10.00 last 168.79) - open/close 3695 (time 1746131003.16 total 20.01 last 200.53) - open/close 5393 (time 1746131013.17 total 30.01 last 169.76) - open/close 7940 (time 1746131023.17 total 40.01 last 254.64) total: 10000 open/close in 49.32 seconds: 202.77 ops/second - unlinked 0 (time 1746131033 ; total 0 ; last 0) total: 10000 unlinks in 23 seconds: 434.782623 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg417-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg417-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg417-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.204.117@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.204.117@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg417-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg417-server: oleg417-server.virtnet: executing set_default_debug -1 all pdsh@oleg417-client: oleg417-server: ssh exited with exit code 1 Started lustre-OST0000 oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg417-server: oleg417-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg417-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 10 sec mount lustre on /mnt/lustre..... Starting client: oleg417-client.virtnet: -o user_xattr,flock 192.168.204.117@tcp:/lustre /mnt/lustre On OST0, 6381 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg417-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg417-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg417-server stop mds service on oleg417-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg417-server unloading modules via unload_modules_local on: 'oleg417-server' oleg417-server: oleg417-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg417-client: oleg417-client: ssh exited with exit code 2 pdsh@oleg417-client: oleg417-server: ssh exited with exit code 2 pdsh@oleg417-client: oleg417-client: ssh exited with exit code 2 pdsh@oleg417-client: oleg417-server: ssh exited with exit code 2 PASS 69 (1079s) == conf-sanity test 111: Adding large_dir with over 2GB directory ========================================================== 16:25:03 (1746131103) SKIP: conf-sanity test_111 Only applicable to ldiskfs-based MDTs SKIP 111 (1s) Stopping clients: oleg417-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg417-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg417-client: oleg417-server: ssh exited with exit code 2 oleg417-server: oleg417-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg417-server' oleg417-server: oleg417-server.virtnet: executing load_modules_local oleg417-server: Loading modules from /home/green/git/lustre-release/lustre oleg417-server: detected 4 online CPUs by sysfs oleg417-server: MODOPTS_LIBCFS= oleg417-server: Force libcfs to create 2 CPU partitions oleg417-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg417-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 1299 sec ========== 16:25:31 (1746131131) === conf-sanity: start cleanup 16:25:32 (1746131132) === === conf-sanity: finish cleanup 16:25:33 (1746131133) ===