-----============= acceptance-small: conf-sanity ============----- Thu Mar 13 16:31:53 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg334-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32b 32c 32newtarball 32b 32c 32e 32h 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a Stopping clients: oleg334-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg334-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg334-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg334-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg334-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg334-server oleg334-server: oleg334-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory loading modules on: 'oleg334-server' oleg334-server: oleg334-server.virtnet: executing load_modules_local oleg334-server: Loading modules from /home/green/git/lustre-release/lustre oleg334-server: detected 4 online CPUs by sysfs oleg334-server: MODOPTS_LIBCFS= oleg334-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg334-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg334-server: oleg334-server.virtnet: executing set_default_debug -1 all pdsh@oleg334-client: oleg334-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg334-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg334-server: oleg334-server.virtnet: executing set_default_debug -1 all pdsh@oleg334-client: oleg334-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg334-server: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg334-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg334-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg334-server stop mds service on oleg334-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg334-server == conf-sanity test 45: long unlink handling in ptlrpcd == 16:33:12 (1741897992) start mds service on oleg334-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg334-server: oleg334-server.virtnet: executing set_default_debug -1 all pdsh@oleg334-client: oleg334-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg334-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg334-server: oleg334-server.virtnet: executing set_default_debug -1 all pdsh@oleg334-client: oleg334-server: ssh exited with exit code 1 Started lustre-OST0000 oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg334-client.virtnet: -o user_xattr,flock 192.168.203.134@tcp:/lustre /mnt/lustre setup single mount lustre success stop mds service on oleg334-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg334-server sleep 60 sec fail_loc=0x8000050f sleep 10 sec manual umount lustre on /mnt/lustre.... df: '/mnt/lustre': Cannot send after transport endpoint shutdown fail_loc=0x0 start mds service on oleg334-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg334-server: oleg334-server.virtnet: executing set_default_debug -1 all pdsh@oleg334-client: oleg334-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg334-client.virtnet: -o user_xattr,flock 192.168.203.134@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg334-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg334-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg334-server stop mds service on oleg334-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg334-server unloading modules via unload_modules_local on: 'oleg334-server' oleg334-server: oleg334-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg334-client: oleg334-client: ssh exited with exit code 2 pdsh@oleg334-client: oleg334-server: ssh exited with exit code 2 pdsh@oleg334-client: oleg334-client: ssh exited with exit code 2 pdsh@oleg334-client: oleg334-server: ssh exited with exit code 2 PASS 45 (145s) == conf-sanity test 69: replace an OST with the same index ========================================================== 16:35:38 (1741898138) start mds service on oleg334-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg334-server' oleg334-server: oleg334-server.virtnet: executing load_modules_local oleg334-server: Loading modules from /home/green/git/lustre-release/lustre oleg334-server: detected 4 online CPUs by sysfs oleg334-server: MODOPTS_LIBCFS= oleg334-server: Force libcfs to create 2 CPU partitions oleg334-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg334-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg334-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg334-server: oleg334-server.virtnet: executing set_default_debug -1 all pdsh@oleg334-client: oleg334-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg334-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg334-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg334-server: oleg334-server.virtnet: executing set_default_debug -1 all pdsh@oleg334-client: oleg334-server: ssh exited with exit code 1 Started lustre-OST0000 oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg334-client.virtnet: -o user_xattr,flock 192.168.203.134@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 1501 (time 1741898178.95 total 10.00 last 150.05) - open/close 2730 (time 1741898188.96 total 20.01 last 122.79) - open/close 4145 (time 1741898198.96 total 30.01 last 141.47) - open/close 5649 (time 1741898208.97 total 40.02 last 150.36) - open/close 7190 (time 1741898218.97 total 50.02 last 154.00) - open/close 8914 (time 1741898228.98 total 60.03 last 172.34) total: 10000 open/close in 66.67 seconds: 150.00 ops/second - unlinked 0 (time 1741898236 ; total 0 ; last 0) total: 10000 unlinks in 38 seconds: 263.157898 unlinks/second - open/close 1200 (time 1741898286.22 total 10.00 last 119.95) - open/close 2440 (time 1741898296.23 total 20.00 last 123.99) - open/close 3840 (time 1741898306.23 total 30.01 last 139.99) - open/close 5617 (time 1741898316.23 total 40.01 last 177.69) - open/close 6799 (time 1741898326.23 total 50.01 last 118.18) - open/close 8127 (time 1741898336.23 total 60.01 last 132.76) - open/close 9532 (time 1741898346.24 total 70.02 last 140.41) total: 10000 open/close in 73.30 seconds: 136.43 ops/second - unlinked 0 (time 1741898350 ; total 0 ; last 0) total: 10000 unlinks in 42 seconds: 238.095245 unlinks/second - open/close 1611 (time 1741898404.02 total 10.00 last 161.10) - open/close 3108 (time 1741898414.02 total 20.00 last 149.65) - open/close 4493 (time 1741898424.02 total 30.00 last 138.49) - open/close 6087 (time 1741898434.03 total 40.01 last 159.34) - open/close 7451 (time 1741898444.03 total 50.01 last 136.36) - open/close 8943 (time 1741898454.03 total 60.01 last 149.15) total: 10000 open/close in 66.92 seconds: 149.44 ops/second - unlinked 0 (time 1741898462 ; total 0 ; last 0) total: 10000 unlinks in 41 seconds: 243.902435 unlinks/second - open/close 1498 (time 1741898514.71 total 10.00 last 149.74) - open/close 3062 (time 1741898524.72 total 20.01 last 156.34) - open/close 4496 (time 1741898534.72 total 30.01 last 143.38) - open/close 5916 (time 1741898544.72 total 40.01 last 142.00) - open/close 7255 (time 1741898554.72 total 50.01 last 133.85) - open/close 8704 (time 1741898564.72 total 60.02 last 144.85) total: 10000 open/close in 68.48 seconds: 146.04 ops/second - unlinked 0 (time 1741898574 ; total 0 ; last 0) total: 10000 unlinks in 31 seconds: 322.580658 unlinks/second - open/close 1775 (time 1741898617.22 total 10.00 last 177.44) - open/close 3636 (time 1741898627.22 total 20.01 last 186.03) - open/close 5609 (time 1741898637.22 total 30.01 last 197.29) - open/close 7606 (time 1741898647.23 total 40.01 last 199.69) - open/close 9453 (time 1741898657.23 total 50.01 last 184.68) total: 10000 open/close in 53.07 seconds: 188.42 ops/second - unlinked 0 (time 1741898661 ; total 0 ; last 0) total: 10000 unlinks in 34 seconds: 294.117645 unlinks/second - open/close 1513 (time 1741898707.35 total 10.00 last 151.29) - open/close 3234 (time 1741898717.35 total 20.01 last 171.98) - open/close 5123 (time 1741898727.36 total 30.01 last 188.81) - open/close 7726 (time 1741898737.36 total 40.01 last 260.28) total: 10000 open/close in 49.17 seconds: 203.36 ops/second - unlinked 0 (time 1741898747 ; total 0 ; last 0) total: 10000 unlinks in 18 seconds: 555.555542 unlinks/second - open/close 3115 (time 1741898776.59 total 10.00 last 311.39) - open/close 5906 (time 1741898786.59 total 20.00 last 279.08) - open/close 8919 (time 1741898796.59 total 30.01 last 301.26) total: 10000 open/close in 33.21 seconds: 301.11 ops/second - unlinked 0 (time 1741898800 ; total 0 ; last 0) total: 10000 unlinks in 19 seconds: 526.315796 unlinks/second - open/close 2084 (time 1741898830.97 total 10.01 last 208.25) - open/close 4013 (time 1741898840.97 total 20.01 last 192.86) - open/close 5950 (time 1741898850.98 total 30.01 last 193.63) - open/close 7785 (time 1741898860.98 total 40.01 last 183.48) - open/close 9861 (time 1741898870.98 total 50.02 last 207.54) total: 10000 open/close in 50.77 seconds: 196.95 ops/second - unlinked 0 (time 1741898872 ; total 0 ; last 0) total: 10000 unlinks in 25 seconds: 400.000000 unlinks/second - open/close 1902 (time 1741898909.18 total 10.01 last 190.09) - open/close 3838 (time 1741898919.19 total 20.01 last 193.46) - open/close 5865 (time 1741898929.19 total 30.02 last 202.63) - open/close 8012 (time 1741898939.19 total 40.02 last 214.65) total: 10000 open/close in 49.76 seconds: 200.95 ops/second - unlinked 0 (time 1741898950 ; total 0 ; last 0) total: 10000 unlinks in 19 seconds: 526.315796 unlinks/second - open/close 2032 (time 1741898980.57 total 10.00 last 203.16) - open/close 4000 (time 1741898990.58 total 20.01 last 196.69) - open/close 6070 (time 1741899000.58 total 30.01 last 206.92) - open/close 8202 (time 1741899010.58 total 40.01 last 213.14) total: 10000 open/close in 47.57 seconds: 210.21 ops/second - unlinked 0 (time 1741899019 ; total 0 ; last 0) total: 10000 unlinks in 23 seconds: 434.782623 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg334-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg334-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg334-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.203.134@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.203.134@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg334-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg334-server: oleg334-server.virtnet: executing set_default_debug -1 all pdsh@oleg334-client: oleg334-server: ssh exited with exit code 1 Started lustre-OST0000 oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg334-server: oleg334-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg334-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 15 sec mount lustre on /mnt/lustre..... Starting client: oleg334-client.virtnet: -o user_xattr,flock 192.168.203.134@tcp:/lustre /mnt/lustre On OST0, 20325 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg334-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg334-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg334-server stop mds service on oleg334-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg334-server unloading modules via unload_modules_local on: 'oleg334-server' oleg334-server: oleg334-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg334-client: oleg334-client: ssh exited with exit code 2 pdsh@oleg334-client: oleg334-server: ssh exited with exit code 2 pdsh@oleg334-client: oleg334-client: ssh exited with exit code 2 pdsh@oleg334-client: oleg334-server: ssh exited with exit code 2 PASS 69 (954s) == conf-sanity test 111: Adding large_dir with over 2GB directory ========================================================== 16:51:32 (1741899092) SKIP: conf-sanity test_111 Only applicable to ldiskfs-based MDTs SKIP 111 (1s) Stopping clients: oleg334-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg334-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg334-client: oleg334-server: ssh exited with exit code 2 oleg334-server: oleg334-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg334-server' oleg334-server: oleg334-server.virtnet: executing load_modules_local oleg334-server: Loading modules from /home/green/git/lustre-release/lustre oleg334-server: detected 4 online CPUs by sysfs oleg334-server: MODOPTS_LIBCFS= oleg334-server: Force libcfs to create 2 CPU partitions oleg334-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg334-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg334-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 1200 sec ========== 16:51:54 (1741899114) === conf-sanity: start cleanup 16:51:54 (1741899114) === === conf-sanity: finish cleanup 16:51:55 (1741899115) ===