-----============= acceptance-small: conf-sanity ============----- Thu May 1 15:03:46 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg224-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32b 32c 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a Stopping clients: oleg224-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg224-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg224-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg224-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg224-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg224-server oleg224-server: oleg224-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory loading modules on: 'oleg224-server' oleg224-server: oleg224-server.virtnet: executing load_modules_local oleg224-server: Loading modules from /home/green/git/lustre-release/lustre oleg224-server: detected 4 online CPUs by sysfs oleg224-server: MODOPTS_LIBCFS= oleg224-server: Force libcfs to create 2 CPU partitions oleg224-server: libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg224-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg224-server: oleg224-server.virtnet: executing set_default_debug -1 all pdsh@oleg224-client: oleg224-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg224-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg224-server: oleg224-server.virtnet: executing set_default_debug -1 all pdsh@oleg224-client: oleg224-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg224-server: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg224-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg224-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg224-server stop mds service on oleg224-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg224-server == conf-sanity test 45: long unlink handling in ptlrpcd == 15:05:29 (1746126329) start mds service on oleg224-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg224-server: oleg224-server.virtnet: executing set_default_debug -1 all pdsh@oleg224-client: oleg224-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg224-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg224-server: oleg224-server.virtnet: executing set_default_debug -1 all pdsh@oleg224-client: oleg224-server: ssh exited with exit code 1 Started lustre-OST0000 oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg224-client.virtnet: -o user_xattr,flock 192.168.202.124@tcp:/lustre /mnt/lustre setup single mount lustre success stop mds service on oleg224-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg224-server sleep 60 sec fail_loc=0x8000050f sleep 10 sec manual umount lustre on /mnt/lustre.... df: '/mnt/lustre': Cannot send after transport endpoint shutdown fail_loc=0x0 start mds service on oleg224-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg224-server: oleg224-server.virtnet: executing set_default_debug -1 all pdsh@oleg224-client: oleg224-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg224-client.virtnet: -o user_xattr,flock 192.168.202.124@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg224-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg224-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg224-server stop mds service on oleg224-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg224-server unloading modules via unload_modules_local on: 'oleg224-server' oleg224-server: oleg224-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg224-client: oleg224-client: ssh exited with exit code 2 pdsh@oleg224-client: oleg224-server: ssh exited with exit code 2 pdsh@oleg224-client: oleg224-client: ssh exited with exit code 2 pdsh@oleg224-client: oleg224-server: ssh exited with exit code 2 PASS 45 (130s) == conf-sanity test 69: replace an OST with the same index ========================================================== 15:07:39 (1746126459) start mds service on oleg224-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg224-server' oleg224-server: oleg224-server.virtnet: executing load_modules_local oleg224-server: Loading modules from /home/green/git/lustre-release/lustre oleg224-server: detected 4 online CPUs by sysfs oleg224-server: MODOPTS_LIBCFS= oleg224-server: Force libcfs to create 2 CPU partitions oleg224-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg224-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg224-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg224-server: oleg224-server.virtnet: executing set_default_debug -1 all pdsh@oleg224-client: oleg224-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg224-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg224-server: oleg224-server.virtnet: executing set_default_debug -1 all pdsh@oleg224-client: oleg224-server: ssh exited with exit code 1 Started lustre-OST0000 oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg224-client.virtnet: -o user_xattr,flock 192.168.202.124@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 1284 (time 1746126500.24 total 10.00 last 128.39) - open/close 2493 (time 1746126510.25 total 20.01 last 120.75) - open/close 3555 (time 1746126520.26 total 30.02 last 106.14) - open/close 4725 (time 1746126530.26 total 40.02 last 116.98) - open/close 5943 (time 1746126540.26 total 50.02 last 121.78) - open/close 7144 (time 1746126550.27 total 60.03 last 120.07) - open/close 8317 (time 1746126560.28 total 70.04 last 117.13) - open/close 9848 (time 1746126570.29 total 80.05 last 152.93) total: 10000 open/close in 80.98 seconds: 123.49 ops/second - unlinked 0 (time 1746126572 ; total 0 ; last 0) total: 10000 unlinks in 38 seconds: 263.157898 unlinks/second - open/close 1790 (time 1746126621.75 total 10.00 last 178.97) - open/close 3227 (time 1746126631.76 total 20.00 last 143.69) - open/close 5088 (time 1746126641.76 total 30.01 last 186.03) - open/close 6785 (time 1746126651.76 total 40.01 last 169.64) - open/close 8511 (time 1746126661.76 total 50.01 last 172.57) total: 10000 open/close in 58.40 seconds: 171.23 ops/second - unlinked 0 (time 1746126671 ; total 0 ; last 0) total: 10000 unlinks in 35 seconds: 285.714294 unlinks/second - open/close 1743 (time 1746126717.79 total 10.00 last 174.29) - open/close 3319 (time 1746126727.80 total 20.01 last 157.39) - open/close 4890 (time 1746126737.80 total 30.01 last 157.09) - open/close 6412 (time 1746126747.81 total 40.03 last 152.02) - open/close 7915 (time 1746126757.81 total 50.03 last 150.29) - open/close 9388 (time 1746126767.82 total 60.04 last 147.18) total: 10000 open/close in 65.24 seconds: 153.29 ops/second - unlinked 0 (time 1746126774 ; total 0 ; last 0) total: 10000 unlinks in 48 seconds: 208.333328 unlinks/second - open/close 1397 (time 1746126833.69 total 10.00 last 139.65) - open/close 3158 (time 1746126843.69 total 20.00 last 176.08) - open/close 4658 (time 1746126853.70 total 30.01 last 149.94) - open/close 5911 (time 1746126863.71 total 40.02 last 125.16) - open/close 7259 (time 1746126873.71 total 50.02 last 134.76) - open/close 8542 (time 1746126883.71 total 60.02 last 128.29) total: 10000 open/close in 69.71 seconds: 143.45 ops/second - unlinked 0 (time 1746126894 ; total 0 ; last 0) total: 10000 unlinks in 35 seconds: 285.714294 unlinks/second - open/close 1841 (time 1746126941.07 total 10.00 last 184.08) - open/close 3450 (time 1746126951.07 total 20.00 last 160.86) - open/close 5094 (time 1746126961.07 total 30.01 last 164.37) - open/close 6861 (time 1746126971.08 total 40.02 last 176.52) - open/close 8443 (time 1746126981.09 total 50.02 last 158.17) - open/close 9774 (time 1746126991.09 total 60.02 last 133.06) total: 10000 open/close in 61.71 seconds: 162.05 ops/second - unlinked 0 (time 1746126994 ; total 0 ; last 0) total: 10000 unlinks in 43 seconds: 232.558136 unlinks/second - open/close 1420 (time 1746127048.76 total 10.00 last 141.99) - open/close 2673 (time 1746127058.78 total 20.02 last 125.09) - open/close 3978 (time 1746127068.79 total 30.03 last 130.39) - open/close 5536 (time 1746127078.79 total 40.03 last 155.72) - open/close 7132 (time 1746127088.80 total 50.04 last 159.50) - open/close 8478 (time 1746127098.80 total 60.04 last 134.56) - open/close 9873 (time 1746127108.80 total 70.04 last 139.48) total: 10000 open/close in 71.01 seconds: 140.83 ops/second - unlinked 0 (time 1746127111 ; total 0 ; last 0) total: 10000 unlinks in 39 seconds: 256.410248 unlinks/second - open/close 1459 (time 1746127162.31 total 10.00 last 145.88) - open/close 3049 (time 1746127172.32 total 20.00 last 158.95) - open/close 4436 (time 1746127182.32 total 30.01 last 138.68) - open/close 5919 (time 1746127192.32 total 40.01 last 148.21) - open/close 7608 (time 1746127202.32 total 50.01 last 168.89) - open/close 9285 (time 1746127212.33 total 60.01 last 167.68) total: 10000 open/close in 64.77 seconds: 154.40 ops/second - unlinked 0 (time 1746127218 ; total 0 ; last 0) total: 10000 unlinks in 38 seconds: 263.157898 unlinks/second - open/close 1821 (time 1746127267.66 total 10.00 last 182.03) - open/close 3712 (time 1746127277.66 total 20.01 last 189.04) - open/close 5491 (time 1746127287.67 total 30.01 last 177.83) - open/close 6894 (time 1746127297.67 total 40.02 last 140.21) - open/close 8022 (time 1746127307.68 total 50.03 last 112.69) - open/close 9369 (time 1746127317.68 total 60.03 last 134.68) total: 10000 open/close in 64.77 seconds: 154.39 ops/second - unlinked 0 (time 1746127324 ; total 0 ; last 0) total: 10000 unlinks in 39 seconds: 256.410248 unlinks/second - open/close 1289 (time 1746127375.11 total 10.00 last 128.85) - open/close 2613 (time 1746127385.12 total 20.01 last 132.38) - open/close 4114 (time 1746127395.12 total 30.01 last 150.08) - open/close 5700 (time 1746127405.12 total 40.01 last 158.48) - open/close 6960 (time 1746127415.13 total 50.02 last 125.93) - open/close 8368 (time 1746127425.14 total 60.03 last 140.65) - open/close 9572 (time 1746127435.15 total 70.04 last 120.32) total: 10000 open/close in 73.92 seconds: 135.29 ops/second - unlinked 0 (time 1746127441 ; total 0 ; last 0) total: 10000 unlinks in 48 seconds: 208.333328 unlinks/second - open/close 1238 (time 1746127500.83 total 10.00 last 123.78) - open/close 2976 (time 1746127510.84 total 20.01 last 173.63) - open/close 4485 (time 1746127520.84 total 30.01 last 150.85) - open/close 5868 (time 1746127530.84 total 40.01 last 138.30) - open/close 7751 (time 1746127540.85 total 50.02 last 188.24) - open/close 9233 (time 1746127550.85 total 60.02 last 148.10) total: 10000 open/close in 66.46 seconds: 150.47 ops/second - unlinked 0 (time 1746127558 ; total 0 ; last 0) total: 10000 unlinks in 47 seconds: 212.765961 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg224-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg224-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg224-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.202.124@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.202.124@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg224-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg224-server: oleg224-server.virtnet: executing set_default_debug -1 all pdsh@oleg224-client: oleg224-server: ssh exited with exit code 1 Started lustre-OST0000 oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg224-server: oleg224-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg224-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 5 sec mount lustre on /mnt/lustre..... Starting client: oleg224-client.virtnet: -o user_xattr,flock 192.168.202.124@tcp:/lustre /mnt/lustre On OST0, 17717 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg224-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg224-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg224-server stop mds service on oleg224-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg224-server unloading modules via unload_modules_local on: 'oleg224-server' oleg224-server: oleg224-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg224-client: oleg224-server: ssh exited with exit code 2 pdsh@oleg224-client: oleg224-client: ssh exited with exit code 2 pdsh@oleg224-client: oleg224-server: ssh exited with exit code 2 pdsh@oleg224-client: oleg224-client: ssh exited with exit code 2 PASS 69 (1197s) == conf-sanity test 111: Adding large_dir with over 2GB directory ========================================================== 15:27:36 (1746127656) SKIP: conf-sanity test_111 Only applicable to ldiskfs-based MDTs SKIP 111 (2s) Stopping clients: oleg224-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg224-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg224-client: oleg224-server: ssh exited with exit code 2 oleg224-server: oleg224-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg224-server' oleg224-server: oleg224-server.virtnet: executing load_modules_local oleg224-server: Loading modules from /home/green/git/lustre-release/lustre oleg224-server: detected 4 online CPUs by sysfs oleg224-server: MODOPTS_LIBCFS= oleg224-server: Force libcfs to create 2 CPU partitions oleg224-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg224-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 1465 sec ========== 15:28:12 (1746127692) === conf-sanity: start cleanup 15:28:13 (1746127693) === === conf-sanity: finish cleanup 15:28:14 (1746127694) ===