-----============= acceptance-small: replay-dual ============----- Mon Dec 22 09:10:29 EST 2025 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg324-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/replay-dual.*ex': No such file or directory excepting tests: 14b 21b 21b skipping tests SLOW=no: 21b === replay-dual: start setup 09:10:46 (1766412646) === Starting client oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre2 Mount client oleg324-client.virtnet: mount -t lustre -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre2 Started clients oleg324-client.virtnet: 192.168.203.124@tcp:/lustre on /mnt/lustre2 type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) oleg324-client.virtnet: executing check_config_client /mnt/lustre oleg324-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg324-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff964dca0e1800.idle_timeout=debug osc.lustre-OST0000-osc-ffff964dcae8f000.idle_timeout=debug osc.lustre-OST0001-osc-ffff964dca0e1800.idle_timeout=debug osc.lustre-OST0001-osc-ffff964dcae8f000.idle_timeout=debug disable quota as required oleg324-server: oleg324-server.virtnet: executing set_default_debug -1 all === replay-dual: finish setup 09:11:12 (1766412672) === == replay-dual test 0a: expired recovery with lost client ========================================================== 09:11:13 (1766412673) Check file is LU482_FAILED=/tmp/replay-dual.lu482.qeTyGs UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre total: 50 open/close in 1.70 seconds: 29.41 ops/second fail_loc=0x80000514 Failing mds1 on oleg324-server Stopping /mnt/lustre-mds1 (opts:) on oleg324-server 09:11:26 (1766412686) shut down facet: mds1 facet_host: oleg324-server facet_failover_host: oleg324-server Failover mds1 to oleg324-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg324-server: oleg324-server.virtnet: executing set_default_debug -1 all pdsh@oleg324-client: oleg324-server: ssh exited with exit code 1 Started lustre-MDT0000 09:11:56 (1766412716) targets are mounted 09:11:56 (1766412716) facet_failover done Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre2 - unlinked 0 (time 1766412839 ; total 0 ; last 0) total: 50 unlinks in 2 seconds: 25.000000 unlinks/second PASS 0a (172s) == replay-dual test 0b: lost client during waiting for next transno ========================================================== 09:14:06 (1766412846) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg324-server Stopping /mnt/lustre-mds1 (opts:) on oleg324-server 09:14:15 (1766412855) shut down facet: mds1 facet_host: oleg324-server facet_failover_host: oleg324-server Failover mds1 to oleg324-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg324-server: oleg324-server.virtnet: executing set_default_debug -1 all pdsh@oleg324-client: oleg324-server: ssh exited with exit code 1 Started lustre-MDT0000 09:14:37 (1766412877) targets are mounted 09:14:37 (1766412877) facet_failover done Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre2 PASS 0b (109s) == replay-dual test 1: |X| simple create ================= 09:15:54 (1766412954) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg324-server Stopping /mnt/lustre-mds1 (opts:) on oleg324-server 09:16:04 (1766412964) shut down facet: mds1 facet_host: oleg324-server facet_failover_host: oleg324-server Failover mds1 to oleg324-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg324-server: oleg324-server.virtnet: executing set_default_debug -1 all pdsh@oleg324-client: oleg324-server: ssh exited with exit code 1 Started lustre-MDT0000 09:16:34 (1766412994) targets are mounted 09:16:34 (1766412994) facet_failover done oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 1 (52s) == replay-dual test 2: |X| mkdir adir ==================== 09:16:46 (1766413006) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg324-server Stopping /mnt/lustre-mds1 (opts:) on oleg324-server 09:16:54 (1766413014) shut down facet: mds1 facet_host: oleg324-server facet_failover_host: oleg324-server Failover mds1 to oleg324-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg324-server: oleg324-server.virtnet: executing set_default_debug -1 all pdsh@oleg324-client: oleg324-server: ssh exited with exit code 1 Started lustre-MDT0000 09:17:12 (1766413032) targets are mounted 09:17:12 (1766413032) facet_failover done oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 2 (36s) == replay-dual test 3: |X| mkdir adir, mkdir adir/bdir === 09:17:22 (1766413042) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg324-server Stopping /mnt/lustre-mds1 (opts:) on oleg324-server 09:17:29 (1766413049) shut down facet: mds1 facet_host: oleg324-server facet_failover_host: oleg324-server Failover mds1 to oleg324-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg324-server: oleg324-server.virtnet: executing set_default_debug -1 all pdsh@oleg324-client: oleg324-server: ssh exited with exit code 1 Started lustre-MDT0000 09:17:49 (1766413069) targets are mounted 09:17:49 (1766413069) facet_failover done oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 3 (39s) == replay-dual test 4: |X| mkdir adir (-EEXIST), mkdir adir/bdir ========================================================== 09:18:01 (1766413081) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre mkdir: cannot create directory '/mnt/lustre/adir': File exists Failing mds1 on oleg324-server Stopping /mnt/lustre-mds1 (opts:) on oleg324-server 09:18:09 (1766413089) shut down facet: mds1 facet_host: oleg324-server facet_failover_host: oleg324-server Failover mds1 to oleg324-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg324-server: oleg324-server.virtnet: executing set_default_debug -1 all pdsh@oleg324-client: oleg324-server: ssh exited with exit code 1 Started lustre-MDT0000 09:18:28 (1766413108) targets are mounted 09:18:28 (1766413108) facet_failover done oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 4 (38s) == replay-dual test 5: open, unlink |X| close ============ 09:18:39 (1766413119) multiop /mnt/lustre2/a vo_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7578 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg324-server Stopping /mnt/lustre-mds1 (opts:) on oleg324-server 09:18:46 (1766413126) shut down facet: mds1 facet_host: oleg324-server facet_failover_host: oleg324-server Failover mds1 to oleg324-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg324-server: oleg324-server.virtnet: executing set_default_debug -1 all pdsh@oleg324-client: oleg324-server: ssh exited with exit code 1 Started lustre-MDT0000 09:19:05 (1766413145) targets are mounted 09:19:05 (1766413145) facet_failover done oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 5 (37s) == replay-dual test 6: open1, open2, unlink |X| close1 [fail mds1] close2 ========================================================== 09:19:16 (1766413156) multiop /mnt/lustre2/a vo_c TMPPIPE=/tmp/multiop_open_wait_pipe.7578 multiop /mnt/lustre/a vo_c TMPPIPE=/tmp/multiop_open_wait_pipe.7578 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg324-server Stopping /mnt/lustre-mds1 (opts:) on oleg324-server 09:19:24 (1766413164) shut down facet: mds1 facet_host: oleg324-server facet_failover_host: oleg324-server Failover mds1 to oleg324-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg324-server: oleg324-server.virtnet: executing set_default_debug -1 all pdsh@oleg324-client: oleg324-server: ssh exited with exit code 1 Started lustre-MDT0000 09:19:43 (1766413183) targets are mounted 09:19:43 (1766413183) facet_failover done oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 6 (40s) == replay-dual test 8: replay of resent request ========== 09:19:56 (1766413196) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre fail_loc=0x119 fail_loc=0 Failing mds1 on oleg324-server Stopping /mnt/lustre-mds1 (opts:) on oleg324-server 09:20:22 (1766413222) shut down facet: mds1 facet_host: oleg324-server facet_failover_host: oleg324-server Failover mds1 to oleg324-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg324-server: oleg324-server.virtnet: executing set_default_debug -1 all pdsh@oleg324-client: oleg324-server: ssh exited with exit code 1 Started lustre-MDT0000 09:20:43 (1766413243) targets are mounted 09:20:43 (1766413243) facet_failover done oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 8 (58s) == replay-dual test 9: resending a replayed create ======= 09:20:54 (1766413254) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre fail_loc=0x80000119 Failing mds1 on oleg324-server Stopping /mnt/lustre-mds1 (opts:) on oleg324-server 09:21:05 (1766413265) shut down facet: mds1 facet_host: oleg324-server facet_failover_host: oleg324-server Failover mds1 to oleg324-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg324-server: oleg324-server.virtnet: executing set_default_debug -1 all pdsh@oleg324-client: oleg324-server: ssh exited with exit code 1 Started lustre-MDT0000 09:21:35 (1766413295) targets are mounted 09:21:35 (1766413295) facet_failover done oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec fail_loc=0 PASS 9 (63s) == replay-dual test 10: resending a replayed unlink ====== 09:21:57 (1766413317) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre fail_loc=0x80000119 Failing mds1 on oleg324-server Stopping /mnt/lustre-mds1 (opts:) on oleg324-server 09:22:06 (1766413326) shut down facet: mds1 facet_host: oleg324-server facet_failover_host: oleg324-server Failover mds1 to oleg324-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg324-server: oleg324-server.virtnet: executing set_default_debug -1 all pdsh@oleg324-client: oleg324-server: ssh exited with exit code 1 Started lustre-MDT0000 09:22:24 (1766413344) targets are mounted 09:22:24 (1766413344) facet_failover done oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec fail_loc=0 PASS 10 (50s) == replay-dual test 11: both clients timeout during replay ========================================================== 09:22:48 (1766413368) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre fail_loc=0x0119 Failing mds1 on oleg324-server Stopping /mnt/lustre-mds1 (opts:) on oleg324-server 09:22:56 (1766413376) shut down facet: mds1 facet_host: oleg324-server facet_failover_host: oleg324-server Failover mds1 to oleg324-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg324-server: oleg324-server.virtnet: executing set_default_debug -1 all pdsh@oleg324-client: oleg324-server: ssh exited with exit code 1 Started lustre-MDT0000 09:23:15 (1766413395) targets are mounted 09:23:15 (1766413395) facet_failover done oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 11 sec fail_loc=0 PASS 11 (49s) == replay-dual test 12: open resend timeout ============== 09:23:37 (1766413417) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre multiop /mnt/lustre/f12.replay-dual vmo_c TMPPIPE=/tmp/multiop_open_wait_pipe.7578 fail_loc=0x80000302 Failing mds1 on oleg324-server Stopping /mnt/lustre-mds1 (opts:) on oleg324-server 09:23:45 (1766413425) shut down facet: mds1 facet_host: oleg324-server facet_failover_host: oleg324-server Failover mds1 to oleg324-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg324-server: oleg324-server.virtnet: executing set_default_debug -1 all pdsh@oleg324-client: oleg324-server: ssh exited with exit code 1 Started lustre-MDT0000 09:24:04 (1766413444) targets are mounted 09:24:04 (1766413444) facet_failover done fail_loc=0 /mnt/lustre/f12.replay-dual /mnt/lustre/f12.replay-dual has type file OK PASS 12 (43s) == replay-dual test 13: close resend timeout ============= 09:24:21 (1766413461) multiop /mnt/lustre/f13.replay-dual vmo_c TMPPIPE=/tmp/multiop_open_wait_pipe.7578 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre fail_loc=0x80000115 Failing mds1 on oleg324-server Stopping /mnt/lustre-mds1 (opts:) on oleg324-server 09:24:28 (1766413468) shut down facet: mds1 facet_host: oleg324-server facet_failover_host: oleg324-server Failover mds1 to oleg324-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg324-server: oleg324-server.virtnet: executing set_default_debug -1 all pdsh@oleg324-client: oleg324-server: ssh exited with exit code 1 Started lustre-MDT0000 09:24:47 (1766413487) targets are mounted 09:24:47 (1766413487) facet_failover done fail_loc=0 /mnt/lustre/f13.replay-dual /mnt/lustre/f13.replay-dual has type file OK PASS 13 (47s) SKIP: replay-dual test_14b skipping ALWAYS excluded test 14b == replay-dual test 15a: timeout waiting for lost client during replay, 1 client completes ========================================================== 09:25:08 (1766413508) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre total: 25 open/close in 0.73 seconds: 34.21 ops/second total: 1 open/close in 0.03 seconds: 33.83 ops/second Failing mds1 on oleg324-server Stopping /mnt/lustre-mds1 (opts:) on oleg324-server 09:25:16 (1766413516) shut down facet: mds1 facet_host: oleg324-server facet_failover_host: oleg324-server Failover mds1 to oleg324-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg324-server: oleg324-server.virtnet: executing set_default_debug -1 all pdsh@oleg324-client: oleg324-server: ssh exited with exit code 1 Started lustre-MDT0000 09:25:38 (1766413538) targets are mounted 09:25:38 (1766413538) facet_failover done oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec - unlinked 0 (time 1766413609 ; total 0 ; last 0) total: 25 unlinks in 1 seconds: 25.000000 unlinks/second Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre2 PASS 15a (106s) == replay-dual test 15c: remove multiple OST orphans ===== 09:26:54 (1766413614) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg324-server Stopping /mnt/lustre-mds1 (opts:) on oleg324-server 09:28:31 (1766413711) shut down facet: mds1 facet_host: oleg324-server facet_failover_host: oleg324-server Failover mds1 to oleg324-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg324-server: oleg324-server.virtnet: executing set_default_debug -1 all pdsh@oleg324-client: oleg324-server: ssh exited with exit code 1 Started lustre-MDT0000 09:28:54 (1766413734) targets are mounted 09:28:54 (1766413734) facet_failover done oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre2 PASS 15c (196s) == replay-dual test 16: fail MDS during recovery (3571) == 09:30:10 (1766413810) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre total: 25 open/close in 0.75 seconds: 33.49 ops/second total: 1 open/close in 0.03 seconds: 39.95 ops/second Failing mds1 on oleg324-server Stopping /mnt/lustre-mds1 (opts:) on oleg324-server 09:30:20 (1766413820) shut down facet: mds1 facet_host: oleg324-server facet_failover_host: oleg324-server Failover mds1 to oleg324-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg324-server: oleg324-server.virtnet: executing set_default_debug -1 all pdsh@oleg324-client: oleg324-server: ssh exited with exit code 1 Started lustre-MDT0000 09:30:51 (1766413851) targets are mounted 09:30:51 (1766413851) facet_failover done Failing mds1 on oleg324-server Stopping /mnt/lustre-mds1 (opts:) on oleg324-server 09:31:15 (1766413875) shut down facet: mds1 facet_host: oleg324-server facet_failover_host: oleg324-server Failover mds1 to oleg324-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg324-server: oleg324-server.virtnet: executing set_default_debug -1 all pdsh@oleg324-client: oleg324-server: ssh exited with exit code 1 Started lustre-MDT0000 09:31:37 (1766413897) targets are mounted 09:31:37 (1766413897) facet_failover done oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec - unlinked 0 (time 1766413969 ; total 0 ; last 0) total: 25 unlinks in 1 seconds: 25.000000 unlinks/second Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre2 PASS 16 (164s) == replay-dual test 17: fail OST during recovery (3571) == 09:32:54 (1766413974) total: 25 open/close in 0.82 seconds: 30.37 ops/second total: 1 open/close in 0.04 seconds: 26.62 ops/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing ost1 on oleg324-server Stopping /mnt/lustre-ost1 (opts:) on oleg324-server 09:33:04 (1766413984) shut down facet: ost1 facet_host: oleg324-server facet_failover_host: oleg324-server Failover ost1 to oleg324-server mount facets: ost1 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg324-server: oleg324-server.virtnet: executing set_default_debug -1 all pdsh@oleg324-client: oleg324-server: ssh exited with exit code 1 Started lustre-OST0000 09:33:23 (1766414003) targets are mounted 09:33:23 (1766414003) facet_failover done Failing ost1 on oleg324-server Stopping /mnt/lustre-ost1 (opts:) on oleg324-server 09:33:48 (1766414028) shut down facet: ost1 facet_host: oleg324-server facet_failover_host: oleg324-server Failover ost1 to oleg324-server mount facets: ost1 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg324-server: oleg324-server.virtnet: executing set_default_debug -1 all pdsh@oleg324-client: oleg324-server: ssh exited with exit code 1 Started lustre-OST0000 09:34:11 (1766414051) targets are mounted 09:34:11 (1766414051) facet_failover done oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec - unlinked 0 (time 1766414122 ; total 0 ; last 0) total: 25 unlinks in 1 seconds: 25.000000 unlinks/second Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre2 PASS 17 (153s) == replay-dual test 18: ldlm_handle_enqueue succeeds on evicted export (3822) ========================================================== 09:35:27 (1766414127) debug=+dlmtrace fail_loc=0x8000030b using seed 591875234 running for 500 iterations total: 500 stats in 1 seconds: 500.000000 stats/second ldlm.namespaces.MGC192.168.203.124@tcp.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0000-mdc-ffff964dc9454000.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0000-mdc-ffff964dc9e7c000.early_lock_cancel=0 ldlm.namespaces.lustre-OST0000-osc-ffff964dc9454000.early_lock_cancel=0 ldlm.namespaces.lustre-OST0000-osc-ffff964dc9e7c000.early_lock_cancel=0 ldlm.namespaces.lustre-OST0001-osc-ffff964dc9454000.early_lock_cancel=0 ldlm.namespaces.lustre-OST0001-osc-ffff964dc9e7c000.early_lock_cancel=0 fail_loc=0x80000305 Error in opening file "/mnt/lustre2/d18.replay-dual/f18.replay-dual"(flags=O_RDONLY) 2: No such file or directory ldlm.namespaces.MGC192.168.203.124@tcp.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0000-mdc-ffff964dc9454000.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0000-mdc-ffff964dc9e7c000.early_lock_cancel=1 ldlm.namespaces.lustre-OST0000-osc-ffff964dc9454000.early_lock_cancel=1 ldlm.namespaces.lustre-OST0000-osc-ffff964dc9e7c000.early_lock_cancel=1 ldlm.namespaces.lustre-OST0001-osc-ffff964dc9454000.early_lock_cancel=1 ldlm.namespaces.lustre-OST0001-osc-ffff964dc9e7c000.early_lock_cancel=1 fail_loc=0 fail_loc=0 PASS 18 (55s) == replay-dual test 19: resend of open request =========== 09:36:22 (1766414182) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre fail_loc=0x157 - open/close 0 (time 1766414276.20 total 86.89 last 0.00) total: 1 open/close in 86.89 seconds: 0.01 ops/second fail_loc=0 Failing mds1 on oleg324-server Stopping /mnt/lustre-mds1 (opts:) on oleg324-server 09:38:01 (1766414281) shut down facet: mds1 facet_host: oleg324-server facet_failover_host: oleg324-server Failover mds1 to oleg324-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg324-server: oleg324-server.virtnet: executing set_default_debug -1 all pdsh@oleg324-client: oleg324-server: ssh exited with exit code 1 Started lustre-MDT0000 09:38:20 (1766414300) targets are mounted 09:38:20 (1766414300) facet_failover done oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 19 (128s) == replay-dual test 20: recovery time is not increasing == 09:38:30 (1766414310) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg324-server Stopping /mnt/lustre-mds1 (opts:) on oleg324-server 09:38:38 (1766414318) shut down facet: mds1 facet_host: oleg324-server facet_failover_host: oleg324-server Failover mds1 to oleg324-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg324-server: oleg324-server.virtnet: executing set_default_debug -1 all pdsh@oleg324-client: oleg324-server: ssh exited with exit code 1 Started lustre-MDT0000 09:38:57 (1766414337) targets are mounted 09:38:57 (1766414337) facet_failover done oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg324-server Stopping /mnt/lustre-mds1 (opts:) on oleg324-server 09:41:28 (1766414488) shut down facet: mds1 facet_host: oleg324-server facet_failover_host: oleg324-server Failover mds1 to oleg324-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg324-server: oleg324-server.virtnet: executing set_default_debug -1 all pdsh@oleg324-client: oleg324-server: ssh exited with exit code 1 Started lustre-MDT0000 09:41:48 (1766414508) targets are mounted 09:41:48 (1766414508) facet_failover done oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre2 PASS 20 (347s) == replay-dual test 21a: commit on sharing =============== 09:44:17 (1766414657) mdt.lustre-MDT0000.commit_on_sharing=1 Replay barrier on lustre-MDT0000 Failing mds1 on oleg324-server Stopping /mnt/lustre-mds1 (opts:) on oleg324-server 09:44:28 (1766414668) shut down facet: mds1 facet_host: oleg324-server facet_failover_host: oleg324-server Failover mds1 to oleg324-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg324-server: oleg324-server.virtnet: executing set_default_debug -1 all pdsh@oleg324-client: oleg324-server: ssh exited with exit code 1 Started lustre-MDT0000 09:45:00 (1766414700) targets are mounted 09:45:00 (1766414700) facet_failover done Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre2 mdt.lustre-MDT0000.commit_on_sharing=0 PASS 21a (185s) SKIP: replay-dual test_21b skipping SLOW test 21b == replay-dual test 22a: c1 lfs mkdir -i 1 dir1, M1 drop reply & fail, c2 mkdir dir1/dir ========================================================== 09:47:24 (1766414844) SKIP: replay-dual test_22a needs >= 2 MDTs SKIP 22a (2s) == replay-dual test 22b: c1 lfs mkdir -i 1 d1, M1 drop reply & fail M0/M1, c2 mkdir d1/dir ========================================================== 09:47:26 (1766414846) SKIP: replay-dual test_22b needs >= 2 MDTs SKIP 22b (3s) == replay-dual test 22c: c1 lfs mkdir -i 1 d1, M1 drop update & fail M1, c2 mkdir d1/dir ========================================================== 09:47:29 (1766414849) SKIP: replay-dual test_22c needs >= 2 MDTs SKIP 22c (2s) == replay-dual test 22d: c1 lfs mkdir -i 1 d1, M1 drop update & fail M0/M1,c2 mkdir d1/dir ========================================================== 09:47:32 (1766414852) SKIP: replay-dual test_22d needs >= 2 MDTs SKIP 22d (3s) == replay-dual test 23a: c1 rmdir d1, M1 drop reply and fail, client2 mkdir d1 ========================================================== 09:47:35 (1766414855) SKIP: replay-dual test_23a needs >= 2 MDTs SKIP 23a (3s) == replay-dual test 23b: c1 rmdir d1, M1 drop reply and fail M0/M1, c2 mkdir d1 ========================================================== 09:47:38 (1766414858) SKIP: replay-dual test_23b needs >= 2 MDTs SKIP 23b (2s) == replay-dual test 23c: c1 rmdir d1, M0 drop update reply and fail M0, c2 mkdir d1 ========================================================== 09:47:40 (1766414860) SKIP: replay-dual test_23c needs >= 2 MDTs SKIP 23c (3s) == replay-dual test 23d: c1 rmdir d1, M0 drop update reply and fail M0/M1, c2 mkdir d1 ========================================================== 09:47:43 (1766414863) SKIP: replay-dual test_23d needs >= 2 MDTs SKIP 23d (2s) == replay-dual test 24: reconstruct on non-existing object ========================================================== 09:47:45 (1766414865) fail_loc=0x119 fail_loc=0 truncate: cannot truncate '/mnt/lustre/f24.replay-dual' to length 100: No such file or directory PASS 24 (92s) == replay-dual test 25: replay|resend ==================== 09:49:17 (1766414957) 1+0 records in 1+0 records out 512 bytes copied, 0.0126107 s, 40.6 kB/s fail_loc=0x304 fail_loc=0x80000325 Failing ost1 on oleg324-server Stopping /mnt/lustre-ost1 (opts:) on oleg324-server 09:49:25 (1766414965) shut down facet: ost1 facet_host: oleg324-server facet_failover_host: oleg324-server Failover ost1 to oleg324-server mount facets: ost1 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg324-server: oleg324-server.virtnet: executing set_default_debug -1 all pdsh@oleg324-client: oleg324-server: ssh exited with exit code 1 Started lustre-OST0000 09:49:46 (1766414986) targets are mounted 09:49:46 (1766414986) facet_failover done oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 6803: 43202 Terminated LUSTRE="/home/green/git/lustre-release/lustre" bash -c "multiop /mnt/lustre2/f25.replay-dual Ow512" fail_loc=0 PASS 25 (42s) == replay-dual test 26: dbench and tar with mds failover ========================================================== 09:49:59 (1766414999) Starting client oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre Started clients oleg324-client.virtnet: 192.168.203.124@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) Started tar loop with pid 44598 Started dbench loop with 44599 looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3200 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3759104 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3729408 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7488512 1% /mnt/lustre '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Mon Dec 22 09:50:04 EST 2025 waiting for dbench pid 44655 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs failed to create barrier semaphore 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 32 1.05 MB/sec warmup 1 sec latency 142.698 ms 1 100 2.29 MB/sec warmup 2 sec latency 131.762 ms 1 147 2.09 MB/sec warmup 3 sec latency 187.371 ms 1 190 1.89 MB/sec warmup 4 sec latency 115.435 ms test_26 fail mds1 1 times 1 255 1.93 MB/sec warmup 5 sec latency 69.838 ms 1 305 1.87 MB/sec warmup 6 sec latency 92.132 ms Failing mds1 on oleg324-server 1 356 1.82 MB/sec warmup 7 sec latency 113.949 ms Stopping /mnt/lustre-mds1 (opts:) on oleg324-server 1 402 1.77 MB/sec warmup 8 sec latency 115.809 ms 1 404 1.59 MB/sec warmup 9 sec latency 989.858 ms 1 404 1.43 MB/sec warmup 10 sec latency 1990.109 ms 09:50:15 (1766415015) shut down facet: mds1 facet_host: oleg324-server facet_failover_host: oleg324-server 1 404 1.30 MB/sec warmup 11 sec latency 2990.299 ms 1 404 1.19 MB/sec warmup 12 sec latency 3990.502 ms 1 404 1.10 MB/sec warmup 13 sec latency 4991.726 ms 1 404 1.02 MB/sec warmup 14 sec latency 5992.924 ms 1 404 0.95 MB/sec warmup 15 sec latency 6993.107 ms 1 404 0.90 MB/sec warmup 16 sec latency 7993.323 ms 1 404 0.84 MB/sec warmup 17 sec latency 8993.553 ms 1 404 0.80 MB/sec warmup 18 sec latency 9993.774 ms 1 404 0.75 MB/sec warmup 19 sec latency 10993.980 ms Failover mds1 to oleg324-server mount facets: mds1 1 404 0.00 MB/sec execute 1 sec latency 12995.107 ms 1 404 0.00 MB/sec execute 2 sec latency 13995.304 ms 1 404 0.00 MB/sec execute 3 sec latency 14997.756 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 404 0.00 MB/sec execute 4 sec latency 15999.747 ms 1 404 0.00 MB/sec execute 5 sec latency 16999.933 ms 1 404 0.00 MB/sec execute 6 sec latency 18001.607 ms 1 404 0.00 MB/sec execute 7 sec latency 19009.752 ms 1 404 0.00 MB/sec execute 8 sec latency 20022.796 ms 1 404 0.00 MB/sec execute 9 sec latency 21022.971 ms oleg324-server: oleg324-server.virtnet: executing set_default_debug -1 all 1 404 0.00 MB/sec execute 10 sec latency 22024.748 ms pdsh@oleg324-client: oleg324-server: ssh exited with exit code 1 Started lustre-MDT0000 09:50:35 (1766415035) targets are mounted 09:50:35 (1766415035) facet_failover done 1 404 0.00 MB/sec execute 11 sec latency 23024.988 ms 1 404 0.00 MB/sec execute 12 sec latency 24025.894 ms 1 404 0.00 MB/sec execute 13 sec latency 25027.191 ms 1 404 0.00 MB/sec execute 14 sec latency 26027.388 ms 1 404 0.00 MB/sec execute 15 sec latency 27030.904 ms 1 404 0.00 MB/sec execute 16 sec latency 28032.397 ms 1 404 0.00 MB/sec execute 17 sec latency 29032.601 ms 1 404 0.00 MB/sec execute 18 sec latency 30032.795 ms 1 404 0.00 MB/sec execute 19 sec latency 31032.990 ms 1 404 0.00 MB/sec execute 20 sec latency 32033.548 ms 1 404 0.00 MB/sec execute 21 sec latency 33033.779 ms 1 404 0.00 MB/sec execute 22 sec latency 34034.813 ms 1 404 0.00 MB/sec execute 23 sec latency 35035.010 ms 1 404 0.00 MB/sec execute 24 sec latency 36035.954 ms 1 404 0.00 MB/sec execute 25 sec latency 37036.611 ms 1 404 0.00 MB/sec execute 26 sec latency 38036.823 ms 1 413 0.01 MB/sec execute 27 sec latency 38637.286 ms 1 467 0.07 MB/sec execute 28 sec latency 105.140 ms 1 511 0.12 MB/sec execute 29 sec latency 103.695 ms oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 593 0.21 MB/sec execute 30 sec latency 114.978 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 648 0.22 MB/sec execute 31 sec latency 115.627 ms 1 676 0.21 MB/sec execute 32 sec latency 290.848 ms 1 721 0.21 MB/sec execute 33 sec latency 144.518 ms 1 759 0.21 MB/sec execute 34 sec latency 242.651 ms 1 801 0.20 MB/sec execute 35 sec latency 144.256 ms 1 870 0.20 MB/sec execute 36 sec latency 83.371 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3584 2204928 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 13312 3733504 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 39936 3683328 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 53248 7416832 1% /mnt/lustre 1 903 0.20 MB/sec execute 37 sec latency 180.708 ms 1 939 0.20 MB/sec execute 38 sec latency 191.063 ms 1 980 0.19 MB/sec execute 39 sec latency 101.147 ms 1 1037 0.21 MB/sec execute 40 sec latency 124.161 ms 1 1084 0.21 MB/sec execute 41 sec latency 74.248 ms test_26 fail mds1 2 times 1 1134 0.21 MB/sec execute 42 sec latency 95.216 ms 1 1179 0.20 MB/sec execute 43 sec latency 121.149 ms Failing mds1 on oleg324-server 1 1224 0.20 MB/sec execute 44 sec latency 80.409 ms Stopping /mnt/lustre-mds1 (opts:) on oleg324-server 1 1267 0.20 MB/sec execute 45 sec latency 93.200 ms 1 1267 0.19 MB/sec execute 46 sec latency 1098.656 ms 09:51:12 (1766415072) shut down 1 1267 0.19 MB/sec execute 47 sec latency 2106.199 ms facet: mds1 facet_host: oleg324-server facet_failover_host: oleg324-server 1 1267 0.18 MB/sec execute 48 sec latency 3106.384 ms 1 1267 0.18 MB/sec execute 49 sec latency 4106.594 ms 1 1267 0.18 MB/sec execute 50 sec latency 5106.813 ms 1 1267 0.17 MB/sec execute 51 sec latency 6107.081 ms 1 1267 0.17 MB/sec execute 52 sec latency 7108.185 ms 1 1267 0.17 MB/sec execute 53 sec latency 8110.602 ms 1 1267 0.16 MB/sec execute 54 sec latency 9110.826 ms 1 1267 0.16 MB/sec execute 55 sec latency 10113.935 ms 1 1267 0.16 MB/sec execute 56 sec latency 11114.182 ms 1 1267 0.15 MB/sec execute 57 sec latency 12114.449 ms Failover mds1 to oleg324-server mount facets: mds1 1 1267 0.15 MB/sec execute 58 sec latency 13114.732 ms 1 1267 0.15 MB/sec execute 59 sec latency 14115.865 ms 1 1267 0.15 MB/sec execute 60 sec latency 15116.160 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 1267 0.14 MB/sec execute 61 sec latency 16116.375 ms 1 1267 0.14 MB/sec execute 62 sec latency 17116.781 ms 1 1267 0.14 MB/sec execute 63 sec latency 18124.067 ms 1 1267 0.14 MB/sec execute 64 sec latency 19124.242 ms 1 1267 0.14 MB/sec execute 65 sec latency 20125.713 ms oleg324-server: oleg324-server.virtnet: executing set_default_debug -1 all 1 1267 0.13 MB/sec execute 66 sec latency 21129.876 ms pdsh@oleg324-client: oleg324-server: ssh exited with exit code 1 1 1267 0.13 MB/sec execute 67 sec latency 22130.087 ms Started lustre-MDT0000 09:51:32 (1766415092) targets are mounted 09:51:32 (1766415092) facet_failover done 1 1285 0.13 MB/sec execute 68 sec latency 22690.408 ms 1 1420 0.17 MB/sec execute 69 sec latency 80.285 ms 1 1481 0.17 MB/sec execute 70 sec latency 88.391 ms 1 1512 0.16 MB/sec execute 71 sec latency 253.168 ms oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 1564 0.16 MB/sec execute 72 sec latency 85.291 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 1609 0.16 MB/sec execute 73 sec latency 95.525 ms 1 1653 0.16 MB/sec execute 74 sec latency 240.277 ms 1 1696 0.16 MB/sec execute 75 sec latency 133.160 ms 1 1770 0.16 MB/sec execute 76 sec latency 85.658 ms 1 1845 0.16 MB/sec execute 77 sec latency 85.099 ms 1 1905 0.15 MB/sec execute 78 sec latency 155.955 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4096 2204416 1% /mnt/lustre[MDT:0] 1 1986 0.15 MB/sec execute 79 sec latency 69.667 ms lustre-OST0000_UUID 3771392 24576 3719168 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 50176 3708928 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 74752 7428096 1% /mnt/lustre 1 2042 0.15 MB/sec execute 80 sec latency 100.160 ms 1 2129 0.15 MB/sec execute 81 sec latency 82.873 ms 1 2166 0.15 MB/sec execute 82 sec latency 98.341 ms 1 2235 0.15 MB/sec execute 83 sec latency 92.687 ms 1 2345 0.16 MB/sec execute 84 sec latency 98.186 ms test_26 fail mds1 3 times 1 2393 0.16 MB/sec execute 85 sec latency 133.018 ms Failing mds1 on oleg324-server 1 2426 0.16 MB/sec execute 86 sec latency 208.007 ms Stopping /mnt/lustre-mds1 (opts:) on oleg324-server 1 2477 0.16 MB/sec execute 87 sec latency 180.716 ms 1 2478 0.16 MB/sec execute 88 sec latency 907.777 ms 1 2478 0.16 MB/sec execute 89 sec latency 1911.150 ms 09:51:54 (1766415114) shut down facet: mds1 facet_host: oleg324-server facet_failover_host: oleg324-server 1 2478 0.16 MB/sec execute 90 sec latency 2911.806 ms 1 2478 0.16 MB/sec execute 91 sec latency 3915.263 ms 1 2478 0.16 MB/sec execute 92 sec latency 4915.471 ms 1 2478 0.15 MB/sec execute 93 sec latency 5917.593 ms 1 2478 0.15 MB/sec execute 94 sec latency 6918.013 ms 1 2478 0.15 MB/sec execute 95 sec latency 7920.553 ms 1 2478 0.15 MB/sec execute 96 sec latency 8921.241 ms 1 2478 0.15 MB/sec execute 97 sec latency 9921.487 ms 1 2478 0.15 MB/sec execute 98 sec latency 10921.964 ms 1 2478 0.14 MB/sec execute 99 sec latency 11923.616 ms Failover mds1 to oleg324-server mount facets: mds1 1 cleanup 100 sec 1 cleanup 101 sec 1 cleanup 102 sec Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 cleanup 103 sec 1 cleanup 104 sec 1 cleanup 105 sec 1 cleanup 106 sec 1 cleanup 107 sec oleg324-server: oleg324-server.virtnet: executing set_default_debug -1 all 1 cleanup 108 sec pdsh@oleg324-client: oleg324-server: ssh exited with exit code 1 1 cleanup 109 sec Started lustre-MDT0000 09:52:14 (1766415134) targets are mounted 09:52:14 (1766415134) facet_failover done 1 cleanup 110 sec 1 cleanup 111 sec 0 cleanup 112 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 359 47.050 255.857 Close 309 209.126 38637.254 Rename 14 74.492 121.122 Unlink 36 21.125 40.951 Qpathinfo 325 13.691 95.201 Qfileinfo 92 1.934 20.927 Qfsinfo 41 4.496 32.505 Sfileinfo 46 531.798 22262.547 Find 110 4.535 52.542 WriteX 326 9.785 73.003 ReadX 383 2.351 75.768 Flush 34 124.671 290.831 Throughput 0.14488 MB/sec 1 clients 1 procs max_latency=38637.286 ms stopping dbench on /mnt/lustre at Mon Dec 22 09:52:17 EST 2025 with return code 0 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Mon Dec 22 09:52:22 EST 2025 waiting for dbench pid 47506 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 44 1.69 MB/sec warmup 1 sec latency 104.910 ms 1 111 2.61 MB/sec warmup 2 sec latency 107.077 ms 1 155 2.17 MB/sec warmup 3 sec latency 144.991 ms 1 199 1.96 MB/sec warmup 4 sec latency 132.190 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4608 2203904 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 17408 3698688 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 40960 3678208 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 58368 7376896 1% /mnt/lustre 1 255 1.93 MB/sec warmup 5 sec latency 97.648 ms 1 302 1.84 MB/sec warmup 6 sec latency 104.896 ms 1 349 1.78 MB/sec warmup 7 sec latency 112.466 ms 1 417 1.83 MB/sec warmup 8 sec latency 80.491 ms 1 493 1.91 MB/sec warmup 9 sec latency 65.854 ms test_26 fail mds1 4 times 1 567 1.91 MB/sec warmup 10 sec latency 88.347 ms Failing mds1 on oleg324-server 1 655 1.91 MB/sec warmup 11 sec latency 71.939 ms Stopping /mnt/lustre-mds1 (opts:) on oleg324-server 1 695 1.76 MB/sec warmup 12 sec latency 129.360 ms 1 700 1.63 MB/sec warmup 13 sec latency 844.693 ms 1 700 1.51 MB/sec warmup 14 sec latency 1847.578 ms 09:52:36 (1766415156) shut down facet: mds1 facet_host: oleg324-server facet_failover_host: oleg324-server 1 700 1.41 MB/sec warmup 15 sec latency 2847.825 ms 1 700 1.32 MB/sec warmup 16 sec latency 3848.049 ms 1 700 1.25 MB/sec warmup 17 sec latency 4848.246 ms 1 700 1.18 MB/sec warmup 18 sec latency 5848.484 ms 1 700 1.11 MB/sec warmup 19 sec latency 6849.589 ms 1 700 0.00 MB/sec execute 1 sec latency 8850.101 ms 1 700 0.00 MB/sec execute 2 sec latency 9850.357 ms 1 700 0.00 MB/sec execute 3 sec latency 10850.564 ms 1 700 0.00 MB/sec execute 4 sec latency 11850.844 ms Failover mds1 to oleg324-server mount facets: mds1 1 700 0.00 MB/sec execute 5 sec latency 12851.136 ms 1 700 0.00 MB/sec execute 6 sec latency 13851.370 ms 1 700 0.00 MB/sec execute 7 sec latency 14851.573 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 700 0.00 MB/sec execute 8 sec latency 15851.773 ms 1 700 0.00 MB/sec execute 9 sec latency 16851.906 ms 1 700 0.00 MB/sec execute 10 sec latency 17852.101 ms 1 700 0.00 MB/sec execute 11 sec latency 18852.763 ms oleg324-server: oleg324-server.virtnet: executing set_default_debug -1 all 1 700 0.00 MB/sec execute 12 sec latency 19860.443 ms pdsh@oleg324-client: oleg324-server: ssh exited with exit code 1 1 700 0.00 MB/sec execute 13 sec latency 20863.622 ms Started lustre-MDT0000 09:52:55 (1766415175) targets are mounted 09:52:55 (1766415175) facet_failover done 1 700 0.00 MB/sec execute 14 sec latency 21864.038 ms 1 706 0.00 MB/sec execute 15 sec latency 22612.914 ms 1 738 0.01 MB/sec execute 16 sec latency 175.313 ms oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 791 0.02 MB/sec execute 17 sec latency 173.724 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 848 0.02 MB/sec execute 18 sec latency 110.638 ms 1 892 0.03 MB/sec execute 19 sec latency 125.877 ms 1 931 0.03 MB/sec execute 20 sec latency 178.062 ms 1 980 0.03 MB/sec execute 21 sec latency 105.247 ms 1 1011 0.05 MB/sec execute 22 sec latency 142.964 ms 1 1082 0.08 MB/sec execute 23 sec latency 206.001 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 5376 2203136 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 26624 3692544 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 50176 3688448 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 76800 7380992 2% /mnt/lustre 1 1136 0.07 MB/sec execute 24 sec latency 117.385 ms 1 1190 0.07 MB/sec execute 25 sec latency 101.773 ms 1 1244 0.07 MB/sec execute 26 sec latency 68.497 ms 1 1389 0.18 MB/sec execute 27 sec latency 72.220 ms 1 1455 0.17 MB/sec execute 28 sec latency 71.269 ms test_26 fail mds1 5 times 1 1504 0.17 MB/sec execute 29 sec latency 210.177 ms Failing mds1 on oleg324-server 1 1551 0.16 MB/sec execute 30 sec latency 155.709 ms Stopping /mnt/lustre-mds1 (opts:) on oleg324-server 1 1591 0.16 MB/sec execute 31 sec latency 58.864 ms 1 1616 0.15 MB/sec execute 32 sec latency 613.070 ms 1 1616 0.15 MB/sec execute 33 sec latency 1614.081 ms 09:53:15 (1766415195) shut down facet: mds1 facet_host: oleg324-server facet_failover_host: oleg324-server 1 1616 0.15 MB/sec execute 34 sec latency 2614.316 ms 1 1616 0.14 MB/sec execute 35 sec latency 3615.924 ms 1 1616 0.14 MB/sec execute 36 sec latency 4616.125 ms 1 1616 0.13 MB/sec execute 37 sec latency 5616.323 ms 1 1616 0.13 MB/sec execute 38 sec latency 6616.550 ms 1 1616 0.13 MB/sec execute 39 sec latency 7616.780 ms 1 1616 0.12 MB/sec execute 40 sec latency 8617.046 ms 1 1616 0.12 MB/sec execute 41 sec latency 9617.282 ms 1 1616 0.12 MB/sec execute 42 sec latency 10617.767 ms 1 1616 0.12 MB/sec execute 43 sec latency 11621.580 ms Failover mds1 to oleg324-server mount facets: mds1 1 1616 0.11 MB/sec execute 44 sec latency 12621.884 ms 1 1616 0.11 MB/sec execute 45 sec latency 13623.461 ms 1 1616 0.11 MB/sec execute 46 sec latency 14624.572 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 1616 0.11 MB/sec execute 47 sec latency 15624.784 ms 1 1616 0.10 MB/sec execute 48 sec latency 16627.182 ms 1 1616 0.10 MB/sec execute 49 sec latency 17627.370 ms 1 1616 0.10 MB/sec execute 50 sec latency 18627.838 ms oleg324-server: oleg324-server.virtnet: executing set_default_debug -1 all 1 1616 0.10 MB/sec execute 51 sec latency 19628.397 ms pdsh@oleg324-client: oleg324-server: ssh exited with exit code 1 1 1616 0.10 MB/sec execute 52 sec latency 20630.926 ms Started lustre-MDT0000 09:53:34 (1766415214) targets are mounted 09:53:34 (1766415214) facet_failover done 1 1618 0.09 MB/sec execute 53 sec latency 21479.273 ms 1 1660 0.09 MB/sec execute 54 sec latency 189.385 ms 1 1699 0.09 MB/sec execute 55 sec latency 131.513 ms 1 1771 0.09 MB/sec execute 56 sec latency 66.360 ms oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 1837 0.09 MB/sec execute 57 sec latency 82.893 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 1884 0.09 MB/sec execute 58 sec latency 133.079 ms 1 1954 0.09 MB/sec execute 59 sec latency 281.126 ms 1 2004 0.09 MB/sec execute 60 sec latency 90.693 ms 1 2062 0.09 MB/sec execute 61 sec latency 85.324 ms 1 2138 0.09 MB/sec execute 62 sec latency 73.738 ms 1 2185 0.09 MB/sec execute 63 sec latency 79.624 ms 1 2280 0.09 MB/sec execute 64 sec latency 125.402 ms 1 2358 0.10 MB/sec execute 65 sec latency 199.044 ms 1 2406 0.10 MB/sec execute 66 sec latency 209.634 ms 1 2460 0.11 MB/sec execute 67 sec latency 139.703 ms 1 2554 0.11 MB/sec execute 68 sec latency 209.585 ms 1 2567 0.11 MB/sec execute 69 sec latency 869.506 ms 1 2622 0.11 MB/sec execute 70 sec latency 116.273 ms 1 2779 0.15 MB/sec execute 71 sec latency 203.297 ms 1 2922 0.17 MB/sec execute 72 sec latency 196.260 ms 1 2987 0.17 MB/sec execute 73 sec latency 276.356 ms 1 3064 0.17 MB/sec execute 74 sec latency 230.249 ms 1 3212 0.17 MB/sec execute 75 sec latency 215.801 ms dbench killed by signal 15 stopping dbench on /mnt/lustre at Mon Dec 22 09:53:57 EST 2025 with return code 0 47506 pts/0 S+ 0:00 dbench -c client.txt 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100 killed dbench main pid 47506 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished PASS 26 (276s) == replay-dual test 28: lock replay should be ordered: waiting after granted ========================================================== 09:54:35 (1766415275) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00790771 s, 518 kB/s fail_loc=0x80000324 fail_loc=0x32a Failing ost1 on oleg324-server Stopping /mnt/lustre-ost1 (opts:) on oleg324-server 09:55:03 (1766415303) shut down facet: ost1 facet_host: oleg324-server facet_failover_host: oleg324-server Failover ost1 to oleg324-server mount facets: ost1 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00785798 s, 521 kB/s oleg324-server: oleg324-server.virtnet: executing set_default_debug -1 all pdsh@oleg324-client: oleg324-server: ssh exited with exit code 1 Started lustre-OST0000 09:55:24 (1766415324) targets are mounted 09:55:24 (1766415324) facet_failover done oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec PASS 28 (63s) == replay-dual test 29: replay vs update with the same xid ========================================================== 09:55:38 (1766415338) SKIP: replay-dual test_29 needs >= 2 MDTs SKIP 29 (3s) == replay-dual test 30: layout lock replay is not blocked on IO ========================================================== 09:55:41 (1766415341) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.025769 s, 1.6 MB/s 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.0203656 s, 2.0 MB/s fail_loc=0x32e fail_val=4 Failing mds1 on oleg324-server Stopping /mnt/lustre-mds1 (opts:) on oleg324-server 09:55:47 (1766415347) shut down facet: mds1 facet_host: oleg324-server facet_failover_host: oleg324-server Failover mds1 to oleg324-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg324-server: oleg324-server.virtnet: executing set_default_debug -1 all pdsh@oleg324-client: oleg324-server: ssh exited with exit code 1 Started lustre-MDT0000 09:56:05 (1766415365) targets are mounted 09:56:05 (1766415365) facet_failover done 160+0 records in 160+0 records out 81920 bytes (82 kB, 80 KiB) copied, 23.8007 s, 3.4 kB/s oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 30 (34s) == replay-dual test 31: deadlock on file_remove_privs and occupied mod rpc slots ========================================================== 09:56:15 (1766415375) Failing ost1 on oleg324-server Stopping /mnt/lustre-ost1 (opts:) on oleg324-server Creating to objid 2337 on ost lustre-OST0000... 09:56:21 (1766415381) shut down facet: ost1 facet_host: oleg324-server facet_failover_host: oleg324-server total: 32 open/close in 0.71 seconds: 44.99 ops/second at_max=0 fail_loc=0x80001420 file /mnt/lustre2/d31.replay-dual/mdtdir/f31.replay-dual is ready Failover ost1 to oleg324-server mount facets: ost1 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg324-server: oleg324-server.virtnet: executing set_default_debug -1 all pdsh@oleg324-client: oleg324-server: ssh exited with exit code 1 Started lustre-OST0000 09:56:41 (1766415401) targets are mounted 09:56:41 (1766415401) facet_failover done oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE FULL state after 0 sec pids: 53651 53652 53657 53658 53659 53660 53661 53662 53663 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3584 2204928 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 496504 393 496111 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 118251 491 117760 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 118153 393 117760 1% /mnt/lustre[OST:1] filesystem_summary: 235913 393 235520 1% /mnt/lustre at_max=600 PASS 31 (36s) == replay-dual test 32: gap in update llog shouldn't break recovery ========================================================== 09:56:51 (1766415411) SKIP: replay-dual test_32 needs >= 2 MDTs SKIP 32 (2s) == replay-dual test 33: Check for OBD_INCOMPAT_MULTI_RPCS in last_rcvd after abort_recovery ========================================================== 09:56:53 (1766415413) SKIP: replay-dual test_33 ldiskfs only test SKIP 33 (2s) == replay-dual test complete, duration 2785 sec ========== 09:56:55 (1766415415) === replay-dual: start cleanup 09:56:56 (1766415416) === Stopping clients: oleg324-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg324-client.virtnet /mnt/lustre2 opts: === replay-dual: finish cleanup 09:57:01 (1766415421) ===