-----============= acceptance-small: replay-dual ============----- Tue Jun 23 10:24:06 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg234-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/replay-dual.*ex': No such file or directory excepting tests: 14b 21b 21b skipping tests SLOW=no: 21b === replay-dual: start setup 10:24:26 (1782224666) === Starting client oleg234-client.virtnet: -o user_xattr,flock 192.168.202.134@tcp:/lustre /mnt/lustre2 Mount client oleg234-client.virtnet: mount -t lustre -o user_xattr,flock 192.168.202.134@tcp:/lustre /mnt/lustre2 Started clients oleg234-client.virtnet: 192.168.202.134@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) oleg234-client.virtnet: executing check_config_client /mnt/lustre oleg234-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg234-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9ee902a3f800.idle_timeout=debug osc.lustre-OST0000-osc-ffff9ee910333800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9ee902a3f800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9ee910333800.idle_timeout=debug disable quota as required oleg234-server: oleg234-server.virtnet: executing set_default_debug -1 all === replay-dual: finish setup 10:25:04 (1782224704) === == replay-dual test 0a: expired recovery with lost client ========================================================== 10:25:06 (1782224706) Check file is LU482_FAILED=/tmp/replay-dual.lu482.ALaLe6 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre total: 50 open/close in 2.45 seconds: 20.43 ops/second fail_loc=0x80000514 Failing mds1 on oleg234-server Stopping /mnt/lustre-mds1 (opts:) on oleg234-server 10:25:21 (1782224721) shut down facet: mds1 facet_host: oleg234-server facet_failover_host: oleg234-server Failover mds1 to oleg234-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg234-server: oleg234-server.virtnet: executing set_default_debug -1 all pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Started lustre-MDT0000 10:25:52 (1782224752) targets are mounted 10:25:52 (1782224752) facet_failover done Starting client: oleg234-client.virtnet: -o user_xattr,flock 192.168.202.134@tcp:/lustre /mnt/lustre2 - unlinked 0 (time 1782224876 ; total 0 ; last 0) total: 50 unlinks in 2 seconds: 25.000000 unlinks/second PASS 0a (179s) == replay-dual test 0b: lost client during waiting for next transno ========================================================== 10:28:05 (1782224885) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre Failing mds1 on oleg234-server Stopping /mnt/lustre-mds1 (opts:) on oleg234-server 10:28:16 (1782224896) shut down facet: mds1 facet_host: oleg234-server facet_failover_host: oleg234-server Failover mds1 to oleg234-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg234-server: oleg234-server.virtnet: executing set_default_debug -1 all pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Started lustre-MDT0000 10:28:40 (1782224920) targets are mounted 10:28:40 (1782224920) facet_failover done Starting client: oleg234-client.virtnet: -o user_xattr,flock 192.168.202.134@tcp:/lustre /mnt/lustre Starting client: oleg234-client.virtnet: -o user_xattr,flock 192.168.202.134@tcp:/lustre /mnt/lustre2 PASS 0b (211s) == replay-dual test 1: |X| simple create ================= 10:31:36 (1782225096) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre Failing mds1 on oleg234-server Stopping /mnt/lustre-mds1 (opts:) on oleg234-server 10:31:48 (1782225108) shut down facet: mds1 facet_host: oleg234-server facet_failover_host: oleg234-server Failover mds1 to oleg234-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg234-server: oleg234-server.virtnet: executing set_default_debug -1 all pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Started lustre-MDT0000 10:32:21 (1782225141) targets are mounted 10:32:21 (1782225141) facet_failover done oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 1 (66s) == replay-dual test 2: |X| mkdir adir ==================== 10:32:42 (1782225162) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre Failing mds1 on oleg234-server Stopping /mnt/lustre-mds1 (opts:) on oleg234-server 10:32:55 (1782225175) shut down facet: mds1 facet_host: oleg234-server facet_failover_host: oleg234-server Failover mds1 to oleg234-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg234-server: oleg234-server.virtnet: executing set_default_debug -1 all pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Started lustre-MDT0000 10:33:29 (1782225209) targets are mounted 10:33:29 (1782225209) facet_failover done oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 2 (64s) == replay-dual test 3: |X| mkdir adir, mkdir adir/bdir === 10:33:46 (1782225226) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre Failing mds1 on oleg234-server Stopping /mnt/lustre-mds1 (opts:) on oleg234-server 10:33:59 (1782225239) shut down facet: mds1 facet_host: oleg234-server facet_failover_host: oleg234-server Failover mds1 to oleg234-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg234-server: oleg234-server.virtnet: executing set_default_debug -1 all pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Started lustre-MDT0000 10:34:25 (1782225265) targets are mounted 10:34:25 (1782225265) facet_failover done oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 3 (57s) == replay-dual test 4: |X| mkdir adir (-EEXIST), mkdir adir/bdir ========================================================== 10:34:43 (1782225283) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre mkdir: cannot create directory '/mnt/lustre/adir': File exists Failing mds1 on oleg234-server Stopping /mnt/lustre-mds1 (opts:) on oleg234-server 10:34:53 (1782225293) shut down facet: mds1 facet_host: oleg234-server facet_failover_host: oleg234-server Failover mds1 to oleg234-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg234-server: oleg234-server.virtnet: executing set_default_debug -1 all pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Started lustre-MDT0000 10:35:18 (1782225318) targets are mounted 10:35:18 (1782225318) facet_failover done oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 4 (54s) == replay-dual test 5: open, unlink |X| close ============ 10:35:37 (1782225337) multiop /mnt/lustre2/a vo_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7635 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre Failing mds1 on oleg234-server Stopping /mnt/lustre-mds1 (opts:) on oleg234-server 10:35:47 (1782225347) shut down facet: mds1 facet_host: oleg234-server facet_failover_host: oleg234-server Failover mds1 to oleg234-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg234-server: oleg234-server.virtnet: executing set_default_debug -1 all pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Started lustre-MDT0000 10:36:12 (1782225372) targets are mounted 10:36:12 (1782225372) facet_failover done oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 5 (51s) == replay-dual test 6: open1, open2, unlink |X| close1 [fail mds1] close2 ========================================================== 10:36:28 (1782225388) multiop /mnt/lustre2/a vo_c TMPPIPE=/tmp/multiop_open_wait_pipe.7635 multiop /mnt/lustre/a vo_c TMPPIPE=/tmp/multiop_open_wait_pipe.7635 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre Failing mds1 on oleg234-server Stopping /mnt/lustre-mds1 (opts:) on oleg234-server 10:36:38 (1782225398) shut down facet: mds1 facet_host: oleg234-server facet_failover_host: oleg234-server Failover mds1 to oleg234-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg234-server: oleg234-server.virtnet: executing set_default_debug -1 all pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Started lustre-MDT0000 10:37:00 (1782225420) targets are mounted 10:37:00 (1782225420) facet_failover done oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 6 (48s) == replay-dual test 8: replay of resent request ========== 10:37:16 (1782225436) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre fail_loc=0x119 fail_loc=0 Failing mds1 on oleg234-server Stopping /mnt/lustre-mds1 (opts:) on oleg234-server 10:37:46 (1782225466) shut down facet: mds1 facet_host: oleg234-server facet_failover_host: oleg234-server Failover mds1 to oleg234-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg234-server: oleg234-server.virtnet: executing set_default_debug -1 all pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Started lustre-MDT0000 10:38:18 (1782225498) targets are mounted 10:38:18 (1782225498) facet_failover done oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 8 (79s) == replay-dual test 9: resending a replayed create ======= 10:38:35 (1782225515) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre fail_loc=0x80000119 Failing mds1 on oleg234-server Stopping /mnt/lustre-mds1 (opts:) on oleg234-server 10:38:47 (1782225527) shut down facet: mds1 facet_host: oleg234-server facet_failover_host: oleg234-server Failover mds1 to oleg234-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg234-server: oleg234-server.virtnet: executing set_default_debug -1 all pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Started lustre-MDT0000 10:39:19 (1782225559) targets are mounted 10:39:19 (1782225559) facet_failover done oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec fail_loc=0 PASS 9 (71s) == replay-dual test 10: resending a replayed unlink ====== 10:39:46 (1782225586) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre fail_loc=0x80000119 Failing mds1 on oleg234-server Stopping /mnt/lustre-mds1 (opts:) on oleg234-server 10:39:59 (1782225599) shut down facet: mds1 facet_host: oleg234-server facet_failover_host: oleg234-server Failover mds1 to oleg234-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg234-server: oleg234-server.virtnet: executing set_default_debug -1 all pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Started lustre-MDT0000 10:40:31 (1782225631) targets are mounted 10:40:31 (1782225631) facet_failover done oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec fail_loc=0 PASS 10 (72s) == replay-dual test 11: both clients timeout during replay ========================================================== 10:40:58 (1782225658) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre fail_loc=0x0119 Failing mds1 on oleg234-server Stopping /mnt/lustre-mds1 (opts:) on oleg234-server 10:41:10 (1782225670) shut down facet: mds1 facet_host: oleg234-server facet_failover_host: oleg234-server Failover mds1 to oleg234-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg234-server: oleg234-server.virtnet: executing set_default_debug -1 all pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Started lustre-MDT0000 10:41:43 (1782225703) targets are mounted 10:41:43 (1782225703) facet_failover done oleg234-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec fail_loc=0 PASS 11 (64s) == replay-dual test 12: open resend timeout ============== 10:42:02 (1782225722) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre multiop /mnt/lustre/f12.replay-dual vmo_c TMPPIPE=/tmp/multiop_open_wait_pipe.7635 fail_loc=0x80000302 Failing mds1 on oleg234-server Stopping /mnt/lustre-mds1 (opts:) on oleg234-server 10:42:15 (1782225735) shut down facet: mds1 facet_host: oleg234-server facet_failover_host: oleg234-server Failover mds1 to oleg234-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg234-server: oleg234-server.virtnet: executing set_default_debug -1 all pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Started lustre-MDT0000 10:42:38 (1782225758) targets are mounted 10:42:38 (1782225758) facet_failover done fail_loc=0 /mnt/lustre/f12.replay-dual /mnt/lustre/f12.replay-dual has type file OK PASS 12 (60s) == replay-dual test 13: close resend timeout ============= 10:43:02 (1782225782) multiop /mnt/lustre/f13.replay-dual vmo_c TMPPIPE=/tmp/multiop_open_wait_pipe.7635 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre fail_loc=0x80000115 Failing mds1 on oleg234-server Stopping /mnt/lustre-mds1 (opts:) on oleg234-server 10:43:13 (1782225793) shut down facet: mds1 facet_host: oleg234-server facet_failover_host: oleg234-server Failover mds1 to oleg234-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg234-server: oleg234-server.virtnet: executing set_default_debug -1 all pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Started lustre-MDT0000 10:43:47 (1782225827) targets are mounted 10:43:47 (1782225827) facet_failover done fail_loc=0 /mnt/lustre/f13.replay-dual /mnt/lustre/f13.replay-dual has type file OK PASS 13 (62s) SKIP: replay-dual test_14b skipping ALWAYS excluded test 14b == replay-dual test 15a: timeout waiting for lost client during replay, 1 client completes ========================================================== 10:44:06 (1782225846) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre total: 25 open/close in 1.45 seconds: 17.29 ops/second total: 1 open/close in 0.05 seconds: 21.55 ops/second Failing mds1 on oleg234-server Stopping /mnt/lustre-mds1 (opts:) on oleg234-server 10:44:19 (1782225859) shut down facet: mds1 facet_host: oleg234-server facet_failover_host: oleg234-server Failover mds1 to oleg234-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg234-server: oleg234-server.virtnet: executing set_default_debug -1 all pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Started lustre-MDT0000 10:44:43 (1782225883) targets are mounted 10:44:43 (1782225883) facet_failover done oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec - unlinked 0 (time 1782225966 ; total 0 ; last 0) total: 25 unlinks in 1 seconds: 25.000000 unlinks/second Starting client: oleg234-client.virtnet: -o user_xattr,flock 192.168.202.134@tcp:/lustre /mnt/lustre2 PASS 15a (128s) == replay-dual test 15c: remove multiple OST orphans ===== 10:46:13 (1782225973) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre Failing mds1 on oleg234-server Stopping /mnt/lustre-mds1 (opts:) on oleg234-server 10:48:29 (1782226109) shut down facet: mds1 facet_host: oleg234-server facet_failover_host: oleg234-server Failover mds1 to oleg234-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg234-server: oleg234-server.virtnet: executing set_default_debug -1 all pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Started lustre-MDT0000 10:48:53 (1782226133) targets are mounted 10:48:53 (1782226133) facet_failover done oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Starting client: oleg234-client.virtnet: -o user_xattr,flock 192.168.202.134@tcp:/lustre /mnt/lustre2 PASS 15c (237s) == replay-dual test 16: fail MDS during recovery (3571) == 10:50:10 (1782226210) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre total: 25 open/close in 1.13 seconds: 22.15 ops/second total: 1 open/close in 0.05 seconds: 18.91 ops/second Failing mds1 on oleg234-server Stopping /mnt/lustre-mds1 (opts:) on oleg234-server 10:50:24 (1782226224) shut down facet: mds1 facet_host: oleg234-server facet_failover_host: oleg234-server Failover mds1 to oleg234-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg234-server: oleg234-server.virtnet: executing set_default_debug -1 all pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Started lustre-MDT0000 10:50:56 (1782226256) targets are mounted 10:50:56 (1782226256) facet_failover done Failing mds1 on oleg234-server Stopping /mnt/lustre-mds1 (opts:) on oleg234-server 10:51:20 (1782226280) shut down facet: mds1 facet_host: oleg234-server facet_failover_host: oleg234-server Failover mds1 to oleg234-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg234-server: oleg234-server.virtnet: executing set_default_debug -1 all pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Started lustre-MDT0000 10:51:52 (1782226312) targets are mounted 10:51:52 (1782226312) facet_failover done oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec - unlinked 0 (time 1782226399 ; total 0 ; last 0) total: 25 unlinks in 1 seconds: 25.000000 unlinks/second Starting client: oleg234-client.virtnet: -o user_xattr,flock 192.168.202.134@tcp:/lustre /mnt/lustre2 PASS 16 (196s) == replay-dual test 17: fail OST during recovery (3571) == 10:53:26 (1782226406) total: 25 open/close in 1.17 seconds: 21.36 ops/second total: 1 open/close in 0.06 seconds: 16.92 ops/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre Failing ost1 on oleg234-server Stopping /mnt/lustre-ost1 (opts:) on oleg234-server 10:53:38 (1782226418) shut down facet: ost1 facet_host: oleg234-server facet_failover_host: oleg234-server Failover ost1 to oleg234-server mount facets: ost1 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg234-server: oleg234-server.virtnet: executing set_default_debug -1 all pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Started lustre-OST0000 10:54:05 (1782226445) targets are mounted 10:54:05 (1782226445) facet_failover done Failing ost1 on oleg234-server Stopping /mnt/lustre-ost1 (opts:) on oleg234-server 10:54:32 (1782226472) shut down facet: ost1 facet_host: oleg234-server facet_failover_host: oleg234-server Failover ost1 to oleg234-server mount facets: ost1 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg234-server: oleg234-server.virtnet: executing set_default_debug -1 all pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Started lustre-OST0000 10:54:58 (1782226498) targets are mounted 10:54:58 (1782226498) facet_failover done oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec - unlinked 0 (time 1782226571 ; total 0 ; last 0) total: 25 unlinks in 1 seconds: 25.000000 unlinks/second Starting client: oleg234-client.virtnet: -o user_xattr,flock 192.168.202.134@tcp:/lustre /mnt/lustre2 PASS 17 (173s) == replay-dual test 18: ldlm_handle_enqueue succeeds on evicted export (3822) ========================================================== 10:56:19 (1782226579) debug=+dlmtrace fail_loc=0x8000030b using seed 4001884472 running for 500 iterations total: 500 stats in 1 seconds: 500.000000 stats/second ldlm.namespaces.MGC192.168.202.134@tcp.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0000-mdc-ffff9ee910332800.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0000-mdc-ffff9ee911d6e000.early_lock_cancel=0 ldlm.namespaces.lustre-OST0000-osc-ffff9ee910332800.early_lock_cancel=0 ldlm.namespaces.lustre-OST0000-osc-ffff9ee911d6e000.early_lock_cancel=0 ldlm.namespaces.lustre-OST0001-osc-ffff9ee910332800.early_lock_cancel=0 ldlm.namespaces.lustre-OST0001-osc-ffff9ee911d6e000.early_lock_cancel=0 fail_loc=0x80000305 Error in opening file "/mnt/lustre2/d18.replay-dual/f18.replay-dual"(flags=O_RDONLY) 2: No such file or directory ldlm.namespaces.MGC192.168.202.134@tcp.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0000-mdc-ffff9ee910332800.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0000-mdc-ffff9ee911d6e000.early_lock_cancel=1 ldlm.namespaces.lustre-OST0000-osc-ffff9ee910332800.early_lock_cancel=1 ldlm.namespaces.lustre-OST0000-osc-ffff9ee911d6e000.early_lock_cancel=1 ldlm.namespaces.lustre-OST0001-osc-ffff9ee910332800.early_lock_cancel=1 ldlm.namespaces.lustre-OST0001-osc-ffff9ee911d6e000.early_lock_cancel=1 fail_loc=0 fail_loc=0 PASS 18 (62s) == replay-dual test 19: resend of open request =========== 10:57:21 (1782226641) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre fail_loc=0x157 - open/close 0 (time 1782226736.14 total 86.62 last 0.00) total: 1 open/close in 86.63 seconds: 0.01 ops/second fail_loc=0 Failing mds1 on oleg234-server Stopping /mnt/lustre-mds1 (opts:) on oleg234-server 10:59:01 (1782226741) shut down facet: mds1 facet_host: oleg234-server facet_failover_host: oleg234-server Failover mds1 to oleg234-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg234-server: oleg234-server.virtnet: executing set_default_debug -1 all pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Started lustre-MDT0000 10:59:25 (1782226765) targets are mounted 10:59:25 (1782226765) facet_failover done oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 19 (141s) == replay-dual test 20: recovery time is not increasing == 10:59:42 (1782226782) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre Failing mds1 on oleg234-server Stopping /mnt/lustre-mds1 (opts:) on oleg234-server 10:59:53 (1782226793) shut down facet: mds1 facet_host: oleg234-server facet_failover_host: oleg234-server Failover mds1 to oleg234-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg234-server: oleg234-server.virtnet: executing set_default_debug -1 all pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Started lustre-MDT0000 11:00:31 (1782226831) targets are mounted 11:00:31 (1782226831) facet_failover done oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Starting client: oleg234-client.virtnet: -o user_xattr,flock 192.168.202.134@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre Failing mds1 on oleg234-server Stopping /mnt/lustre-mds1 (opts:) on oleg234-server 11:03:06 (1782226986) shut down facet: mds1 facet_host: oleg234-server facet_failover_host: oleg234-server Failover mds1 to oleg234-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg234-server: oleg234-server.virtnet: executing set_default_debug -1 all pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Started lustre-MDT0000 11:03:41 (1782227021) targets are mounted 11:03:41 (1782227021) facet_failover done oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Starting client: oleg234-client.virtnet: -o user_xattr,flock 192.168.202.134@tcp:/lustre /mnt/lustre2 PASS 20 (386s) == replay-dual test 21a: commit on sharing =============== 11:06:08 (1782227168) mdt.lustre-MDT0000.commit_on_sharing=1 Replay barrier on lustre-MDT0000 Failing mds1 on oleg234-server Stopping /mnt/lustre-mds1 (opts:) on oleg234-server 11:06:21 (1782227181) shut down facet: mds1 facet_host: oleg234-server facet_failover_host: oleg234-server Failover mds1 to oleg234-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg234-server: oleg234-server.virtnet: executing set_default_debug -1 all pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Started lustre-MDT0000 11:06:57 (1782227217) targets are mounted 11:06:57 (1782227217) facet_failover done Starting client: oleg234-client.virtnet: -o user_xattr,flock 192.168.202.134@tcp:/lustre /mnt/lustre2 mdt.lustre-MDT0000.commit_on_sharing=0 PASS 21a (190s) SKIP: replay-dual test_21b skipping SLOW test 21b == replay-dual test 22a: c1 lfs mkdir -i 1 dir1, M1 drop reply & fail, c2 mkdir dir1/dir ========================================================== 11:09:20 (1782227360) SKIP: replay-dual test_22a needs >= 2 MDTs SKIP 22a (3s) == replay-dual test 22b: c1 lfs mkdir -i 1 d1, M1 drop reply & fail M0/M1, c2 mkdir d1/dir ========================================================== 11:09:23 (1782227363) SKIP: replay-dual test_22b needs >= 2 MDTs SKIP 22b (3s) == replay-dual test 22c: c1 lfs mkdir -i 1 d1, M1 drop update & fail M1, c2 mkdir d1/dir ========================================================== 11:09:27 (1782227367) SKIP: replay-dual test_22c needs >= 2 MDTs SKIP 22c (3s) == replay-dual test 22d: c1 lfs mkdir -i 1 d1, M1 drop update & fail M0/M1,c2 mkdir d1/dir ========================================================== 11:09:30 (1782227370) SKIP: replay-dual test_22d needs >= 2 MDTs SKIP 22d (3s) == replay-dual test 23a: c1 rmdir d1, M1 drop reply and fail, client2 mkdir d1 ========================================================== 11:09:33 (1782227373) SKIP: replay-dual test_23a needs >= 2 MDTs SKIP 23a (4s) == replay-dual test 23b: c1 rmdir d1, M1 drop reply and fail M0/M1, c2 mkdir d1 ========================================================== 11:09:37 (1782227377) SKIP: replay-dual test_23b needs >= 2 MDTs SKIP 23b (4s) == replay-dual test 23c: c1 rmdir d1, M0 drop update reply and fail M0, c2 mkdir d1 ========================================================== 11:09:41 (1782227381) SKIP: replay-dual test_23c needs >= 2 MDTs SKIP 23c (3s) == replay-dual test 23d: c1 rmdir d1, M0 drop update reply and fail M0/M1, c2 mkdir d1 ========================================================== 11:09:44 (1782227384) SKIP: replay-dual test_23d needs >= 2 MDTs SKIP 23d (4s) == replay-dual test 24: reconstruct on non-existing object ========================================================== 11:09:48 (1782227388) fail_loc=0x119 fail_loc=0 truncate: cannot truncate '/mnt/lustre/f24.replay-dual' to length 100: No such file or directory PASS 24 (95s) == replay-dual test 25: replay|resend ==================== 11:11:23 (1782227483) 1+0 records in 1+0 records out 512 bytes copied, 0.0129028 s, 39.7 kB/s fail_loc=0x304 fail_loc=0x80000325 Failing ost1 on oleg234-server Stopping /mnt/lustre-ost1 (opts:) on oleg234-server 11:11:33 (1782227493) shut down facet: ost1 facet_host: oleg234-server facet_failover_host: oleg234-server Failover ost1 to oleg234-server mount facets: ost1 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg234-server: oleg234-server.virtnet: executing set_default_debug -1 all pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 Started lustre-OST0000 11:11:59 (1782227519) targets are mounted 11:11:59 (1782227519) facet_failover done oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 6933: 43552 Terminated LUSTRE="/home/green/git/lustre-release/lustre" bash -c "multiop /mnt/lustre2/f25.replay-dual Ow512" fail_loc=0 PASS 25 (56s) == replay-dual test 26: dbench and tar with mds failover ========================================================== 11:12:19 (1782227539) Starting client oleg234-client.virtnet: -o user_xattr,flock 192.168.202.134@tcp:/lustre /mnt/lustre Started clients oleg234-client.virtnet: 192.168.202.134@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) Started tar loop with pid 44968 Started dbench loop with 44969 looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2048 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 14336 3738624 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3762176 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 16384 7500800 1% /mnt/lustre '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Tue Jun 23 11:12:25 EDT 2026 waiting for dbench pid 45037 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs failed to create barrier semaphore 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 14 0.00 MB/sec warmup 1 sec latency 222.971 ms 1 72 1.57 MB/sec warmup 2 sec latency 251.825 ms 1 113 1.74 MB/sec warmup 3 sec latency 206.754 ms 1 146 1.57 MB/sec warmup 4 sec latency 181.283 ms 1 181 1.46 MB/sec warmup 5 sec latency 133.837 ms test_26 fail mds1 1 times 1 207 1.35 MB/sec warmup 6 sec latency 176.561 ms 1 235 1.27 MB/sec warmup 7 sec latency 206.396 ms Failing mds1 on oleg234-server 1 263 1.24 MB/sec warmup 8 sec latency 151.864 ms Stopping /mnt/lustre-mds1 (opts:) on oleg234-server 1 290 1.19 MB/sec warmup 9 sec latency 169.729 ms 1 322 1.17 MB/sec warmup 10 sec latency 134.439 ms 1 322 1.07 MB/sec warmup 11 sec latency 1093.626 ms 1 322 0.98 MB/sec warmup 12 sec latency 2094.290 ms 1 322 0.90 MB/sec warmup 13 sec latency 3094.660 ms 1 322 0.84 MB/sec warmup 14 sec latency 4094.894 ms 11:12:40 (1782227560) shut down facet: mds1 facet_host: oleg234-server facet_failover_host: oleg234-server 1 322 0.78 MB/sec warmup 15 sec latency 5097.804 ms 1 322 0.73 MB/sec warmup 16 sec latency 6098.028 ms 1 322 0.69 MB/sec warmup 17 sec latency 7098.302 ms 1 322 0.65 MB/sec warmup 18 sec latency 8099.259 ms 1 322 0.62 MB/sec warmup 19 sec latency 9099.575 ms 1 322 0.00 MB/sec execute 1 sec latency 11103.548 ms 1 322 0.00 MB/sec execute 2 sec latency 12106.641 ms 1 322 0.00 MB/sec execute 3 sec latency 13106.869 ms 1 322 0.00 MB/sec execute 4 sec latency 14108.702 ms 1 322 0.00 MB/sec execute 5 sec latency 15120.568 ms Failover mds1 to oleg234-server mount facets: mds1 1 322 0.00 MB/sec execute 6 sec latency 16123.840 ms 1 322 0.00 MB/sec execute 7 sec latency 17124.244 ms 1 322 0.00 MB/sec execute 8 sec latency 18124.833 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 322 0.00 MB/sec execute 9 sec latency 19125.082 ms 1 322 0.00 MB/sec execute 10 sec latency 20125.480 ms 1 322 0.00 MB/sec execute 11 sec latency 21125.827 ms 1 322 0.00 MB/sec execute 12 sec latency 22126.088 ms 1 322 0.00 MB/sec execute 13 sec latency 23127.867 ms 1 322 0.00 MB/sec execute 14 sec latency 24128.697 ms 1 322 0.00 MB/sec execute 15 sec latency 25129.012 ms 1 322 0.00 MB/sec execute 16 sec latency 26129.365 ms oleg234-server: oleg234-server.virtnet: executing set_default_debug -1 all 1 322 0.00 MB/sec execute 17 sec latency 27132.353 ms pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 1 322 0.00 MB/sec execute 18 sec latency 28133.453 ms Started lustre-MDT0000 11:13:04 (1782227584) targets are mounted 11:13:04 (1782227584) facet_failover done 1 322 0.00 MB/sec execute 19 sec latency 29134.443 ms 1 322 0.00 MB/sec execute 20 sec latency 30136.365 ms 1 322 0.00 MB/sec execute 21 sec latency 31136.887 ms 1 322 0.00 MB/sec execute 22 sec latency 32137.132 ms 1 322 0.00 MB/sec execute 23 sec latency 33137.429 ms 1 322 0.00 MB/sec execute 24 sec latency 34137.676 ms 1 322 0.00 MB/sec execute 25 sec latency 35140.286 ms 1 322 0.00 MB/sec execute 26 sec latency 36140.523 ms 1 322 0.00 MB/sec execute 27 sec latency 37140.733 ms 1 322 0.00 MB/sec execute 28 sec latency 38140.931 ms 1 322 0.00 MB/sec execute 29 sec latency 39141.925 ms 1 323 0.00 MB/sec execute 30 sec latency 39919.155 ms 1 360 0.03 MB/sec execute 31 sec latency 325.590 ms 1 401 0.08 MB/sec execute 32 sec latency 140.553 ms 1 437 0.11 MB/sec execute 33 sec latency 122.076 ms 1 472 0.14 MB/sec execute 34 sec latency 193.014 ms 1 503 0.16 MB/sec execute 35 sec latency 140.101 ms oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 534 0.19 MB/sec execute 36 sec latency 194.484 ms 1 611 0.25 MB/sec execute 37 sec latency 141.592 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 659 0.25 MB/sec execute 38 sec latency 109.282 ms 1 676 0.24 MB/sec execute 39 sec latency 237.626 ms 1 710 0.24 MB/sec execute 40 sec latency 136.217 ms 1 730 0.23 MB/sec execute 41 sec latency 181.789 ms 1 762 0.23 MB/sec execute 42 sec latency 144.001 ms 1 790 0.23 MB/sec execute 43 sec latency 236.444 ms 1 808 0.22 MB/sec execute 44 sec latency 193.175 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2304 2206208 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 35840 3704832 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 15360 3718144 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 51200 7422976 1% /mnt/lustre 1 851 0.22 MB/sec execute 45 sec latency 157.922 ms 1 877 0.22 MB/sec execute 46 sec latency 221.057 ms 1 902 0.21 MB/sec execute 47 sec latency 147.271 ms 1 916 0.21 MB/sec execute 48 sec latency 237.109 ms 1 962 0.21 MB/sec execute 49 sec latency 85.683 ms 1 983 0.20 MB/sec execute 50 sec latency 327.333 ms test_26 fail mds1 2 times 1 1055 0.22 MB/sec execute 51 sec latency 152.096 ms 1 1099 0.22 MB/sec execute 52 sec latency 102.533 ms Failing mds1 on oleg234-server 1 1138 0.21 MB/sec execute 53 sec latency 99.177 ms Stopping /mnt/lustre-mds1 (opts:) on oleg234-server 1 1176 0.21 MB/sec execute 54 sec latency 111.825 ms 1 1196 0.21 MB/sec execute 55 sec latency 425.628 ms 1 1196 0.20 MB/sec execute 56 sec latency 1425.782 ms 1 1196 0.20 MB/sec execute 57 sec latency 2426.186 ms 11:13:43 (1782227623) shut down facet: mds1 facet_host: oleg234-server facet_failover_host: oleg234-server 1 1196 0.19 MB/sec execute 58 sec latency 3426.538 ms 1 1196 0.19 MB/sec execute 59 sec latency 4426.930 ms 1 1196 0.19 MB/sec execute 60 sec latency 5427.253 ms 1 1196 0.18 MB/sec execute 61 sec latency 6427.795 ms 1 1196 0.18 MB/sec execute 62 sec latency 7431.360 ms 1 1196 0.18 MB/sec execute 63 sec latency 8431.551 ms 1 1196 0.18 MB/sec execute 64 sec latency 9431.799 ms 1 1196 0.17 MB/sec execute 65 sec latency 10432.157 ms 1 1196 0.17 MB/sec execute 66 sec latency 11432.352 ms 1 1196 0.17 MB/sec execute 67 sec latency 12432.625 ms Failover mds1 to oleg234-server mount facets: mds1 1 1196 0.17 MB/sec execute 68 sec latency 13434.224 ms 1 1196 0.16 MB/sec execute 69 sec latency 14435.242 ms 1 1196 0.16 MB/sec execute 70 sec latency 15435.606 ms 1 1196 0.16 MB/sec execute 71 sec latency 16435.907 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 1196 0.16 MB/sec execute 72 sec latency 17436.181 ms 1 1196 0.15 MB/sec execute 73 sec latency 18436.777 ms 1 1196 0.15 MB/sec execute 74 sec latency 19438.314 ms 1 1196 0.15 MB/sec execute 75 sec latency 20441.575 ms 1 1196 0.15 MB/sec execute 76 sec latency 21442.903 ms 1 1196 0.15 MB/sec execute 77 sec latency 22444.250 ms 1 1196 0.14 MB/sec execute 78 sec latency 23444.813 ms 1 1196 0.14 MB/sec execute 79 sec latency 24445.233 ms 1 1196 0.14 MB/sec execute 80 sec latency 25446.107 ms 1 1196 0.14 MB/sec execute 81 sec latency 26447.108 ms 1 1196 0.14 MB/sec execute 82 sec latency 27448.871 ms 1 1196 0.14 MB/sec execute 83 sec latency 28452.606 ms 1 1196 0.13 MB/sec execute 84 sec latency 29452.934 ms 1 1196 0.13 MB/sec execute 85 sec latency 30453.166 ms 1 1196 0.13 MB/sec execute 86 sec latency 31454.417 ms 1 1196 0.13 MB/sec execute 87 sec latency 32454.830 ms 1 1196 0.13 MB/sec execute 88 sec latency 33457.838 ms 1 1196 0.13 MB/sec execute 89 sec latency 34458.385 ms oleg234-server: oleg234-server.virtnet: executing set_default_debug -1 all 1 1196 0.13 MB/sec execute 90 sec latency 35458.794 ms 1 1196 0.12 MB/sec execute 91 sec latency 36459.115 ms pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 1 1196 0.12 MB/sec execute 92 sec latency 37459.380 ms Started lustre-MDT0000 11:14:18 (1782227658) targets are mounted 11:14:18 (1782227658) facet_failover done 1 1196 0.12 MB/sec execute 93 sec latency 38459.697 ms 1 1196 0.12 MB/sec execute 94 sec latency 39459.849 ms 1 1196 0.12 MB/sec execute 95 sec latency 40460.091 ms 1 1200 0.12 MB/sec execute 96 sec latency 41161.863 ms 1 1229 0.12 MB/sec execute 97 sec latency 131.214 ms 1 1253 0.12 MB/sec execute 98 sec latency 150.309 ms 1 1384 0.14 MB/sec execute 99 sec latency 105.832 ms 1 cleanup 100 sec 1 cleanup 101 sec oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 cleanup 102 sec 1 cleanup 103 sec mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 0 cleanup 104 sec 0 cleanup 104 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 150 80.551 325.570 Close 134 310.600 39919.007 Rename 10 112.910 237.079 Unlink 17 30.544 63.397 Qpathinfo 99 441.828 41161.827 Qfileinfo 58 2.606 13.290 Qfsinfo 33 3.778 20.984 Sfileinfo 34 66.346 136.986 Find 48 7.946 75.744 WriteX 266 14.409 97.242 ReadX 224 3.176 77.803 Flush 24 152.024 327.310 Throughput 0.143765 MB/sec 1 clients 1 procs max_latency=41161.863 ms stopping dbench on /mnt/lustre at Tue Jun 23 11:14:30 EDT 2026 with return code 0 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2560 2205952 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 26624 3720192 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 24576 3724288 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 51200 7444480 1% /mnt/lustre '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Tue Jun 23 11:14:36 EDT 2026 waiting for dbench pid 47043 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 31 1.05 MB/sec warmup 1 sec latency 23.052 ms 1 84 1.90 MB/sec warmup 2 sec latency 126.671 ms 1 121 1.83 MB/sec warmup 3 sec latency 202.078 ms 1 155 1.63 MB/sec warmup 4 sec latency 137.591 ms 1 184 1.48 MB/sec warmup 5 sec latency 168.868 ms 1 207 1.35 MB/sec warmup 6 sec latency 185.448 ms test_26 fail mds1 3 times 1 239 1.30 MB/sec warmup 7 sec latency 153.671 ms 1 271 1.27 MB/sec warmup 8 sec latency 153.072 ms Failing mds1 on oleg234-server 1 305 1.24 MB/sec warmup 9 sec latency 138.961 ms Stopping /mnt/lustre-mds1 (opts:) on oleg234-server 1 339 1.22 MB/sec warmup 10 sec latency 182.870 ms 1 357 1.16 MB/sec warmup 11 sec latency 390.705 ms 1 357 1.06 MB/sec warmup 12 sec latency 1391.537 ms 1 357 0.98 MB/sec warmup 13 sec latency 2391.813 ms 11:14:50 (1782227690) shut down facet: mds1 facet_host: oleg234-server facet_failover_host: oleg234-server 1 357 0.91 MB/sec warmup 14 sec latency 3392.001 ms 1 357 0.85 MB/sec warmup 15 sec latency 4392.166 ms 1 357 0.80 MB/sec warmup 16 sec latency 5393.508 ms 1 357 0.75 MB/sec warmup 17 sec latency 6393.865 ms 1 357 0.71 MB/sec warmup 18 sec latency 7394.064 ms 1 357 0.67 MB/sec warmup 19 sec latency 8394.482 ms 1 357 0.00 MB/sec execute 1 sec latency 10399.989 ms 1 357 0.00 MB/sec execute 2 sec latency 11401.011 ms 1 357 0.00 MB/sec execute 3 sec latency 12401.542 ms 1 357 0.00 MB/sec execute 4 sec latency 13402.093 ms Failover mds1 to oleg234-server mount facets: mds1 1 357 0.00 MB/sec execute 5 sec latency 14402.288 ms 1 357 0.00 MB/sec execute 6 sec latency 15402.786 ms 1 357 0.00 MB/sec execute 7 sec latency 16403.112 ms 1 357 0.00 MB/sec execute 8 sec latency 17407.104 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 357 0.00 MB/sec execute 9 sec latency 18407.386 ms 1 357 0.00 MB/sec execute 10 sec latency 19407.658 ms 1 357 0.00 MB/sec execute 11 sec latency 20407.939 ms 1 357 0.00 MB/sec execute 12 sec latency 21409.827 ms 1 357 0.00 MB/sec execute 13 sec latency 22414.103 ms 1 357 0.00 MB/sec execute 14 sec latency 23414.544 ms oleg234-server: oleg234-server.virtnet: executing set_default_debug -1 all 1 357 0.00 MB/sec execute 15 sec latency 24414.736 ms pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 1 357 0.00 MB/sec execute 16 sec latency 25415.022 ms Started lustre-MDT0000 11:15:13 (1782227713) targets are mounted 11:15:13 (1782227713) facet_failover done 1 357 0.00 MB/sec execute 17 sec latency 26415.675 ms 1 357 0.00 MB/sec execute 18 sec latency 27416.897 ms 1 357 0.00 MB/sec execute 19 sec latency 28417.239 ms 1 357 0.00 MB/sec execute 20 sec latency 29417.721 ms 1 357 0.00 MB/sec execute 21 sec latency 30419.454 ms 1 357 0.00 MB/sec execute 22 sec latency 31419.718 ms 1 357 0.00 MB/sec execute 23 sec latency 32420.917 ms 1 357 0.00 MB/sec execute 24 sec latency 33421.382 ms 1 357 0.00 MB/sec execute 25 sec latency 34422.321 ms 1 357 0.00 MB/sec execute 26 sec latency 35423.661 ms 1 357 0.00 MB/sec execute 27 sec latency 36427.784 ms 1 357 0.00 MB/sec execute 28 sec latency 37430.509 ms 1 357 0.00 MB/sec execute 29 sec latency 38431.024 ms 1 357 0.00 MB/sec execute 30 sec latency 39434.427 ms 1 381 0.03 MB/sec execute 31 sec latency 39615.914 ms 1 421 0.06 MB/sec execute 32 sec latency 148.374 ms 1 461 0.10 MB/sec execute 33 sec latency 142.513 ms 1 493 0.13 MB/sec execute 34 sec latency 124.065 ms 1 522 0.15 MB/sec execute 35 sec latency 135.441 ms oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 606 0.23 MB/sec execute 36 sec latency 166.999 ms 1 655 0.22 MB/sec execute 37 sec latency 106.782 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 676 0.22 MB/sec execute 38 sec latency 314.072 ms 1 708 0.22 MB/sec execute 39 sec latency 130.440 ms 1 733 0.21 MB/sec execute 40 sec latency 228.108 ms 1 772 0.21 MB/sec execute 41 sec latency 166.769 ms 1 799 0.21 MB/sec execute 42 sec latency 222.513 ms 1 845 0.20 MB/sec execute 43 sec latency 124.211 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2944 2205568 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 36864 3705856 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 31744 3722240 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 68608 7428096 1% /mnt/lustre 1 874 0.20 MB/sec execute 44 sec latency 324.818 ms 1 900 0.20 MB/sec execute 45 sec latency 144.423 ms 1 921 0.20 MB/sec execute 46 sec latency 184.444 ms 1 969 0.19 MB/sec execute 47 sec latency 88.803 ms 1 988 0.19 MB/sec execute 48 sec latency 204.436 ms 1 1059 0.21 MB/sec execute 49 sec latency 98.277 ms test_26 fail mds1 4 times 1 1116 0.20 MB/sec execute 50 sec latency 75.600 ms 1 1146 0.20 MB/sec execute 51 sec latency 123.718 ms Failing mds1 on oleg234-server 1 1190 0.20 MB/sec execute 52 sec latency 162.042 ms Stopping /mnt/lustre-mds1 (opts:) on oleg234-server 1 1225 0.19 MB/sec execute 53 sec latency 105.129 ms 1 1236 0.19 MB/sec execute 54 sec latency 555.187 ms 1 1236 0.19 MB/sec execute 55 sec latency 1555.404 ms 1 1236 0.18 MB/sec execute 56 sec latency 2555.668 ms 11:15:53 (1782227753) shut down facet: mds1 facet_host: oleg234-server facet_failover_host: oleg234-server 1 1236 0.18 MB/sec execute 57 sec latency 3555.975 ms 1 1236 0.18 MB/sec execute 58 sec latency 4556.280 ms 1 1236 0.17 MB/sec execute 59 sec latency 5556.769 ms 1 1236 0.17 MB/sec execute 60 sec latency 6556.977 ms 1 1236 0.17 MB/sec execute 61 sec latency 7557.615 ms 1 1236 0.17 MB/sec execute 62 sec latency 8557.822 ms 1 1236 0.16 MB/sec execute 63 sec latency 9558.118 ms 1 1236 0.16 MB/sec execute 64 sec latency 10558.652 ms 1 1236 0.16 MB/sec execute 65 sec latency 11559.624 ms 1 1236 0.16 MB/sec execute 66 sec latency 12560.508 ms Failover mds1 to oleg234-server mount facets: mds1 1 1236 0.15 MB/sec execute 67 sec latency 13561.016 ms 1 1236 0.15 MB/sec execute 68 sec latency 14561.281 ms 1 1236 0.15 MB/sec execute 69 sec latency 15563.159 ms 1 1236 0.15 MB/sec execute 70 sec latency 16563.348 ms 1 1236 0.14 MB/sec execute 71 sec latency 17563.583 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 1236 0.14 MB/sec execute 72 sec latency 18563.853 ms 1 1236 0.14 MB/sec execute 73 sec latency 19564.459 ms 1 1236 0.14 MB/sec execute 74 sec latency 20565.167 ms 1 1236 0.14 MB/sec execute 75 sec latency 21569.000 ms 1 1236 0.14 MB/sec execute 76 sec latency 22569.464 ms 1 1236 0.13 MB/sec execute 77 sec latency 23570.132 ms 1 1236 0.13 MB/sec execute 78 sec latency 24570.399 ms 1 1236 0.13 MB/sec execute 79 sec latency 25571.730 ms 1 1236 0.13 MB/sec execute 80 sec latency 26572.888 ms 1 1236 0.13 MB/sec execute 81 sec latency 27573.189 ms 1 1236 0.13 MB/sec execute 82 sec latency 28573.453 ms 1 1236 0.12 MB/sec execute 83 sec latency 29577.314 ms 1 1236 0.12 MB/sec execute 84 sec latency 30577.558 ms 1 1236 0.12 MB/sec execute 85 sec latency 31577.833 ms 1 1236 0.12 MB/sec execute 86 sec latency 32578.943 ms oleg234-server: oleg234-server.virtnet: executing set_default_debug -1 all 1 1236 0.12 MB/sec execute 87 sec latency 33580.372 ms pdsh@oleg234-client: oleg234-server: ssh exited with exit code 1 1 1236 0.12 MB/sec execute 88 sec latency 34581.981 ms Started lustre-MDT0000 11:16:25 (1782227785) targets are mounted 11:16:25 (1782227785) facet_failover done 1 1236 0.12 MB/sec execute 89 sec latency 35583.067 ms 1 1236 0.11 MB/sec execute 90 sec latency 36583.293 ms 1 1236 0.11 MB/sec execute 91 sec latency 37584.443 ms 1 1236 0.11 MB/sec execute 92 sec latency 38585.974 ms 1 1236 0.11 MB/sec execute 93 sec latency 39586.792 ms 1 1236 0.11 MB/sec execute 94 sec latency 40587.061 ms 1 1236 0.11 MB/sec execute 95 sec latency 41588.280 ms 1 1251 0.11 MB/sec execute 96 sec latency 42051.128 ms 1 1318 0.12 MB/sec execute 97 sec latency 101.418 ms 1 1387 0.13 MB/sec execute 98 sec latency 485.562 ms 1 1419 0.13 MB/sec execute 99 sec latency 118.597 ms 1 cleanup 100 sec oleg234-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 cleanup 101 sec 1 cleanup 102 sec mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 cleanup 103 sec 0 cleanup 104 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 154 329.074 39615.898 Close 141 14.127 40.808 Rename 10 101.182 162.020 Unlink 19 34.083 55.495 Qpathinfo 108 24.257 198.949 Qfileinfo 55 3.009 26.160 Qfsinfo 33 3.129 18.273 Sfileinfo 34 1297.568 42051.032 Find 49 7.073 66.848 WriteX 248 13.511 66.517 ReadX 230 2.568 61.916 Flush 24 137.070 324.753 Throughput 0.133461 MB/sec 1 clients 1 procs max_latency=42051.128 ms stopping dbench on /mnt/lustre at Tue Jun 23 11:16:41 EDT 2026 with return code 0 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Tue Jun 23 11:16:46 EDT 2026 waiting for dbench pid 49025 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 39 1.37 MB/sec warmup 1 sec latency 118.569 ms 1 94 2.09 MB/sec warmup 2 sec latency 131.135 ms 1 137 2.01 MB/sec warmup 3 sec latency 159.942 ms 1 173 1.76 MB/sec warmup 4 sec latency 126.649 ms 1 214 1.67 MB/sec warmup 5 sec latency 142.602 ms tar: Unexpected EOF in archive tar: Unexpected EOF in archive 1 255 1.61 MB/sec warmup 6 sec latency 143.483 ms 1 289 1.53 MB/sec warmup 7 sec latency 206.639 ms