-----============= acceptance-small: replay-dual ============----- Fri Jun 12 12:10:39 EDT 2026
oleg343-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/replay-dual.*ex': No such file or directory
excepting tests: 14b 21b 21b
skipping tests SLOW=no: 21b
Starting client oleg343-client.virtnet:  -o user_xattr,flock oleg343-server@tcp:/lustre /mnt/lustre2
Started clients oleg343-client.virtnet: 
192.168.203.143@tcp:/lustre on /mnt/lustre2 type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt)
oleg343-client.virtnet: executing check_config_client /mnt/lustre
oleg343-client.virtnet: Checking config lustre mounted on /mnt/lustre
Checking servers environments
Checking clients oleg343-client.virtnet environments
Using TIMEOUT=20
osc.lustre-OST0000-osc-ffff9353d195e000.idle_timeout=debug
osc.lustre-OST0000-osc-ffff9353d2bfc800.idle_timeout=debug
osc.lustre-OST0001-osc-ffff9353d195e000.idle_timeout=debug
osc.lustre-OST0001-osc-ffff9353d2bfc800.idle_timeout=debug
disable quota as required
oleg343-server: oleg343-server.virtnet: executing set_default_debug -1 all 8

== replay-dual test 0a: expired recovery with lost client ========================================================== 12:11:35 (1781280695)
Check file is LU482_FAILED=/tmp/replay-dual.lu482.sexRhY
UUID                   1K-blocks        Used   Available Use% Mounted on
lustre-MDT0000_UUID      2210688        3200     2205440   1% /mnt/lustre[MDT:0]
lustre-OST0000_UUID      3771392        3072     3766272   1% /mnt/lustre[OST:0]
lustre-OST0001_UUID      3771392        3072     3766272   1% /mnt/lustre[OST:1]

filesystem_summary:      7542784        6144     7532544   1% /mnt/lustre

total: 50 open/close in 1.22 seconds: 41.13 ops/second
fail_loc=0x80000514
Failing mds1 on oleg343-server
Stopping /mnt/lustre-mds1 (opts:) on oleg343-server
reboot facets: mds1
Failover mds1 to oleg343-server
mount facets: mds1
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg343-server: oleg343-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg343-client: oleg343-server: ssh exited with exit code 1
Started lustre-MDT0000
Starting client: oleg343-client.virtnet:  -o user_xattr,flock oleg343-server@tcp:/lustre /mnt/lustre2
 - unlinked 0 (time 1781280825 ; total 0 ; last 0)
total: 50 unlinks in 1 seconds: 50.000000 unlinks/second
PASS 0a (140s)

== replay-dual test 0b: lost client during waiting for next transno ========================================================== 12:13:54 (1781280834)
UUID                   1K-blocks        Used   Available Use% Mounted on
lustre-MDT0000_UUID      2210688        3200     2205440   1% /mnt/lustre[MDT:0]
lustre-OST0000_UUID      3771392        3072     3766272   1% /mnt/lustre[OST:0]
lustre-OST0001_UUID      3771392        3072     3766272   1% /mnt/lustre[OST:1]

filesystem_summary:      7542784        6144     7532544   1% /mnt/lustre

Failing mds1 on oleg343-server
Stopping /mnt/lustre-mds1 (opts:) on oleg343-server
reboot facets: mds1
Failover mds1 to oleg343-server
mount facets: mds1
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg343-server: oleg343-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg343-client: oleg343-server: ssh exited with exit code 1
Started lustre-MDT0000
Starting client: oleg343-client.virtnet:  -o user_xattr,flock oleg343-server@tcp:/lustre /mnt/lustre
Starting client: oleg343-client.virtnet:  -o user_xattr,flock oleg343-server@tcp:/lustre /mnt/lustre2
PASS 0b (147s)

== replay-dual test 1: |X| simple create ================= 12:16:21 (1781280981)
UUID                   1K-blocks        Used   Available Use% Mounted on
lustre-MDT0000_UUID      2210688        3200     2205440   1% /mnt/lustre[MDT:0]
lustre-OST0000_UUID      3771392        3072     3766272   1% /mnt/lustre[OST:0]
lustre-OST0001_UUID      3771392        3072     3766272   1% /mnt/lustre[OST:1]

filesystem_summary:      7542784        6144     7532544   1% /mnt/lustre

Failing mds1 on oleg343-server
Stopping /mnt/lustre-mds1 (opts:) on oleg343-server
reboot facets: mds1
Failover mds1 to oleg343-server
mount facets: mds1
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg343-server: oleg343-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg343-client: oleg343-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
PASS 1 (72s)

== replay-dual test 2: |X| mkdir adir ==================== 12:17:33 (1781281053)
UUID                   1K-blocks        Used   Available Use% Mounted on
lustre-MDT0000_UUID      2210688        3200     2205440   1% /mnt/lustre[MDT:0]
lustre-OST0000_UUID      3771392        3072     3766272   1% /mnt/lustre[OST:0]
lustre-OST0001_UUID      3771392        3072     3766272   1% /mnt/lustre[OST:1]

filesystem_summary:      7542784        6144     7532544   1% /mnt/lustre

Failing mds1 on oleg343-server
Stopping /mnt/lustre-mds1 (opts:) on oleg343-server
reboot facets: mds1
Failover mds1 to oleg343-server
mount facets: mds1
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg343-server: oleg343-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg343-client: oleg343-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
PASS 2 (60s)

== replay-dual test 3: |X| mkdir adir, mkdir adir/bdir === 12:18:33 (1781281113)
UUID                   1K-blocks        Used   Available Use% Mounted on
lustre-MDT0000_UUID      2210688        3200     2205440   1% /mnt/lustre[MDT:0]
lustre-OST0000_UUID      3771392        3072     3766272   1% /mnt/lustre[OST:0]
lustre-OST0001_UUID      3771392        3072     3766272   1% /mnt/lustre[OST:1]

filesystem_summary:      7542784        6144     7532544   1% /mnt/lustre

Failing mds1 on oleg343-server
Stopping /mnt/lustre-mds1 (opts:) on oleg343-server
reboot facets: mds1
Failover mds1 to oleg343-server
mount facets: mds1
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg343-server: oleg343-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg343-client: oleg343-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
PASS 3 (59s)

== replay-dual test 4: |X| mkdir adir (-EEXIST), mkdir adir/bdir ========================================================== 12:19:32 (1781281172)
UUID                   1K-blocks        Used   Available Use% Mounted on
lustre-MDT0000_UUID      2210688        3200     2205440   1% /mnt/lustre[MDT:0]
lustre-OST0000_UUID      3771392        3072     3766272   1% /mnt/lustre[OST:0]
lustre-OST0001_UUID      3771392        3072     3766272   1% /mnt/lustre[OST:1]

filesystem_summary:      7542784        6144     7532544   1% /mnt/lustre

mkdir: cannot create directory '/mnt/lustre/adir': File exists
Failing mds1 on oleg343-server
Stopping /mnt/lustre-mds1 (opts:) on oleg343-server
reboot facets: mds1
Failover mds1 to oleg343-server
mount facets: mds1
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg343-server: oleg343-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg343-client: oleg343-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
PASS 4 (70s)

== replay-dual test 5: open, unlink |X| close ============ 12:20:43 (1781281243)
multiop /mnt/lustre2/a vo_tSc
TMPPIPE=/tmp/multiop_open_wait_pipe.7155
UUID                   1K-blocks        Used   Available Use% Mounted on
lustre-MDT0000_UUID      2210688        3200     2205440   1% /mnt/lustre[MDT:0]
lustre-OST0000_UUID      3771392        3072     3766272   1% /mnt/lustre[OST:0]
lustre-OST0001_UUID      3771392        3072     3766272   1% /mnt/lustre[OST:1]

filesystem_summary:      7542784        6144     7532544   1% /mnt/lustre

Failing mds1 on oleg343-server
Stopping /mnt/lustre-mds1 (opts:) on oleg343-server
reboot facets: mds1
Failover mds1 to oleg343-server
mount facets: mds1
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg343-server: oleg343-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg343-client: oleg343-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
PASS 5 (68s)

== replay-dual test 6: open1, open2, unlink |X| close1 [fail mds1] close2 ========================================================== 12:21:51 (1781281311)
multiop /mnt/lustre2/a vo_c
TMPPIPE=/tmp/multiop_open_wait_pipe.7155
multiop /mnt/lustre/a vo_c
TMPPIPE=/tmp/multiop_open_wait_pipe.7155
UUID                   1K-blocks        Used   Available Use% Mounted on
lustre-MDT0000_UUID      2210688        3200     2205440   1% /mnt/lustre[MDT:0]
lustre-OST0000_UUID      3771392        3072     3766272   1% /mnt/lustre[OST:0]
lustre-OST0001_UUID      3771392        3072     3766272   1% /mnt/lustre[OST:1]

filesystem_summary:      7542784        6144     7532544   1% /mnt/lustre

Failing mds1 on oleg343-server
Stopping /mnt/lustre-mds1 (opts:) on oleg343-server
reboot facets: mds1
Failover mds1 to oleg343-server
mount facets: mds1
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg343-server: oleg343-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg343-client: oleg343-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
PASS 6 (67s)

== replay-dual test 8: replay of resent request ========== 12:22:58 (1781281378)
UUID                   1K-blocks        Used   Available Use% Mounted on
lustre-MDT0000_UUID      2210688        3200     2205440   1% /mnt/lustre[MDT:0]
lustre-OST0000_UUID      3771392        3072     3766272   1% /mnt/lustre[OST:0]
lustre-OST0001_UUID      3771392        3072     3766272   1% /mnt/lustre[OST:1]

filesystem_summary:      7542784        6144     7532544   1% /mnt/lustre

fail_loc=0x119
fail_loc=0
Failing mds1 on oleg343-server
Stopping /mnt/lustre-mds1 (opts:) on oleg343-server
reboot facets: mds1
Failover mds1 to oleg343-server
mount facets: mds1
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg343-server: oleg343-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg343-client: oleg343-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
PASS 8 (74s)

== replay-dual test 9: resending a replayed create ======= 12:24:12 (1781281452)
UUID                   1K-blocks        Used   Available Use% Mounted on
lustre-MDT0000_UUID      2210688        3200     2205440   1% /mnt/lustre[MDT:0]
lustre-OST0000_UUID      3771392        3072     3766272   1% /mnt/lustre[OST:0]
lustre-OST0001_UUID      3771392        3072     3766272   1% /mnt/lustre[OST:1]

filesystem_summary:      7542784        6144     7532544   1% /mnt/lustre

fail_loc=0x80000119
Failing mds1 on oleg343-server
Stopping /mnt/lustre-mds1 (opts:) on oleg343-server
reboot facets: mds1
Failover mds1 to oleg343-server
mount facets: mds1
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg343-server: oleg343-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg343-client: oleg343-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
fail_loc=0
PASS 9 (85s)

== replay-dual test 10: resending a replayed unlink ====== 12:25:38 (1781281538)
UUID                   1K-blocks        Used   Available Use% Mounted on
lustre-MDT0000_UUID      2210688        3200     2205440   1% /mnt/lustre[MDT:0]
lustre-OST0000_UUID      3771392        3072     3766272   1% /mnt/lustre[OST:0]
lustre-OST0001_UUID      3771392        3072     3766272   1% /mnt/lustre[OST:1]

filesystem_summary:      7542784        6144     7532544   1% /mnt/lustre

fail_loc=0x80000119
Failing mds1 on oleg343-server
Stopping /mnt/lustre-mds1 (opts:) on oleg343-server
reboot facets: mds1
Failover mds1 to oleg343-server
mount facets: mds1
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg343-server: oleg343-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg343-client: oleg343-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
fail_loc=0
PASS 10 (85s)

== replay-dual test 11: both clients timeout during replay ========================================================== 12:27:03 (1781281623)
UUID                   1K-blocks        Used   Available Use% Mounted on
lustre-MDT0000_UUID      2210688        3200     2205440   1% /mnt/lustre[MDT:0]
lustre-OST0000_UUID      3771392        3072     3766272   1% /mnt/lustre[OST:0]
lustre-OST0001_UUID      3771392        3072     3766272   1% /mnt/lustre[OST:1]

filesystem_summary:      7542784        6144     7532544   1% /mnt/lustre

fail_loc=0x0119
Failing mds1 on oleg343-server
Stopping /mnt/lustre-mds1 (opts:) on oleg343-server
reboot facets: mds1
Failover mds1 to oleg343-server
mount facets: mds1
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg343-server: oleg343-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg343-client: oleg343-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 8 sec
fail_loc=0
PASS 11 (67s)

== replay-dual test 12: open resend timeout ============== 12:28:10 (1781281690)
UUID                   1K-blocks        Used   Available Use% Mounted on
lustre-MDT0000_UUID      2210688        3200     2205440   1% /mnt/lustre[MDT:0]
lustre-OST0000_UUID      3771392        3072     3766272   1% /mnt/lustre[OST:0]
lustre-OST0001_UUID      3771392        3072     3766272   1% /mnt/lustre[OST:1]

filesystem_summary:      7542784        6144     7532544   1% /mnt/lustre

multiop /mnt/lustre/f12.replay-dual vmo_c
TMPPIPE=/tmp/multiop_open_wait_pipe.7155
fail_loc=0x80000302
Failing mds1 on oleg343-server
Stopping /mnt/lustre-mds1 (opts:) on oleg343-server
reboot facets: mds1
Failover mds1 to oleg343-server
mount facets: mds1
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg343-server: oleg343-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg343-client: oleg343-server: ssh exited with exit code 1
Started lustre-MDT0000
fail_loc=0
/mnt/lustre/f12.replay-dual
/mnt/lustre/f12.replay-dual has type file OK
PASS 12 (46s)

== replay-dual test 13: close resend timeout ============= 12:28:56 (1781281736)
multiop /mnt/lustre/f13.replay-dual vmo_c
TMPPIPE=/tmp/multiop_open_wait_pipe.7155
UUID                   1K-blocks        Used   Available Use% Mounted on
lustre-MDT0000_UUID      2210688        3200     2205440   1% /mnt/lustre[MDT:0]
lustre-OST0000_UUID      3771392        3072     3766272   1% /mnt/lustre[OST:0]
lustre-OST0001_UUID      3771392        3072     3766272   1% /mnt/lustre[OST:1]

filesystem_summary:      7542784        6144     7532544   1% /mnt/lustre

fail_loc=0x80000115
Failing mds1 on oleg343-server
Stopping /mnt/lustre-mds1 (opts:) on oleg343-server
reboot facets: mds1
Failover mds1 to oleg343-server
mount facets: mds1
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg343-server: oleg343-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg343-client: oleg343-server: ssh exited with exit code 1
Started lustre-MDT0000
fail_loc=0
/mnt/lustre/f13.replay-dual
/mnt/lustre/f13.replay-dual has type file OK
PASS 13 (41s)

 SKIP: replay-dual test_14b skipping ALWAYS excluded test 14b

== replay-dual test 15a: timeout waiting for lost client during replay, 1 client completes ========================================================== 12:29:40 (1781281780)
UUID                   1K-blocks        Used   Available Use% Mounted on
lustre-MDT0000_UUID      2210688        3200     2205440   1% /mnt/lustre[MDT:0]
lustre-OST0000_UUID      3771392        3072     3766272   1% /mnt/lustre[OST:0]
lustre-OST0001_UUID      3771392        3072     3766272   1% /mnt/lustre[OST:1]

filesystem_summary:      7542784        6144     7532544   1% /mnt/lustre

total: 25 open/close in 0.82 seconds: 30.36 ops/second
total: 1 open/close in 0.03 seconds: 33.75 ops/second
Failing mds1 on oleg343-server
Stopping /mnt/lustre-mds1 (opts:) on oleg343-server
reboot facets: mds1
Failover mds1 to oleg343-server
mount facets: mds1
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg343-server: oleg343-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg343-client: oleg343-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
 - unlinked 0 (time 1781281892 ; total 0 ; last 0)
total: 25 unlinks in 1 seconds: 25.000000 unlinks/second
Starting client: oleg343-client.virtnet:  -o user_xattr,flock oleg343-server@tcp:/lustre /mnt/lustre2
PASS 15a (119s)

== replay-dual test 15c: remove multiple OST orphans ===== 12:31:38 (1781281898)
UUID                   1K-blocks        Used   Available Use% Mounted on
lustre-MDT0000_UUID      2210688        3200     2205440   1% /mnt/lustre[MDT:0]
lustre-OST0000_UUID      3771392        3072     3766272   1% /mnt/lustre[OST:0]
lustre-OST0001_UUID      3771392        3072     3766272   1% /mnt/lustre[OST:1]

filesystem_summary:      7542784        6144     7532544   1% /mnt/lustre

Failing mds1 on oleg343-server
Stopping /mnt/lustre-mds1 (opts:) on oleg343-server
reboot facets: mds1
Failover mds1 to oleg343-server
mount facets: mds1
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg343-server: oleg343-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg343-client: oleg343-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
Starting client: oleg343-client.virtnet:  -o user_xattr,flock oleg343-server@tcp:/lustre /mnt/lustre2
PASS 15c (216s)

== replay-dual test 16: fail MDS during recovery (3571) == 12:35:14 (1781282114)
UUID                   1K-blocks        Used   Available Use% Mounted on
lustre-MDT0000_UUID      2210688        3200     2205440   1% /mnt/lustre[MDT:0]
lustre-OST0000_UUID      3771392        3072     3766272   1% /mnt/lustre[OST:0]
lustre-OST0001_UUID      3771392        3072     3766272   1% /mnt/lustre[OST:1]

filesystem_summary:      7542784        6144     7532544   1% /mnt/lustre

total: 25 open/close in 0.56 seconds: 45.04 ops/second
total: 1 open/close in 0.02 seconds: 49.26 ops/second
Failing mds1 on oleg343-server
Stopping /mnt/lustre-mds1 (opts:) on oleg343-server
reboot facets: mds1
Failover mds1 to oleg343-server
mount facets: mds1
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg343-server: oleg343-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg343-client: oleg343-server: ssh exited with exit code 1
Started lustre-MDT0000
Failing mds1 on oleg343-server
Stopping /mnt/lustre-mds1 (opts:) on oleg343-server
reboot facets: mds1
Failover mds1 to oleg343-server
mount facets: mds1
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg343-server: oleg343-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg343-client: oleg343-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
 - unlinked 0 (time 1781282323 ; total 0 ; last 0)
total: 25 unlinks in 1 seconds: 25.000000 unlinks/second
Starting client: oleg343-client.virtnet:  -o user_xattr,flock oleg343-server@tcp:/lustre /mnt/lustre2
PASS 16 (215s)

== replay-dual test 17: fail OST during recovery (3571) == 12:38:49 (1781282329)
total: 25 open/close in 0.49 seconds: 50.57 ops/second
total: 1 open/close in 0.04 seconds: 23.57 ops/second
UUID                   1K-blocks        Used   Available Use% Mounted on
lustre-MDT0000_UUID      2210688        3200     2205440   1% /mnt/lustre[MDT:0]
lustre-OST0000_UUID      3771392        3072     3766272   1% /mnt/lustre[OST:0]
lustre-OST0001_UUID      3771392        3072     3766272   1% /mnt/lustre[OST:1]

filesystem_summary:      7542784        6144     7532544   1% /mnt/lustre

Failing ost1 on oleg343-server
Stopping /mnt/lustre-ost1 (opts:) on oleg343-server
reboot facets: ost1
Failover ost1 to oleg343-server
mount facets: ost1
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
oleg343-server: oleg343-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg343-client: oleg343-server: ssh exited with exit code 1
Started lustre-OST0000
Failing ost1 on oleg343-server
Stopping /mnt/lustre-ost1 (opts:) on oleg343-server
reboot facets: ost1
Failover ost1 to oleg343-server
mount facets: ost1
Starting ost1: -o localrecov  lustre-ost1/ost1 /mnt/lustre-ost1
oleg343-server: oleg343-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg343-client: oleg343-server: ssh exited with exit code 1
Started lustre-OST0000
oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
 - unlinked 0 (time 1781282513 ; total 0 ; last 0)
total: 25 unlinks in 0 seconds: inf unlinks/second
Starting client: oleg343-client.virtnet:  -o user_xattr,flock oleg343-server@tcp:/lustre /mnt/lustre2
PASS 17 (189s)

== replay-dual test 18: ldlm_handle_enqueue succeeds on evicted export (3822) ========================================================== 12:41:58 (1781282518)
debug=+dlmtrace
fail_loc=0x8000030b
using seed 143381450
running for 500 iterations
total: 500 stats in 1 seconds: 500.000000 stats/second
ldlm.namespaces.MGC192.168.203.143@tcp.early_lock_cancel=0
ldlm.namespaces.lustre-MDT0000-mdc-ffff9353c6ead000.early_lock_cancel=0
ldlm.namespaces.lustre-MDT0000-mdc-ffff9353d28d3000.early_lock_cancel=0
ldlm.namespaces.lustre-OST0000-osc-ffff9353c6ead000.early_lock_cancel=0
ldlm.namespaces.lustre-OST0000-osc-ffff9353d28d3000.early_lock_cancel=0
ldlm.namespaces.lustre-OST0001-osc-ffff9353c6ead000.early_lock_cancel=0
ldlm.namespaces.lustre-OST0001-osc-ffff9353d28d3000.early_lock_cancel=0
fail_loc=0x80000305
Error in opening file "/mnt/lustre2/d18.replay-dual/f18.replay-dual"(flags=O_RDONLY) 2: No such file or directory
ldlm.namespaces.MGC192.168.203.143@tcp.early_lock_cancel=1
ldlm.namespaces.lustre-MDT0000-mdc-ffff9353c6ead000.early_lock_cancel=1
ldlm.namespaces.lustre-MDT0000-mdc-ffff9353d28d3000.early_lock_cancel=1
ldlm.namespaces.lustre-OST0000-osc-ffff9353c6ead000.early_lock_cancel=1
ldlm.namespaces.lustre-OST0000-osc-ffff9353d28d3000.early_lock_cancel=1
ldlm.namespaces.lustre-OST0001-osc-ffff9353c6ead000.early_lock_cancel=1
ldlm.namespaces.lustre-OST0001-osc-ffff9353d28d3000.early_lock_cancel=1
fail_loc=0
fail_loc=0
PASS 18 (54s)

== replay-dual test 19: resend of open request =========== 12:42:52 (1781282572)
UUID                   1K-blocks        Used   Available Use% Mounted on
lustre-MDT0000_UUID      2210688        3328     2205312   1% /mnt/lustre[MDT:0]
lustre-OST0000_UUID      3771392        3072     3766272   1% /mnt/lustre[OST:0]
lustre-OST0001_UUID      3771392        3072     3766272   1% /mnt/lustre[OST:1]

filesystem_summary:      7542784        6144     7532544   1% /mnt/lustre

fail_loc=0x157
 - open/close 0 (time 1781282674.10 total 94.65 last 0.00)
total: 1 open/close in 94.65 seconds: 0.01 ops/second
fail_loc=0
Failing mds1 on oleg343-server
Stopping /mnt/lustre-mds1 (opts:) on oleg343-server
reboot facets: mds1
Failover mds1 to oleg343-server
mount facets: mds1
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg343-server: oleg343-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg343-client: oleg343-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
PASS 19 (154s)

== replay-dual test 20: recovery time is not increasing == 12:45:26 (1781282726)
UUID                   1K-blocks        Used   Available Use% Mounted on
lustre-MDT0000_UUID      2210688        3328     2205312   1% /mnt/lustre[MDT:0]
lustre-OST0000_UUID      3771392        3072     3766272   1% /mnt/lustre[OST:0]
lustre-OST0001_UUID      3771392        3072     3766272   1% /mnt/lustre[OST:1]

filesystem_summary:      7542784        6144     7532544   1% /mnt/lustre

Failing mds1 on oleg343-server
Stopping /mnt/lustre-mds1 (opts:) on oleg343-server
reboot facets: mds1
Failover mds1 to oleg343-server
mount facets: mds1
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg343-server: oleg343-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg343-client: oleg343-server: ssh exited with exit code 1
Started lustre-MDT0000
oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
Starting client: oleg343-client.virtnet:  -o user_xattr,flock oleg343-server@tcp:/lustre /mnt/lustre2
UUID                   1K-blocks        Used   Available Use% Mounted on
lustre-MDT0000_UUID      2210688        3328     2205312   1% /mnt/lustre[MDT:0]
lustre-OST0000_UUID      3771392        3072     3766272   1% /mnt/lustre[OST:0]
lustre-OST0001_UUID      3771392        3072     3766272   1% /mnt/lustre[OST:1]

filesystem_summary:      7542784        6144     7532544   1% /mnt/lustre

Failing mds1 on oleg343-server
Stopping /mnt/lustre-mds1 (opts:) on oleg343-server
reboot facets: mds1
Failover mds1 to oleg343-server
mount facets: mds1
Starting mds1: -o localrecov  lustre-mdt1/mdt1 /mnt/lustre-mds1
oleg343-server: oleg343-server.virtnet: executing set_default_debug -1 all 8
pdsh@oleg343-client: oleg343-server: ssh exited with exit code 1
Started lustre-MDT0000