-----============= acceptance-small: replay-dual ============----- Fri Oct 24 08:51:47 EDT 2025 oleg441-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/replay-dual.*ex': No such file or directory excepting tests: 14b 21b skipping tests SLOW=no: 21b Starting client oleg441-client.virtnet: -o user_xattr,flock oleg441-server@tcp:/lustre /mnt/lustre2 Started clients oleg441-client.virtnet: 192.168.204.141@tcp:/lustre on /mnt/lustre2 type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt) oleg441-client.virtnet: executing check_config_client /mnt/lustre oleg441-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg441-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9dd8c8434800.idle_timeout=debug osc.lustre-OST0000-osc-ffff9dd8d8d67800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9dd8c8434800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9dd8d8d67800.idle_timeout=debug disable quota as required oleg441-server: oleg441-server.virtnet: executing set_default_debug -1 all 8 osd-ldiskfs.track_declares_assert=1 == replay-dual test 0a: expired recovery with lost client ========================================================== 08:52:03 (1761310323) Check file is LU482_FAILED=/tmp/replay-dual.lu482.c3BMrY UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1768 1285920 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1612 1286076 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre total: 50 open/close in 0.35 seconds: 142.60 ops/second fail_loc=0x80000514 Failing mds1 on oleg441-server Stopping /mnt/lustre-mds1 (opts:) on oleg441-server reboot facets: mds1 Failover mds1 to oleg441-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg441-server: oleg441-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg441-client: oleg441-server: ssh exited with exit code 1 Started lustre-MDT0000 Starting client: oleg441-client.virtnet: -o user_xattr,flock oleg441-server@tcp:/lustre /mnt/lustre2 - unlinked 0 (time 1761310432 ; total 0 ; last 0) total: 50 unlinks in 0 seconds: inf unlinks/second PASS 0a (111s) == replay-dual test 0b: lost client during waiting for next transno ========================================================== 08:53:55 (1761310435) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1768 1285920 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1612 1286076 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre Failing mds1 on oleg441-server Stopping /mnt/lustre-mds1 (opts:) on oleg441-server reboot facets: mds1 Failover mds1 to oleg441-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg441-server: oleg441-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg441-client: oleg441-server: ssh exited with exit code 1 Started lustre-MDT0000 Starting client: oleg441-client.virtnet: -o user_xattr,flock oleg441-server@tcp:/lustre /mnt/lustre Starting client: oleg441-client.virtnet: -o user_xattr,flock oleg441-server@tcp:/lustre /mnt/lustre2 PASS 0b (131s) == replay-dual test 1: |X| simple create ================= 08:56:06 (1761310566) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1768 1285920 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1612 1286076 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre mcreate: cannot create `/mnt/lustre/fsa-oleg441-client.virtnet' with mode 0100644: File exists Failing mds1 on oleg441-server Stopping /mnt/lustre-mds1 (opts:) on oleg441-server reboot facets: mds1 Failover mds1 to oleg441-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg441-server: oleg441-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg441-client: oleg441-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 1 (41s) == replay-dual test 2: |X| mkdir adir ==================== 08:56:47 (1761310607) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1768 1285920 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1612 1286076 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre Failing mds1 on oleg441-server Stopping /mnt/lustre-mds1 (opts:) on oleg441-server reboot facets: mds1 Failover mds1 to oleg441-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg441-server: oleg441-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg441-client: oleg441-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 2 (33s) == replay-dual test 3: |X| mkdir adir, mkdir adir/bdir === 08:57:21 (1761310641) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1768 1285920 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1612 1286076 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre Failing mds1 on oleg441-server Stopping /mnt/lustre-mds1 (opts:) on oleg441-server reboot facets: mds1 Failover mds1 to oleg441-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg441-server: oleg441-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg441-client: oleg441-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 3 (40s) == replay-dual test 4: |X| mkdir adir (-EEXIST), mkdir adir/bdir ========================================================== 08:58:00 (1761310680) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1916 1285772 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1760 1285928 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre mkdir: cannot create directory '/mnt/lustre/adir': File exists Failing mds1 on oleg441-server Stopping /mnt/lustre-mds1 (opts:) on oleg441-server reboot facets: mds1 Failover mds1 to oleg441-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg441-server: oleg441-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg441-client: oleg441-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 4 (32s) == replay-dual test 5: open, unlink |X| close ============ 08:58:32 (1761310712) multiop /mnt/lustre2/a vo_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7718 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1880 1285808 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1724 1285964 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre Failing mds1 on oleg441-server Stopping /mnt/lustre-mds1 (opts:) on oleg441-server reboot facets: mds1 Failover mds1 to oleg441-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg441-server: oleg441-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg441-client: oleg441-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 5 (40s) == replay-dual test 6: open1, open2, unlink |X| close1 [fail mds1] close2 ========================================================== 08:59:12 (1761310752) multiop /mnt/lustre2/a vo_c TMPPIPE=/tmp/multiop_open_wait_pipe.7718 multiop /mnt/lustre/a vo_c TMPPIPE=/tmp/multiop_open_wait_pipe.7718 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1880 1285808 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1724 1285964 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre Failing mds1 on oleg441-server Stopping /mnt/lustre-mds1 (opts:) on oleg441-server reboot facets: mds1 Failover mds1 to oleg441-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg441-server: oleg441-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg441-client: oleg441-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 6 (40s) == replay-dual test 8: replay of resent request ========== 08:59:52 (1761310792) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1880 1285808 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1724 1285964 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre fail_loc=0x119 fail_loc=0 Failing mds1 on oleg441-server Stopping /mnt/lustre-mds1 (opts:) on oleg441-server reboot facets: mds1 Failover mds1 to oleg441-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg441-server: oleg441-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg441-client: oleg441-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 8 (92s) == replay-dual test 9: resending a replayed create ======= 09:01:24 (1761310884) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1880 1285808 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1724 1285964 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre fail_loc=0x80000119 Failing mds1 on oleg441-server Stopping /mnt/lustre-mds1 (opts:) on oleg441-server reboot facets: mds1 Failover mds1 to oleg441-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg441-server: oleg441-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg441-client: oleg441-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec fail_loc=0 PASS 9 (75s) == replay-dual test 10: resending a replayed unlink ====== 09:02:39 (1761310959) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1880 1285808 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1724 1285964 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre fail_loc=0x80000119 Failing mds1 on oleg441-server Stopping /mnt/lustre-mds1 (opts:) on oleg441-server reboot facets: mds1 Failover mds1 to oleg441-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg441-server: oleg441-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg441-client: oleg441-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec fail_loc=0 PASS 10 (73s) == replay-dual test 11: both clients timeout during replay ========================================================== 09:03:52 (1761311032) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1880 1285808 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1724 1285964 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre fail_loc=0x0119 Failing mds1 on oleg441-server Stopping /mnt/lustre-mds1 (opts:) on oleg441-server reboot facets: mds1 Failover mds1 to oleg441-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg441-server: oleg441-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg441-client: oleg441-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 39 sec fail_loc=0 PASS 11 (76s) == replay-dual test 12: open resend timeout ============== 09:05:08 (1761311108) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1880 1285808 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1724 1285964 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre multiop /mnt/lustre/f12.replay-dual vmo_c TMPPIPE=/tmp/multiop_open_wait_pipe.7718 fail_loc=0x80000302 Failing mds1 on oleg441-server Stopping /mnt/lustre-mds1 (opts:) on oleg441-server reboot facets: mds1 Failover mds1 to oleg441-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg441-server: oleg441-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg441-client: oleg441-server: ssh exited with exit code 1 Started lustre-MDT0000 fail_loc=0 /mnt/lustre/f12.replay-dual /mnt/lustre/f12.replay-dual has type file OK PASS 12 (29s) == replay-dual test 13: close resend timeout ============= 09:05:37 (1761311137) multiop /mnt/lustre/f13.replay-dual vmo_c TMPPIPE=/tmp/multiop_open_wait_pipe.7718 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1880 1285808 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1724 1285964 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre fail_loc=0x80000115 Failing mds1 on oleg441-server Stopping /mnt/lustre-mds1 (opts:) on oleg441-server reboot facets: mds1 Failover mds1 to oleg441-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg441-server: oleg441-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg441-client: oleg441-server: ssh exited with exit code 1 Started lustre-MDT0000 fail_loc=0 /mnt/lustre/f13.replay-dual /mnt/lustre/f13.replay-dual has type file OK PASS 13 (29s) SKIP: replay-dual test_14b skipping ALWAYS excluded test 14b == replay-dual test 15a: timeout waiting for lost client during replay, 1 client completes ========================================================== 09:06:06 (1761311166) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1880 1285808 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1724 1285964 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre total: 25 open/close in 0.18 seconds: 135.24 ops/second total: 1 open/close in 0.01 seconds: 147.88 ops/second Failing mds1 on oleg441-server Stopping /mnt/lustre-mds1 (opts:) on oleg441-server reboot facets: mds1 Failover mds1 to oleg441-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg441-server: oleg441-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg441-client: oleg441-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec - unlinked 0 (time 1761311299 ; total 0 ; last 0) total: 25 unlinks in 0 seconds: inf unlinks/second Starting client: oleg441-client.virtnet: -o user_xattr,flock oleg441-server@tcp:/lustre /mnt/lustre2 PASS 15a (137s) == replay-dual test 15c: remove multiple OST orphans ===== 09:08:23 (1761311303) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1880 1285808 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1724 1285964 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre Failing mds1 on oleg441-server Stopping /mnt/lustre-mds1 (opts:) on oleg441-server reboot facets: mds1 Failover mds1 to oleg441-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg441-server: oleg441-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg441-client: oleg441-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Starting client: oleg441-client.virtnet: -o user_xattr,flock oleg441-server@tcp:/lustre /mnt/lustre2 PASS 15c (228s) == replay-dual test 16: fail MDS during recovery (3571) == 09:12:11 (1761311531) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1880 1285808 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1724 1285964 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre total: 25 open/close in 0.79 seconds: 31.79 ops/second total: 1 open/close in 0.02 seconds: 50.20 ops/second Failing mds1 on oleg441-server Stopping /mnt/lustre-mds1 (opts:) on oleg441-server reboot facets: mds1 Failover mds1 to oleg441-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg441-server: oleg441-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg441-client: oleg441-server: ssh exited with exit code 1 Started lustre-MDT0000 Failing mds1 on oleg441-server Stopping /mnt/lustre-mds1 (opts:) on oleg441-server reboot facets: mds1 Failover mds1 to oleg441-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg441-server: oleg441-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg441-client: oleg441-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec - unlinked 0 (time 1761311737 ; total 0 ; last 0) total: 25 unlinks in 1 seconds: 25.000000 unlinks/second Starting client: oleg441-client.virtnet: -o user_xattr,flock oleg441-server@tcp:/lustre /mnt/lustre2 PASS 16 (212s) == replay-dual test 17: fail OST during recovery (3571) == 09:15:43 (1761311743) total: 25 open/close in 0.72 seconds: 34.77 ops/second total: 1 open/close in 0.04 seconds: 23.38 ops/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1880 1285808 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1724 1285964 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre Failing ost1 on oleg441-server Stopping /mnt/lustre-ost1 (opts:) on oleg441-server reboot facets: ost1 Failover ost1 to oleg441-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg441-server: oleg441-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg441-client: oleg441-server: ssh exited with exit code 1 Started lustre-OST0000 Failing ost1 on oleg441-server Stopping /mnt/lustre-ost1 (opts:) on oleg441-server reboot facets: ost1 Failover ost1 to oleg441-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg441-server: oleg441-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg441-client: oleg441-server: ssh exited with exit code 1 Started lustre-OST0000 oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec - unlinked 0 (time 1761311940 ; total 0 ; last 0) total: 25 unlinks in 0 seconds: inf unlinks/second Starting client: oleg441-client.virtnet: -o user_xattr,flock oleg441-server@tcp:/lustre /mnt/lustre2 PASS 17 (202s) == replay-dual test 18: ldlm_handle_enqueue succeeds on evicted export (3822) ========================================================== 09:19:05 (1761311945) debug=+dlmtrace using seed 3726350975 running for 500 iterations total: 500 stats in 0 seconds: inf stats/second fail_loc=0x8000030b ldlm.namespaces.MGC192.168.204.141@tcp.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0000-mdc-ffff9dd8c4fb6800.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0000-mdc-ffff9dd8c7872000.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0001-mdc-ffff9dd8c4fb6800.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0001-mdc-ffff9dd8c7872000.early_lock_cancel=0 ldlm.namespaces.lustre-OST0000-osc-ffff9dd8c4fb6800.early_lock_cancel=0 ldlm.namespaces.lustre-OST0000-osc-ffff9dd8c7872000.early_lock_cancel=0 ldlm.namespaces.lustre-OST0001-osc-ffff9dd8c4fb6800.early_lock_cancel=0 ldlm.namespaces.lustre-OST0001-osc-ffff9dd8c7872000.early_lock_cancel=0 fail_loc=0x80000305 Error in opening file "/mnt/lustre2/d18.replay-dual/f18.replay-dual"(flags=O_RDONLY) 2: No such file or directory ldlm.namespaces.MGC192.168.204.141@tcp.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0000-mdc-ffff9dd8c4fb6800.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0000-mdc-ffff9dd8c7872000.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0001-mdc-ffff9dd8c4fb6800.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0001-mdc-ffff9dd8c7872000.early_lock_cancel=1 ldlm.namespaces.lustre-OST0000-osc-ffff9dd8c4fb6800.early_lock_cancel=1 ldlm.namespaces.lustre-OST0000-osc-ffff9dd8c7872000.early_lock_cancel=1 ldlm.namespaces.lustre-OST0001-osc-ffff9dd8c4fb6800.early_lock_cancel=1 ldlm.namespaces.lustre-OST0001-osc-ffff9dd8c7872000.early_lock_cancel=1 fail_loc=0 fail_loc=0 PASS 18 (55s) == replay-dual test 19: resend of open request =========== 09:20:00 (1761312000) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1884 1285804 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1724 1285964 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1404 3605616 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2792 7211248 1% /mnt/lustre fail_loc=0x157 - open/close 0 (time 1761312104.89 total 93.48 last 0.00) total: 1 open/close in 93.48 seconds: 0.01 ops/second fail_loc=0 Failing mds1 on oleg441-server Stopping /mnt/lustre-mds1 (opts:) on oleg441-server reboot facets: mds1 Failover mds1 to oleg441-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg441-server: oleg441-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg441-client: oleg441-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 19 (153s) == replay-dual test 20: recovery time is not increasing == 09:22:33 (1761312153) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1884 1285804 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1724 1285964 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1404 3605616 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2792 7211248 1% /mnt/lustre Failing mds1 on oleg441-server Stopping /mnt/lustre-mds1 (opts:) on oleg441-server reboot facets: mds1 Failover mds1 to oleg441-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg441-server: oleg441-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg441-client: oleg441-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Starting client: oleg441-client.virtnet: -o user_xattr,flock oleg441-server@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1884 1285804 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1724 1285964 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1404 3605616 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2792 7211248 1% /mnt/lustre Failing mds1 on oleg441-server Stopping /mnt/lustre-mds1 (opts:) on oleg441-server reboot facets: mds1 Failover mds1 to oleg441-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg441-server: oleg441-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg441-client: oleg441-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Starting client: oleg441-client.virtnet: -o user_xattr,flock oleg441-server@tcp:/lustre /mnt/lustre2