-----============= acceptance-small: replay-dual ============----- Wed Apr 15 10:59:10 EDT 2026 oleg652-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/replay-dual.*ex': No such file or directory excepting tests: 14b 21b skipping tests SLOW=no: 21b Starting client oleg652-client.virtnet: -o user_xattr,flock oleg652-server@tcp:/lustre /mnt/lustre2 Started clients oleg652-client.virtnet: 192.168.206.152@tcp:/lustre on /mnt/lustre2 type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt) oleg652-client.virtnet: executing check_config_client /mnt/lustre oleg652-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg652-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff96f582f7e000.idle_timeout=debug osc.lustre-OST0000-osc-ffff96f5863d0000.idle_timeout=debug osc.lustre-OST0001-osc-ffff96f582f7e000.idle_timeout=debug osc.lustre-OST0001-osc-ffff96f5863d0000.idle_timeout=debug disable quota as required oleg652-server: oleg652-server.virtnet: executing set_default_debug -1 all 8 osd-ldiskfs.track_declares_assert=1 == replay-dual test 0a: expired recovery with lost client ========================================================== 10:59:58 (1776265198) Check file is LU482_FAILED=/tmp/replay-dual.lu482.2qSG3H UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1768 1285920 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1612 1286076 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre total: 50 open/close in 1.34 seconds: 37.26 ops/second fail_loc=0x80000514 Failing mds1 on oleg652-server Stopping /mnt/lustre-mds1 (opts:) on oleg652-server reboot facets: mds1 Failover mds1 to oleg652-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg652-server: oleg652-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg652-client: oleg652-server: ssh exited with exit code 1 Started lustre-MDT0000 Starting client: oleg652-client.virtnet: -o user_xattr,flock oleg652-server@tcp:/lustre /mnt/lustre2 - unlinked 0 (time 1776265352 ; total 0 ; last 0) total: 50 unlinks in 1 seconds: 50.000000 unlinks/second PASS 0a (161s) == replay-dual test 0b: lost client during waiting for next transno ========================================================== 11:02:39 (1776265359) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1768 1285920 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1612 1286076 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre Failing mds1 on oleg652-server Stopping /mnt/lustre-mds1 (opts:) on oleg652-server reboot facets: mds1 Failover mds1 to oleg652-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg652-server: oleg652-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg652-client: oleg652-server: ssh exited with exit code 1 Started lustre-MDT0000 Starting client: oleg652-client.virtnet: -o user_xattr,flock oleg652-server@tcp:/lustre /mnt/lustre Starting client: oleg652-client.virtnet: -o user_xattr,flock oleg652-server@tcp:/lustre /mnt/lustre2 PASS 0b (198s) == replay-dual test 1: |X| simple create ================= 11:05:57 (1776265557) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1768 1285920 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1612 1286076 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre mcreate: cannot create `/mnt/lustre/fsa-oleg652-client.virtnet' with mode 0100644: File exists Failing mds1 on oleg652-server Stopping /mnt/lustre-mds1 (opts:) on oleg652-server reboot facets: mds1 Failover mds1 to oleg652-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg652-server: oleg652-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg652-client: oleg652-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 1 (68s) == replay-dual test 2: |X| mkdir adir ==================== 11:07:06 (1776265626) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1768 1285920 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1612 1286076 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre Failing mds1 on oleg652-server Stopping /mnt/lustre-mds1 (opts:) on oleg652-server reboot facets: mds1 Failover mds1 to oleg652-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg652-server: oleg652-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg652-client: oleg652-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 2 (64s) == replay-dual test 3: |X| mkdir adir, mkdir adir/bdir === 11:08:09 (1776265689) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1768 1285920 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1612 1286076 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre Failing mds1 on oleg652-server Stopping /mnt/lustre-mds1 (opts:) on oleg652-server reboot facets: mds1 Failover mds1 to oleg652-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg652-server: oleg652-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg652-client: oleg652-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 3 (61s) == replay-dual test 4: |X| mkdir adir (-EEXIST), mkdir adir/bdir ========================================================== 11:09:10 (1776265750) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1920 1285768 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1760 1285928 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre mkdir: cannot create directory '/mnt/lustre/adir': File exists Failing mds1 on oleg652-server Stopping /mnt/lustre-mds1 (opts:) on oleg652-server reboot facets: mds1 Failover mds1 to oleg652-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg652-server: oleg652-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg652-client: oleg652-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 4 (63s) == replay-dual test 5: open, unlink |X| close ============ 11:10:13 (1776265813) multiop /mnt/lustre2/a vo_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7722 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1880 1285808 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1724 1285964 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre Failing mds1 on oleg652-server Stopping /mnt/lustre-mds1 (opts:) on oleg652-server reboot facets: mds1 Failover mds1 to oleg652-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg652-server: oleg652-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg652-client: oleg652-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 5 (64s) == replay-dual test 6: open1, open2, unlink |X| close1 [fail mds1] close2 ========================================================== 11:11:17 (1776265877) multiop /mnt/lustre2/a vo_c TMPPIPE=/tmp/multiop_open_wait_pipe.7722 multiop /mnt/lustre/a vo_c TMPPIPE=/tmp/multiop_open_wait_pipe.7722 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1880 1285808 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1724 1285964 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre Failing mds1 on oleg652-server Stopping /mnt/lustre-mds1 (opts:) on oleg652-server reboot facets: mds1 Failover mds1 to oleg652-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg652-server: oleg652-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg652-client: oleg652-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 6 (66s) == replay-dual test 8: replay of resent request ========== 11:12:23 (1776265943) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1880 1285808 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1724 1285964 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre fail_loc=0x119 fail_loc=0 Failing mds1 on oleg652-server Stopping /mnt/lustre-mds1 (opts:) on oleg652-server reboot facets: mds1 Failover mds1 to oleg652-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg652-server: oleg652-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg652-client: oleg652-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 8 (109s) == replay-dual test 9: resending a replayed create ======= 11:14:12 (1776266052) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1880 1285808 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1724 1285964 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre fail_loc=0x80000119 Failing mds1 on oleg652-server Stopping /mnt/lustre-mds1 (opts:) on oleg652-server reboot facets: mds1 Failover mds1 to oleg652-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg652-server: oleg652-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg652-client: oleg652-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec fail_loc=0 PASS 9 (98s) == replay-dual test 10: resending a replayed unlink ====== 11:15:50 (1776266150) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1880 1285808 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1724 1285964 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre fail_loc=0x80000119 Failing mds1 on oleg652-server Stopping /mnt/lustre-mds1 (opts:) on oleg652-server reboot facets: mds1 Failover mds1 to oleg652-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg652-server: oleg652-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg652-client: oleg652-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec fail_loc=0 PASS 10 (103s) == replay-dual test 11: both clients timeout during replay ========================================================== 11:17:33 (1776266253) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1880 1285808 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1724 1285964 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre fail_loc=0x0119 Failing mds1 on oleg652-server Stopping /mnt/lustre-mds1 (opts:) on oleg652-server reboot facets: mds1 Failover mds1 to oleg652-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg652-server: oleg652-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg652-client: oleg652-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg652-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 29 sec fail_loc=0 PASS 11 (93s) == replay-dual test 12: open resend timeout ============== 11:19:06 (1776266346) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1880 1285808 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1724 1285964 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre multiop /mnt/lustre/f12.replay-dual vmo_c TMPPIPE=/tmp/multiop_open_wait_pipe.7722 fail_loc=0x80000302 Failing mds1 on oleg652-server Stopping /mnt/lustre-mds1 (opts:) on oleg652-server reboot facets: mds1 Failover mds1 to oleg652-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg652-server: oleg652-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg652-client: oleg652-server: ssh exited with exit code 1 Started lustre-MDT0000 fail_loc=0 /mnt/lustre/f12.replay-dual /mnt/lustre/f12.replay-dual has type file OK PASS 12 (95s) == replay-dual test 13: close resend timeout ============= 11:20:41 (1776266441) multiop /mnt/lustre/f13.replay-dual vmo_c TMPPIPE=/tmp/multiop_open_wait_pipe.7722 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1880 1285808 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1724 1285964 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre fail_loc=0x80000115 Failing mds1 on oleg652-server Stopping /mnt/lustre-mds1 (opts:) on oleg652-server reboot facets: mds1 Failover mds1 to oleg652-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg652-server: oleg652-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg652-client: oleg652-server: ssh exited with exit code 1 Started lustre-MDT0000 fail_loc=0 /mnt/lustre/f13.replay-dual /mnt/lustre/f13.replay-dual has type file OK PASS 13 (59s) SKIP: replay-dual test_14b skipping ALWAYS excluded test 14b == replay-dual test 15a: timeout waiting for lost client during replay, 1 client completes ========================================================== 11:21:42 (1776266502) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1880 1285808 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1724 1285964 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre total: 25 open/close in 0.65 seconds: 38.33 ops/second total: 1 open/close in 0.02 seconds: 44.97 ops/second Failing mds1 on oleg652-server Stopping /mnt/lustre-mds1 (opts:) on oleg652-server reboot facets: mds1 Failover mds1 to oleg652-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg652-server: oleg652-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg652-client: oleg652-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec - unlinked 0 (time 1776266671 ; total 0 ; last 0) total: 25 unlinks in 1 seconds: 25.000000 unlinks/second Starting client: oleg652-client.virtnet: -o user_xattr,flock oleg652-server@tcp:/lustre /mnt/lustre2 PASS 15a (176s) == replay-dual test 15c: remove multiple OST orphans ===== 11:24:39 (1776266679) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1880 1285808 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1724 1285964 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre Failing mds1 on oleg652-server Stopping /mnt/lustre-mds1 (opts:) on oleg652-server reboot facets: mds1 Failover mds1 to oleg652-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg652-server: oleg652-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg652-client: oleg652-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Starting client: oleg652-client.virtnet: -o user_xattr,flock oleg652-server@tcp:/lustre /mnt/lustre2 PASS 15c (254s) == replay-dual test 16: fail MDS during recovery (3571) == 11:28:54 (1776266934) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1880 1285808 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1724 1285964 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre total: 25 open/close in 0.60 seconds: 41.89 ops/second total: 1 open/close in 0.02 seconds: 46.97 ops/second Failing mds1 on oleg652-server Stopping /mnt/lustre-mds1 (opts:) on oleg652-server reboot facets: mds1 Failover mds1 to oleg652-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg652-server: oleg652-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg652-client: oleg652-server: ssh exited with exit code 1 Started lustre-MDT0000 Failing mds1 on oleg652-server Stopping /mnt/lustre-mds1 (opts:) on oleg652-server reboot facets: mds1 Failover mds1 to oleg652-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg652-server: oleg652-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg652-client: oleg652-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec - unlinked 0 (time 1776267137 ; total 0 ; last 0) total: 25 unlinks in 1 seconds: 25.000000 unlinks/second Starting client: oleg652-client.virtnet: -o user_xattr,flock oleg652-server@tcp:/lustre /mnt/lustre2 PASS 16 (210s) == replay-dual test 17: fail OST during recovery (3571) == 11:32:24 (1776267144) total: 25 open/close in 0.69 seconds: 36.25 ops/second total: 1 open/close in 0.05 seconds: 21.45 ops/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1880 1285808 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1724 1285964 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2776 7211264 1% /mnt/lustre Failing ost1 on oleg652-server Stopping /mnt/lustre-ost1 (opts:) on oleg652-server reboot facets: ost1 Failover ost1 to oleg652-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg652-server: oleg652-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg652-client: oleg652-server: ssh exited with exit code 1 Started lustre-OST0000 Failing ost1 on oleg652-server Stopping /mnt/lustre-ost1 (opts:) on oleg652-server reboot facets: ost1 Failover ost1 to oleg652-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg652-server: oleg652-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg652-client: oleg652-server: ssh exited with exit code 1 Started lustre-OST0000 oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec - unlinked 0 (time 1776267344 ; total 0 ; last 0) total: 25 unlinks in 0 seconds: inf unlinks/second Starting client: oleg652-client.virtnet: -o user_xattr,flock oleg652-server@tcp:/lustre /mnt/lustre2 PASS 17 (207s) == replay-dual test 18: ldlm_handle_enqueue succeeds on evicted export (3822) ========================================================== 11:35:52 (1776267352) debug=+dlmtrace fail_loc=0x8000030b using seed 1040877461 running for 500 iterations total: 500 stats in 1 seconds: 500.000000 stats/second ldlm.namespaces.MGC192.168.206.152@tcp.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0000-mdc-ffff96f5863d4800.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0000-mdc-ffff96f5880ce000.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0001-mdc-ffff96f5863d4800.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0001-mdc-ffff96f5880ce000.early_lock_cancel=0 ldlm.namespaces.lustre-OST0000-osc-ffff96f5863d4800.early_lock_cancel=0 ldlm.namespaces.lustre-OST0000-osc-ffff96f5880ce000.early_lock_cancel=0 ldlm.namespaces.lustre-OST0001-osc-ffff96f5863d4800.early_lock_cancel=0 ldlm.namespaces.lustre-OST0001-osc-ffff96f5880ce000.early_lock_cancel=0 fail_loc=0x80000305 Error in opening file "/mnt/lustre2/d18.replay-dual/f18.replay-dual"(flags=O_RDONLY) 2: No such file or directory ldlm.namespaces.MGC192.168.206.152@tcp.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0000-mdc-ffff96f5863d4800.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0000-mdc-ffff96f5880ce000.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0001-mdc-ffff96f5863d4800.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0001-mdc-ffff96f5880ce000.early_lock_cancel=1 ldlm.namespaces.lustre-OST0000-osc-ffff96f5863d4800.early_lock_cancel=1 ldlm.namespaces.lustre-OST0000-osc-ffff96f5880ce000.early_lock_cancel=1 ldlm.namespaces.lustre-OST0001-osc-ffff96f5863d4800.early_lock_cancel=1 ldlm.namespaces.lustre-OST0001-osc-ffff96f5880ce000.early_lock_cancel=1 fail_loc=0 fail_loc=0 PASS 18 (61s) == replay-dual test 19: resend of open request =========== 11:36:53 (1776267413) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1884 1285804 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1724 1285964 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1404 3605616 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2792 7211248 1% /mnt/lustre fail_loc=0x157 - open/close 0 (time 1776267518.76 total 90.51 last 0.00) total: 1 open/close in 90.51 seconds: 0.01 ops/second fail_loc=0 Failing mds1 on oleg652-server Stopping /mnt/lustre-mds1 (opts:) on oleg652-server reboot facets: mds1 Failover mds1 to oleg652-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg652-server: oleg652-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg652-client: oleg652-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 19 (162s) == replay-dual test 20: recovery time is not increasing == 11:39:35 (1776267575) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1884 1285804 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1724 1285964 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1404 3605616 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2792 7211248 1% /mnt/lustre Failing mds1 on oleg652-server Stopping /mnt/lustre-mds1 (opts:) on oleg652-server reboot facets: mds1 Failover mds1 to oleg652-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg652-server: oleg652-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg652-client: oleg652-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Starting client: oleg652-client.virtnet: -o user_xattr,flock oleg652-server@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1884 1285804 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1724 1285964 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1404 3605616 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2792 7211248 1% /mnt/lustre Failing mds1 on oleg652-server Stopping /mnt/lustre-mds1 (opts:) on oleg652-server reboot facets: mds1 Failover mds1 to oleg652-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg652-server: oleg652-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg652-client: oleg652-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg652-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Starting client: oleg652-client.virtnet: -o user_xattr,flock oleg652-server@tcp:/lustre /mnt/lustre2