-----============= acceptance-small: replay-dual ============----- Tue Apr 14 14:55:09 EDT 2026 oleg450-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/replay-dual.*ex': No such file or directory excepting tests: 14b 21b skipping tests SLOW=no: 21b Starting client oleg450-client.virtnet: -o user_xattr,flock oleg450-server@tcp:/lustre /mnt/lustre2 Started clients oleg450-client.virtnet: 192.168.204.150@tcp:/lustre on /mnt/lustre2 type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt) oleg450-client.virtnet: executing check_config_client /mnt/lustre oleg450-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg450-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8fec0905b000.idle_timeout=debug osc.lustre-OST0000-osc-ffff8fec11fdd000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8fec0905b000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8fec11fdd000.idle_timeout=debug disable quota as required oleg450-server: oleg450-server.virtnet: executing set_default_debug -1 all 8 osd-ldiskfs.track_declares_assert=1 == replay-dual test 0a: expired recovery with lost client ========================================================== 14:56:07 (1776192967) Check file is LU482_FAILED=/tmp/replay-dual.lu482.bPTAq6 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1768 1285920 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1612 1286076 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1252 3605768 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2640 7211400 1% /mnt/lustre total: 50 open/close in 1.75 seconds: 28.57 ops/second fail_loc=0x80000514 Failing mds1 on oleg450-server Stopping /mnt/lustre-mds1 (opts:) on oleg450-server reboot facets: mds1 Failover mds1 to oleg450-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg450-server: oleg450-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg450-client: oleg450-server: ssh exited with exit code 1 Started lustre-MDT0000 Starting client: oleg450-client.virtnet: -o user_xattr,flock oleg450-server@tcp:/lustre /mnt/lustre2 - unlinked 0 (time 1776193141 ; total 0 ; last 0) total: 50 unlinks in 2 seconds: 25.000000 unlinks/second PASS 0a (183s) == replay-dual test 0b: lost client during waiting for next transno ========================================================== 14:59:10 (1776193150) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1768 1285920 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1612 1286076 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1252 3605768 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2640 7211400 1% /mnt/lustre Failing mds1 on oleg450-server Stopping /mnt/lustre-mds1 (opts:) on oleg450-server reboot facets: mds1 Failover mds1 to oleg450-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg450-server: oleg450-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg450-client: oleg450-server: ssh exited with exit code 1 Started lustre-MDT0000 Starting client: oleg450-client.virtnet: -o user_xattr,flock oleg450-server@tcp:/lustre /mnt/lustre Starting client: oleg450-client.virtnet: -o user_xattr,flock oleg450-server@tcp:/lustre /mnt/lustre2 PASS 0b (199s) == replay-dual test 1: |X| simple create ================= 15:02:30 (1776193350) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1768 1285920 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1612 1286076 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1252 3605768 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2640 7211400 1% /mnt/lustre mcreate: cannot create `/mnt/lustre/fsa-oleg450-client.virtnet' with mode 0100644: File exists Failing mds1 on oleg450-server Stopping /mnt/lustre-mds1 (opts:) on oleg450-server reboot facets: mds1 Failover mds1 to oleg450-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg450-server: oleg450-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg450-client: oleg450-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 1 (79s) == replay-dual test 2: |X| mkdir adir ==================== 15:03:48 (1776193428) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1768 1285920 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1612 1286076 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1252 3605768 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2640 7211400 1% /mnt/lustre Failing mds1 on oleg450-server Stopping /mnt/lustre-mds1 (opts:) on oleg450-server reboot facets: mds1 Failover mds1 to oleg450-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg450-server: oleg450-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg450-client: oleg450-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 2 (67s) == replay-dual test 3: |X| mkdir adir, mkdir adir/bdir === 15:04:56 (1776193496) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1768 1285920 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1612 1286076 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1252 3605768 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2640 7211400 1% /mnt/lustre Failing mds1 on oleg450-server Stopping /mnt/lustre-mds1 (opts:) on oleg450-server reboot facets: mds1 Failover mds1 to oleg450-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg450-server: oleg450-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg450-client: oleg450-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 3 (88s) == replay-dual test 4: |X| mkdir adir (-EEXIST), mkdir adir/bdir ========================================================== 15:06:24 (1776193584) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1920 1285768 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1760 1285928 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1252 3605768 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2640 7211400 1% /mnt/lustre mkdir: cannot create directory '/mnt/lustre/adir': File exists Failing mds1 on oleg450-server Stopping /mnt/lustre-mds1 (opts:) on oleg450-server reboot facets: mds1 Failover mds1 to oleg450-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg450-server: oleg450-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg450-client: oleg450-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 4 (68s) == replay-dual test 5: open, unlink |X| close ============ 15:07:32 (1776193652) multiop /mnt/lustre2/a vo_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7729 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1880 1285808 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1724 1285964 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1252 3605768 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2640 7211400 1% /mnt/lustre Failing mds1 on oleg450-server Stopping /mnt/lustre-mds1 (opts:) on oleg450-server reboot facets: mds1 Failover mds1 to oleg450-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg450-server: oleg450-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg450-client: oleg450-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 5 (82s) == replay-dual test 6: open1, open2, unlink |X| close1 [fail mds1] close2 ========================================================== 15:08:55 (1776193735) multiop /mnt/lustre2/a vo_c TMPPIPE=/tmp/multiop_open_wait_pipe.7729 multiop /mnt/lustre/a vo_c TMPPIPE=/tmp/multiop_open_wait_pipe.7729 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1880 1285808 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1724 1285964 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1252 3605768 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2640 7211400 1% /mnt/lustre Failing mds1 on oleg450-server Stopping /mnt/lustre-mds1 (opts:) on oleg450-server reboot facets: mds1 Failover mds1 to oleg450-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg450-server: oleg450-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg450-client: oleg450-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 6 (67s) == replay-dual test 8: replay of resent request ========== 15:10:02 (1776193802) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1880 1285808 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1724 1285964 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1252 3605768 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2640 7211400 1% /mnt/lustre fail_loc=0x119 fail_loc=0 Failing mds1 on oleg450-server Stopping /mnt/lustre-mds1 (opts:) on oleg450-server reboot facets: mds1 Failover mds1 to oleg450-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg450-server: oleg450-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg450-client: oleg450-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 8 (111s) == replay-dual test 9: resending a replayed create ======= 15:11:53 (1776193913) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1880 1285808 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1724 1285964 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1252 3605768 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2640 7211400 1% /mnt/lustre fail_loc=0x80000119 Failing mds1 on oleg450-server Stopping /mnt/lustre-mds1 (opts:) on oleg450-server reboot facets: mds1 Failover mds1 to oleg450-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg450-server: oleg450-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg450-client: oleg450-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec fail_loc=0 PASS 9 (107s) == replay-dual test 10: resending a replayed unlink ====== 15:13:40 (1776194020) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1880 1285808 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1724 1285964 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1252 3605768 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2640 7211400 1% /mnt/lustre fail_loc=0x80000119 Failing mds1 on oleg450-server Stopping /mnt/lustre-mds1 (opts:) on oleg450-server reboot facets: mds1 Failover mds1 to oleg450-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg450-server: oleg450-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg450-client: oleg450-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec fail_loc=0 PASS 10 (107s) == replay-dual test 11: both clients timeout during replay ========================================================== 15:15:28 (1776194128) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1880 1285808 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1724 1285964 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1252 3605768 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2640 7211400 1% /mnt/lustre fail_loc=0x0119 Failing mds1 on oleg450-server Stopping /mnt/lustre-mds1 (opts:) on oleg450-server reboot facets: mds1 Failover mds1 to oleg450-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg450-server: oleg450-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg450-client: oleg450-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 33 sec fail_loc=0 PASS 11 (102s) == replay-dual test 12: open resend timeout ============== 15:17:10 (1776194230) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1880 1285808 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1724 1285964 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1252 3605768 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2640 7211400 1% /mnt/lustre multiop /mnt/lustre/f12.replay-dual vmo_c TMPPIPE=/tmp/multiop_open_wait_pipe.7729 fail_loc=0x80000302 Failing mds1 on oleg450-server Stopping /mnt/lustre-mds1 (opts:) on oleg450-server reboot facets: mds1 Failover mds1 to oleg450-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg450-server: oleg450-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg450-client: oleg450-server: ssh exited with exit code 1 Started lustre-MDT0000 fail_loc=0 /mnt/lustre/f12.replay-dual /mnt/lustre/f12.replay-dual has type file OK PASS 12 (94s) == replay-dual test 13: close resend timeout ============= 15:18:44 (1776194324) multiop /mnt/lustre/f13.replay-dual vmo_c TMPPIPE=/tmp/multiop_open_wait_pipe.7729 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1880 1285808 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1724 1285964 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1252 3605768 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2640 7211400 1% /mnt/lustre fail_loc=0x80000115 Failing mds1 on oleg450-server Stopping /mnt/lustre-mds1 (opts:) on oleg450-server reboot facets: mds1 Failover mds1 to oleg450-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg450-server: oleg450-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg450-client: oleg450-server: ssh exited with exit code 1 Started lustre-MDT0000 fail_loc=0 /mnt/lustre/f13.replay-dual /mnt/lustre/f13.replay-dual has type file OK PASS 13 (58s) SKIP: replay-dual test_14b skipping ALWAYS excluded test 14b == replay-dual test 15a: timeout waiting for lost client during replay, 1 client completes ========================================================== 15:19:44 (1776194384) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1880 1285808 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1724 1285964 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1252 3605768 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2640 7211400 1% /mnt/lustre total: 25 open/close in 0.73 seconds: 34.05 ops/second total: 1 open/close in 0.03 seconds: 29.83 ops/second Failing mds1 on oleg450-server Stopping /mnt/lustre-mds1 (opts:) on oleg450-server reboot facets: mds1 Failover mds1 to oleg450-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg450-server: oleg450-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg450-client: oleg450-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec - unlinked 0 (time 1776194538 ; total 0 ; last 0) total: 25 unlinks in 0 seconds: inf unlinks/second Starting client: oleg450-client.virtnet: -o user_xattr,flock oleg450-server@tcp:/lustre /mnt/lustre2 PASS 15a (159s) == replay-dual test 15c: remove multiple OST orphans ===== 15:22:24 (1776194544) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1880 1285808 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1724 1285964 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1252 3605768 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2640 7211400 1% /mnt/lustre Failing mds1 on oleg450-server Stopping /mnt/lustre-mds1 (opts:) on oleg450-server reboot facets: mds1 Failover mds1 to oleg450-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg450-server: oleg450-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg450-client: oleg450-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Starting client: oleg450-client.virtnet: -o user_xattr,flock oleg450-server@tcp:/lustre /mnt/lustre2 PASS 15c (235s) == replay-dual test 16: fail MDS during recovery (3571) == 15:26:18 (1776194778) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1880 1285808 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1724 1285964 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1252 3605768 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2640 7211400 1% /mnt/lustre total: 25 open/close in 0.72 seconds: 34.88 ops/second total: 1 open/close in 0.06 seconds: 17.03 ops/second Failing mds1 on oleg450-server Stopping /mnt/lustre-mds1 (opts:) on oleg450-server reboot facets: mds1 Failover mds1 to oleg450-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg450-server: oleg450-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg450-client: oleg450-server: ssh exited with exit code 1 Started lustre-MDT0000 Failing mds1 on oleg450-server Stopping /mnt/lustre-mds1 (opts:) on oleg450-server reboot facets: mds1 Failover mds1 to oleg450-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg450-server: oleg450-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg450-client: oleg450-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec - unlinked 0 (time 1776194986 ; total 0 ; last 0) total: 25 unlinks in 1 seconds: 25.000000 unlinks/second Starting client: oleg450-client.virtnet: -o user_xattr,flock oleg450-server@tcp:/lustre /mnt/lustre2 PASS 16 (214s) == replay-dual test 17: fail OST during recovery (3571) == 15:29:52 (1776194992) total: 25 open/close in 0.67 seconds: 37.38 ops/second total: 1 open/close in 0.08 seconds: 12.56 ops/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1880 1285808 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1724 1285964 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1388 3605632 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1252 3605768 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2640 7211400 1% /mnt/lustre Failing ost1 on oleg450-server Stopping /mnt/lustre-ost1 (opts:) on oleg450-server reboot facets: ost1 Failover ost1 to oleg450-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg450-server: oleg450-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg450-client: oleg450-server: ssh exited with exit code 1 Started lustre-OST0000 Failing ost1 on oleg450-server Stopping /mnt/lustre-ost1 (opts:) on oleg450-server reboot facets: ost1 Failover ost1 to oleg450-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg450-server: oleg450-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg450-client: oleg450-server: ssh exited with exit code 1 Started lustre-OST0000 oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec - unlinked 0 (time 1776195194 ; total 0 ; last 0) total: 25 unlinks in 0 seconds: inf unlinks/second Starting client: oleg450-client.virtnet: -o user_xattr,flock oleg450-server@tcp:/lustre /mnt/lustre2 PASS 17 (207s) == replay-dual test 18: ldlm_handle_enqueue succeeds on evicted export (3822) ========================================================== 15:33:20 (1776195200) debug=+dlmtrace using seed 1654379210 running for 500 iterations total: 500 stats in 0 seconds: inf stats/second fail_loc=0x8000030b ldlm.namespaces.MGC192.168.204.150@tcp.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0000-mdc-ffff8fec0700b000.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0000-mdc-ffff8fec11446800.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0001-mdc-ffff8fec0700b000.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0001-mdc-ffff8fec11446800.early_lock_cancel=0 ldlm.namespaces.lustre-OST0000-osc-ffff8fec0700b000.early_lock_cancel=0 ldlm.namespaces.lustre-OST0000-osc-ffff8fec11446800.early_lock_cancel=0 ldlm.namespaces.lustre-OST0001-osc-ffff8fec0700b000.early_lock_cancel=0 ldlm.namespaces.lustre-OST0001-osc-ffff8fec11446800.early_lock_cancel=0 fail_loc=0x80000305 Error in opening file "/mnt/lustre2/d18.replay-dual/f18.replay-dual"(flags=O_RDONLY) 2: No such file or directory ldlm.namespaces.MGC192.168.204.150@tcp.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0000-mdc-ffff8fec0700b000.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0000-mdc-ffff8fec11446800.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0001-mdc-ffff8fec0700b000.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0001-mdc-ffff8fec11446800.early_lock_cancel=1 ldlm.namespaces.lustre-OST0000-osc-ffff8fec0700b000.early_lock_cancel=1 ldlm.namespaces.lustre-OST0000-osc-ffff8fec11446800.early_lock_cancel=1 ldlm.namespaces.lustre-OST0001-osc-ffff8fec0700b000.early_lock_cancel=1 ldlm.namespaces.lustre-OST0001-osc-ffff8fec11446800.early_lock_cancel=1 fail_loc=0 fail_loc=0 PASS 18 (52s) == replay-dual test 19: resend of open request =========== 15:34:12 (1776195252) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1884 1285804 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1724 1285964 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1404 3605616 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1252 3605768 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2656 7211384 1% /mnt/lustre fail_loc=0x157 - open/close 0 (time 1776195355.11 total 94.09 last 0.00) total: 1 open/close in 94.09 seconds: 0.01 ops/second fail_loc=0 Failing mds1 on oleg450-server Stopping /mnt/lustre-mds1 (opts:) on oleg450-server reboot facets: mds1 Failover mds1 to oleg450-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg450-server: oleg450-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg450-client: oleg450-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 19 (156s) == replay-dual test 20: recovery time is not increasing == 15:36:48 (1776195408) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1884 1285804 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1724 1285964 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1404 3605616 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1252 3605768 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2656 7211384 1% /mnt/lustre Failing mds1 on oleg450-server Stopping /mnt/lustre-mds1 (opts:) on oleg450-server reboot facets: mds1 Failover mds1 to oleg450-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg450-server: oleg450-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg450-client: oleg450-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Starting client: oleg450-client.virtnet: -o user_xattr,flock oleg450-server@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1884 1285804 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1724 1285964 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1404 3605616 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1252 3605768 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 2656 7211384 1% /mnt/lustre Failing mds1 on oleg450-server Stopping /mnt/lustre-mds1 (opts:) on oleg450-server reboot facets: mds1 Failover mds1 to oleg450-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg450-server: oleg450-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg450-client: oleg450-server: ssh exited with exit code 1 Started lustre-MDT0000