-----============= acceptance-small: replay-dual ============----- Fri Sep 4 10:58:49 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg655-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/replay-dual.*ex': No such file or directory excepting tests: 14b 21b skipping tests SLOW=no: 21b === replay-dual: start setup 10:59:14 (1788533954) === Starting client oleg655-client.virtnet: -o user_xattr,flock 192.168.206.155@tcp:/lustre /mnt/lustre2 Mount client oleg655-client.virtnet: mount -t lustre -o user_xattr,flock 192.168.206.155@tcp:/lustre /mnt/lustre2 Started clients oleg655-client.virtnet: 192.168.206.155@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) oleg655-client.virtnet: executing check_config_client /mnt/lustre oleg655-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg655-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9fc8c9270800.idle_timeout=debug osc.lustre-OST0000-osc-ffff9fc8d0a15800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9fc8c9270800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9fc8d0a15800.idle_timeout=debug disable quota as required oleg655-server: oleg655-server.virtnet: executing set_default_debug -1 all osd-ldiskfs.track_declares_assert=1 === replay-dual: finish setup 10:59:57 (1788533997) === == replay-dual test 0a: expired recovery with lost client ========================================================== 10:59:59 (1788533999) Check file is LU482_FAILED=/tmp/replay-dual.lu482.FkC5OH UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1708 1269596 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1532 1269772 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3088 7210952 1% /mnt/lustre total: 50 open/close in 2.15 seconds: 23.23 ops/second fail_loc=0x80000514 Failing mds1 on oleg655-server Stopping /mnt/lustre-mds1 (opts:) on oleg655-server 11:00:16 (1788534016) shut down facet: mds1 facet_host: oleg655-server facet_failover_host: oleg655-server Failover mds1 to oleg655-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg655-server: oleg655-server.virtnet: executing set_default_debug -1 all pdsh@oleg655-client: oleg655-server: ssh exited with exit code 1 Started lustre-MDT0000 11:00:53 (1788534053) targets are mounted 11:00:53 (1788534053) facet_failover done Starting client: oleg655-client.virtnet: -o user_xattr,flock 192.168.206.155@tcp:/lustre /mnt/lustre2 - unlinked 0 (time 1788534167 ; total 0 ; last 0) total: 50 unlinks in 2 seconds: 25.000000 unlinks/second PASS 0a (176s) == replay-dual test 0b: lost client during waiting for next transno ========================================================== 11:02:56 (1788534176) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1676 1269628 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1532 1269772 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3088 7210952 1% /mnt/lustre Failing mds1 on oleg655-server Stopping /mnt/lustre-mds1 (opts:) on oleg655-server 11:03:10 (1788534190) shut down facet: mds1 facet_host: oleg655-server facet_failover_host: oleg655-server Failover mds1 to oleg655-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg655-server: oleg655-server.virtnet: executing set_default_debug -1 all pdsh@oleg655-client: oleg655-server: ssh exited with exit code 1 Started lustre-MDT0000 11:03:42 (1788534222) targets are mounted 11:03:42 (1788534222) facet_failover done Starting client: oleg655-client.virtnet: -o user_xattr,flock 192.168.206.155@tcp:/lustre /mnt/lustre Starting client: oleg655-client.virtnet: -o user_xattr,flock 192.168.206.155@tcp:/lustre /mnt/lustre2 PASS 0b (222s) == replay-dual test 1: |X| simple create ================= 11:06:38 (1788534398) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1676 1269628 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1532 1269772 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3088 7210952 1% /mnt/lustre mcreate: cannot create `/mnt/lustre/fsa-oleg655-client.virtnet' with mode 0100644: File exists Failing mds1 on oleg655-server Stopping /mnt/lustre-mds1 (opts:) on oleg655-server 11:06:51 (1788534411) shut down facet: mds1 facet_host: oleg655-server facet_failover_host: oleg655-server Failover mds1 to oleg655-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg655-server: oleg655-server.virtnet: executing set_default_debug -1 all pdsh@oleg655-client: oleg655-server: ssh exited with exit code 1 Started lustre-MDT0000 11:07:30 (1788534450) targets are mounted 11:07:30 (1788534450) facet_failover done oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 1 (69s) == replay-dual test 2: |X| mkdir adir ==================== 11:07:47 (1788534467) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1676 1269628 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1532 1269772 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3088 7210952 1% /mnt/lustre Failing mds1 on oleg655-server Stopping /mnt/lustre-mds1 (opts:) on oleg655-server 11:08:03 (1788534483) shut down facet: mds1 facet_host: oleg655-server facet_failover_host: oleg655-server Failover mds1 to oleg655-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg655-server: oleg655-server.virtnet: executing set_default_debug -1 all pdsh@oleg655-client: oleg655-server: ssh exited with exit code 1 Started lustre-MDT0000 11:08:36 (1788534516) targets are mounted 11:08:36 (1788534516) facet_failover done oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 2 (65s) == replay-dual test 3: |X| mkdir adir, mkdir adir/bdir === 11:08:52 (1788534532) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1676 1269628 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1532 1269772 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3088 7210952 1% /mnt/lustre Failing mds1 on oleg655-server Stopping /mnt/lustre-mds1 (opts:) on oleg655-server 11:09:07 (1788534547) shut down facet: mds1 facet_host: oleg655-server facet_failover_host: oleg655-server Failover mds1 to oleg655-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg655-server: oleg655-server.virtnet: executing set_default_debug -1 all pdsh@oleg655-client: oleg655-server: ssh exited with exit code 1 Started lustre-MDT0000 11:09:39 (1788534579) targets are mounted 11:09:39 (1788534579) facet_failover done oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 3 (65s) == replay-dual test 4: |X| mkdir adir (-EEXIST), mkdir adir/bdir ========================================================== 11:09:57 (1788534597) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1852 1269452 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1740 1269564 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3088 7210952 1% /mnt/lustre mkdir: cannot create directory '/mnt/lustre/adir': File exists Failing mds1 on oleg655-server Stopping /mnt/lustre-mds1 (opts:) on oleg655-server 11:10:13 (1788534613) shut down facet: mds1 facet_host: oleg655-server facet_failover_host: oleg655-server Failover mds1 to oleg655-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg655-server: oleg655-server.virtnet: executing set_default_debug -1 all pdsh@oleg655-client: oleg655-server: ssh exited with exit code 1 Started lustre-MDT0000 11:10:48 (1788534648) targets are mounted 11:10:48 (1788534648) facet_failover done oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 4 (66s) == replay-dual test 5: open, unlink |X| close ============ 11:11:04 (1788534664) multiop /mnt/lustre2/a vo_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.8223 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1788 1269516 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1668 1269636 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3088 7210952 1% /mnt/lustre Failing mds1 on oleg655-server Stopping /mnt/lustre-mds1 (opts:) on oleg655-server 11:11:17 (1788534677) shut down facet: mds1 facet_host: oleg655-server facet_failover_host: oleg655-server Failover mds1 to oleg655-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg655-server: oleg655-server.virtnet: executing set_default_debug -1 all pdsh@oleg655-client: oleg655-server: ssh exited with exit code 1 Started lustre-MDT0000 11:11:51 (1788534711) targets are mounted 11:11:51 (1788534711) facet_failover done oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 5 (65s) == replay-dual test 6: open1, open2, unlink |X| close1 [fail mds1] close2 ========================================================== 11:12:09 (1788534729) multiop /mnt/lustre2/a vo_c TMPPIPE=/tmp/multiop_open_wait_pipe.8223 multiop /mnt/lustre/a vo_c TMPPIPE=/tmp/multiop_open_wait_pipe.8223 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1788 1269516 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1668 1269636 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3088 7210952 1% /mnt/lustre Failing mds1 on oleg655-server Stopping /mnt/lustre-mds1 (opts:) on oleg655-server 11:12:26 (1788534746) shut down facet: mds1 facet_host: oleg655-server facet_failover_host: oleg655-server Failover mds1 to oleg655-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg655-server: oleg655-server.virtnet: executing set_default_debug -1 all pdsh@oleg655-client: oleg655-server: ssh exited with exit code 1 Started lustre-MDT0000 11:12:58 (1788534778) targets are mounted 11:12:58 (1788534778) facet_failover done oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 6 (69s) == replay-dual test 8: replay of resent request ========== 11:13:18 (1788534798) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1788 1269516 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1668 1269636 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3088 7210952 1% /mnt/lustre fail_loc=0x119 fail_loc=0 Failing mds1 on oleg655-server Stopping /mnt/lustre-mds1 (opts:) on oleg655-server 11:13:49 (1788534829) shut down facet: mds1 facet_host: oleg655-server facet_failover_host: oleg655-server Failover mds1 to oleg655-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg655-server: oleg655-server.virtnet: executing set_default_debug -1 all pdsh@oleg655-client: oleg655-server: ssh exited with exit code 1 Started lustre-MDT0000 11:14:27 (1788534867) targets are mounted 11:14:27 (1788534867) facet_failover done oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 8 (84s) == replay-dual test 9: resending a replayed create ======= 11:14:43 (1788534883) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1788 1269516 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1668 1269636 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3088 7210952 1% /mnt/lustre fail_loc=0x80000119 Failing mds1 on oleg655-server Stopping /mnt/lustre-mds1 (opts:) on oleg655-server 11:14:59 (1788534899) shut down facet: mds1 facet_host: oleg655-server facet_failover_host: oleg655-server Failover mds1 to oleg655-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg655-server: oleg655-server.virtnet: executing set_default_debug -1 all pdsh@oleg655-client: oleg655-server: ssh exited with exit code 1 Started lustre-MDT0000 11:15:34 (1788534934) targets are mounted 11:15:34 (1788534934) facet_failover done oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec fail_loc=0 PASS 9 (80s) == replay-dual test 10: resending a replayed unlink ====== 11:16:02 (1788534962) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1788 1269516 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1668 1269636 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3088 7210952 1% /mnt/lustre fail_loc=0x80000119 Failing mds1 on oleg655-server Stopping /mnt/lustre-mds1 (opts:) on oleg655-server 11:16:18 (1788534978) shut down facet: mds1 facet_host: oleg655-server facet_failover_host: oleg655-server Failover mds1 to oleg655-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg655-server: oleg655-server.virtnet: executing set_default_debug -1 all pdsh@oleg655-client: oleg655-server: ssh exited with exit code 1 Started lustre-MDT0000 11:16:53 (1788535013) targets are mounted 11:16:53 (1788535013) facet_failover done oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec fail_loc=0 PASS 10 (79s) == replay-dual test 11: both clients timeout during replay ========================================================== 11:17:21 (1788535041) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1788 1269516 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1668 1269636 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3088 7210952 1% /mnt/lustre fail_loc=0x0119 Failing mds1 on oleg655-server Stopping /mnt/lustre-mds1 (opts:) on oleg655-server 11:17:39 (1788535059) shut down facet: mds1 facet_host: oleg655-server facet_failover_host: oleg655-server Failover mds1 to oleg655-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg655-server: oleg655-server.virtnet: executing set_default_debug -1 all pdsh@oleg655-client: oleg655-server: ssh exited with exit code 1 Started lustre-MDT0000 11:18:16 (1788535096) targets are mounted 11:18:16 (1788535096) facet_failover done oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec fail_loc=0 PASS 11 (73s) == replay-dual test 12: open resend timeout ============== 11:18:35 (1788535115) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1788 1269516 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1668 1269636 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3088 7210952 1% /mnt/lustre multiop /mnt/lustre/f12.replay-dual vmo_c TMPPIPE=/tmp/multiop_open_wait_pipe.8223 fail_loc=0x80000302 Failing mds1 on oleg655-server Stopping /mnt/lustre-mds1 (opts:) on oleg655-server 11:18:51 (1788535131) shut down facet: mds1 facet_host: oleg655-server facet_failover_host: oleg655-server Failover mds1 to oleg655-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg655-server: oleg655-server.virtnet: executing set_default_debug -1 all pdsh@oleg655-client: oleg655-server: ssh exited with exit code 1 Started lustre-MDT0000 11:19:23 (1788535163) targets are mounted 11:19:23 (1788535163) facet_failover done fail_loc=0 /mnt/lustre/f12.replay-dual /mnt/lustre/f12.replay-dual has type file OK PASS 12 (69s) == replay-dual test 13: close resend timeout ============= 11:19:44 (1788535184) multiop /mnt/lustre/f13.replay-dual vmo_c TMPPIPE=/tmp/multiop_open_wait_pipe.8223 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1788 1269516 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1668 1269636 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3088 7210952 1% /mnt/lustre fail_loc=0x80000115 Failing mds1 on oleg655-server Stopping /mnt/lustre-mds1 (opts:) on oleg655-server 11:20:01 (1788535201) shut down facet: mds1 facet_host: oleg655-server facet_failover_host: oleg655-server Failover mds1 to oleg655-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg655-server: oleg655-server.virtnet: executing set_default_debug -1 all pdsh@oleg655-client: oleg655-server: ssh exited with exit code 1 Started lustre-MDT0000 11:20:36 (1788535236) targets are mounted 11:20:36 (1788535236) facet_failover done fail_loc=0 /mnt/lustre/f13.replay-dual /mnt/lustre/f13.replay-dual has type file OK PASS 13 (72s) SKIP: replay-dual test_14b skipping ALWAYS excluded test 14b == replay-dual test 15a: timeout waiting for lost client during replay, 1 client completes ========================================================== 11:20:58 (1788535258) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1788 1269516 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1668 1269636 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3088 7210952 1% /mnt/lustre total: 25 open/close in 0.76 seconds: 33.06 ops/second total: 1 open/close in 0.03 seconds: 35.58 ops/second Failing mds1 on oleg655-server Stopping /mnt/lustre-mds1 (opts:) on oleg655-server 11:21:14 (1788535274) shut down facet: mds1 facet_host: oleg655-server facet_failover_host: oleg655-server Failover mds1 to oleg655-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg655-server: oleg655-server.virtnet: executing set_default_debug -1 all pdsh@oleg655-client: oleg655-server: ssh exited with exit code 1 Started lustre-MDT0000 11:21:48 (1788535308) targets are mounted 11:21:48 (1788535308) facet_failover done oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec - unlinked 0 (time 1788535385 ; total 1 ; last 1) total: 25 unlinks in 2 seconds: 12.500000 unlinks/second Starting client: oleg655-client.virtnet: -o user_xattr,flock 192.168.206.155@tcp:/lustre /mnt/lustre2 PASS 15a (134s) == replay-dual test 15c: remove multiple OST orphans ===== 11:23:12 (1788535392) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1788 1269516 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1668 1269636 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3088 7210952 1% /mnt/lustre Failing mds1 on oleg655-server Stopping /mnt/lustre-mds1 (opts:) on oleg655-server 11:25:10 (1788535510) shut down facet: mds1 facet_host: oleg655-server facet_failover_host: oleg655-server Failover mds1 to oleg655-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg655-server: oleg655-server.virtnet: executing set_default_debug -1 all pdsh@oleg655-client: oleg655-server: ssh exited with exit code 1 Started lustre-MDT0000 11:25:45 (1788535545) targets are mounted 11:25:45 (1788535545) facet_failover done oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Starting client: oleg655-client.virtnet: -o user_xattr,flock 192.168.206.155@tcp:/lustre /mnt/lustre2 PASS 15c (232s) == replay-dual test 16: fail MDS during recovery (3571) == 11:27:04 (1788535624) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1788 1269516 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1668 1269636 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3088 7210952 1% /mnt/lustre total: 25 open/close in 0.97 seconds: 25.81 ops/second total: 1 open/close in 0.07 seconds: 14.61 ops/second Failing mds1 on oleg655-server Stopping /mnt/lustre-mds1 (opts:) on oleg655-server 11:27:19 (1788535639) shut down facet: mds1 facet_host: oleg655-server facet_failover_host: oleg655-server Failover mds1 to oleg655-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg655-server: oleg655-server.virtnet: executing set_default_debug -1 all pdsh@oleg655-client: oleg655-server: ssh exited with exit code 1 Started lustre-MDT0000 11:27:54 (1788535674) targets are mounted 11:27:54 (1788535674) facet_failover done Failing mds1 on oleg655-server Stopping /mnt/lustre-mds1 (opts:) on oleg655-server 11:28:20 (1788535700) shut down facet: mds1 facet_host: oleg655-server facet_failover_host: oleg655-server Failover mds1 to oleg655-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg655-server: oleg655-server.virtnet: executing set_default_debug -1 all pdsh@oleg655-client: oleg655-server: ssh exited with exit code 1 Started lustre-MDT0000 11:28:57 (1788535737) targets are mounted 11:28:57 (1788535737) facet_failover done oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec - unlinked 0 (time 1788535811 ; total 0 ; last 0) total: 25 unlinks in 1 seconds: 25.000000 unlinks/second Starting client: oleg655-client.virtnet: -o user_xattr,flock 192.168.206.155@tcp:/lustre /mnt/lustre2 PASS 16 (194s) == replay-dual test 17: fail OST during recovery (3571) == 11:30:18 (1788535818) total: 25 open/close in 0.75 seconds: 33.19 ops/second total: 1 open/close in 0.06 seconds: 16.99 ops/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1788 1269516 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1668 1269636 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3088 7210952 1% /mnt/lustre Failing ost1 on oleg655-server Stopping /mnt/lustre-ost1 (opts:) on oleg655-server 11:30:35 (1788535835) shut down facet: ost1 facet_host: oleg655-server facet_failover_host: oleg655-server Failover ost1 to oleg655-server mount facets: ost1 Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg655-server: oleg655-server.virtnet: executing set_default_debug -1 all pdsh@oleg655-client: oleg655-server: ssh exited with exit code 1 Started lustre-OST0000 11:31:08 (1788535868) targets are mounted 11:31:08 (1788535868) facet_failover done Failing ost1 on oleg655-server Stopping /mnt/lustre-ost1 (opts:) on oleg655-server 11:31:34 (1788535894) shut down facet: ost1 facet_host: oleg655-server facet_failover_host: oleg655-server Failover ost1 to oleg655-server mount facets: ost1 Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg655-server: oleg655-server.virtnet: executing set_default_debug -1 all pdsh@oleg655-client: oleg655-server: ssh exited with exit code 1 Started lustre-OST0000 11:31:59 (1788535919) targets are mounted 11:31:59 (1788535919) facet_failover done oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec - unlinked 0 (time 1788535992 ; total 0 ; last 0) total: 25 unlinks in 1 seconds: 25.000000 unlinks/second Starting client: oleg655-client.virtnet: -o user_xattr,flock 192.168.206.155@tcp:/lustre /mnt/lustre2 PASS 17 (181s) == replay-dual test 18: ldlm_handle_enqueue succeeds on evicted export (3822) ========================================================== 11:33:19 (1788535999) debug=+dlmtrace fail_loc=0x8000030b using seed 1167119333 running for 500 iterations total: 500 stats in 2 seconds: 250.000000 stats/second ldlm.namespaces.MGC192.168.206.155@tcp.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0000-mdc-ffff9fc8c405d800.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0000-mdc-ffff9fc8d12f3800.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0001-mdc-ffff9fc8c405d800.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0001-mdc-ffff9fc8d12f3800.early_lock_cancel=0 ldlm.namespaces.lustre-OST0000-osc-ffff9fc8c405d800.early_lock_cancel=0 ldlm.namespaces.lustre-OST0000-osc-ffff9fc8d12f3800.early_lock_cancel=0 ldlm.namespaces.lustre-OST0001-osc-ffff9fc8c405d800.early_lock_cancel=0 ldlm.namespaces.lustre-OST0001-osc-ffff9fc8d12f3800.early_lock_cancel=0 fail_loc=0x80000305 Error in opening file "/mnt/lustre2/d18.replay-dual/f18.replay-dual"(flags=O_RDONLY) 2: No such file or directory ldlm.namespaces.MGC192.168.206.155@tcp.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0000-mdc-ffff9fc8c405d800.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0000-mdc-ffff9fc8d12f3800.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0001-mdc-ffff9fc8c405d800.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0001-mdc-ffff9fc8d12f3800.early_lock_cancel=1 ldlm.namespaces.lustre-OST0000-osc-ffff9fc8c405d800.early_lock_cancel=1 ldlm.namespaces.lustre-OST0000-osc-ffff9fc8d12f3800.early_lock_cancel=1 ldlm.namespaces.lustre-OST0001-osc-ffff9fc8c405d800.early_lock_cancel=1 ldlm.namespaces.lustre-OST0001-osc-ffff9fc8d12f3800.early_lock_cancel=1 fail_loc=0 fail_loc=0 PASS 18 (59s) == replay-dual test 19: resend of open request =========== 11:34:18 (1788536058) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1792 1269512 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1668 1269636 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1560 3605460 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3104 7210936 1% /mnt/lustre fail_loc=0x157 - open/close 0 (time 1788536156.49 total 87.37 last 0.00) total: 1 open/close in 87.37 seconds: 0.01 ops/second fail_loc=0 Failing mds1 on oleg655-server Stopping /mnt/lustre-mds1 (opts:) on oleg655-server 11:36:02 (1788536162) shut down facet: mds1 facet_host: oleg655-server facet_failover_host: oleg655-server Failover mds1 to oleg655-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg655-server: oleg655-server.virtnet: executing set_default_debug -1 all pdsh@oleg655-client: oleg655-server: ssh exited with exit code 1 Started lustre-MDT0000 11:36:37 (1788536197) targets are mounted 11:36:37 (1788536197) facet_failover done oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 19 (155s) == replay-dual test 20: recovery time is not increasing == 11:36:54 (1788536214) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1792 1269512 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1668 1269636 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1560 3605460 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3104 7210936 1% /mnt/lustre Failing mds1 on oleg655-server Stopping /mnt/lustre-mds1 (opts:) on oleg655-server 11:37:10 (1788536230) shut down facet: mds1 facet_host: oleg655-server facet_failover_host: oleg655-server Failover mds1 to oleg655-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg655-server: oleg655-server.virtnet: executing set_default_debug -1 all pdsh@oleg655-client: oleg655-server: ssh exited with exit code 1 Started lustre-MDT0000 11:37:45 (1788536265) targets are mounted 11:37:45 (1788536265) facet_failover done oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Starting client: oleg655-client.virtnet: -o user_xattr,flock 192.168.206.155@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1792 1269512 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1668 1269636 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1560 3605460 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3104 7210936 1% /mnt/lustre Failing mds1 on oleg655-server Stopping /mnt/lustre-mds1 (opts:) on oleg655-server 11:40:31 (1788536431) shut down facet: mds1 facet_host: oleg655-server facet_failover_host: oleg655-server Failover mds1 to oleg655-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg655-server: oleg655-server.virtnet: executing set_default_debug -1 all pdsh@oleg655-client: oleg655-server: ssh exited with exit code 1 Started lustre-MDT0000 11:41:05 (1788536465) targets are mounted 11:41:05 (1788536465) facet_failover done oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Starting client: oleg655-client.virtnet: -o user_xattr,flock 192.168.206.155@tcp:/lustre /mnt/lustre2