-----============= acceptance-small: replay-dual ============----- Thu Jan 22 04:40:58 EST 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg106-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/replay-dual.*ex': No such file or directory excepting tests: 14b 21b skipping tests SLOW=no: 21b === replay-dual: start setup 04:41:21 (1769074881) === Starting client oleg106-client.virtnet: -o user_xattr,flock 192.168.201.106@tcp:/lustre /mnt/lustre2 Mount client oleg106-client.virtnet: mount -t lustre -o user_xattr,flock 192.168.201.106@tcp:/lustre /mnt/lustre2 Started clients oleg106-client.virtnet: 192.168.201.106@tcp:/lustre on /mnt/lustre2 type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) oleg106-client.virtnet: executing check_config_client /mnt/lustre oleg106-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg106-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9cf7d076b800.idle_timeout=debug osc.lustre-OST0000-osc-ffff9cf7d8a35800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9cf7d076b800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9cf7d8a35800.idle_timeout=debug disable quota as required oleg106-server: oleg106-server.virtnet: executing set_default_debug -1 all osd-ldiskfs.track_declares_assert=1 === replay-dual: finish setup 04:42:03 (1769074923) === == replay-dual test 0a: expired recovery with lost client ========================================================== 04:42:05 (1769074925) Check file is LU482_FAILED=/tmp/replay-dual.lu482.FUJUko UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1708 1269596 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1532 1269772 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3088 7210952 1% /mnt/lustre total: 50 open/close in 2.65 seconds: 18.88 ops/second fail_loc=0x80000514 Failing mds1 on oleg106-server Stopping /mnt/lustre-mds1 (opts:) on oleg106-server 04:42:26 (1769074946) shut down facet: mds1 facet_host: oleg106-server facet_failover_host: oleg106-server Failover mds1 to oleg106-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg106-server: oleg106-server.virtnet: executing set_default_debug -1 all pdsh@oleg106-client: oleg106-server: ssh exited with exit code 1 Started lustre-MDT0000 04:42:57 (1769074977) targets are mounted 04:42:57 (1769074977) facet_failover done Starting client: oleg106-client.virtnet: -o user_xattr,flock 192.168.201.106@tcp:/lustre /mnt/lustre2 - unlinked 0 (time 1769075090 ; total 0 ; last 0) total: 50 unlinks in 2 seconds: 25.000000 unlinks/second PASS 0a (173s) == replay-dual test 0b: lost client during waiting for next transno ========================================================== 04:44:59 (1769075099) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1676 1269628 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1532 1269772 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3088 7210952 1% /mnt/lustre Failing mds1 on oleg106-server Stopping /mnt/lustre-mds1 (opts:) on oleg106-server 04:45:15 (1769075115) shut down facet: mds1 facet_host: oleg106-server facet_failover_host: oleg106-server Failover mds1 to oleg106-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg106-server: oleg106-server.virtnet: executing set_default_debug -1 all pdsh@oleg106-client: oleg106-server: ssh exited with exit code 1 Started lustre-MDT0000 04:45:47 (1769075147) targets are mounted 04:45:47 (1769075147) facet_failover done Starting client: oleg106-client.virtnet: -o user_xattr,flock 192.168.201.106@tcp:/lustre /mnt/lustre Starting client: oleg106-client.virtnet: -o user_xattr,flock 192.168.201.106@tcp:/lustre /mnt/lustre2 PASS 0b (224s) == replay-dual test 1: |X| simple create ================= 04:48:43 (1769075323) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1676 1269628 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1532 1269772 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3088 7210952 1% /mnt/lustre mcreate: cannot create `/mnt/lustre/fsa-oleg106-client.virtnet' with mode 0100644: File exists Failing mds1 on oleg106-server Stopping /mnt/lustre-mds1 (opts:) on oleg106-server 04:49:00 (1769075340) shut down facet: mds1 facet_host: oleg106-server facet_failover_host: oleg106-server Failover mds1 to oleg106-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg106-server: oleg106-server.virtnet: executing set_default_debug -1 all pdsh@oleg106-client: oleg106-server: ssh exited with exit code 1 Started lustre-MDT0000 04:49:36 (1769075376) targets are mounted 04:49:36 (1769075376) facet_failover done oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 1 (71s) == replay-dual test 2: |X| mkdir adir ==================== 04:49:54 (1769075394) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1676 1269628 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1532 1269772 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3088 7210952 1% /mnt/lustre Failing mds1 on oleg106-server Stopping /mnt/lustre-mds1 (opts:) on oleg106-server 04:50:11 (1769075411) shut down facet: mds1 facet_host: oleg106-server facet_failover_host: oleg106-server Failover mds1 to oleg106-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg106-server: oleg106-server.virtnet: executing set_default_debug -1 all pdsh@oleg106-client: oleg106-server: ssh exited with exit code 1 Started lustre-MDT0000 04:50:47 (1769075447) targets are mounted 04:50:47 (1769075447) facet_failover done oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 2 (71s) == replay-dual test 3: |X| mkdir adir, mkdir adir/bdir === 04:51:05 (1769075465) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1676 1269628 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1532 1269772 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3088 7210952 1% /mnt/lustre Failing mds1 on oleg106-server Stopping /mnt/lustre-mds1 (opts:) on oleg106-server 04:51:21 (1769075481) shut down facet: mds1 facet_host: oleg106-server facet_failover_host: oleg106-server Failover mds1 to oleg106-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg106-server: oleg106-server.virtnet: executing set_default_debug -1 all pdsh@oleg106-client: oleg106-server: ssh exited with exit code 1 Started lustre-MDT0000 04:51:59 (1769075519) targets are mounted 04:51:59 (1769075519) facet_failover done oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 3 (73s) == replay-dual test 4: |X| mkdir adir (-EEXIST), mkdir adir/bdir ========================================================== 04:52:18 (1769075538) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1852 1269452 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1704 1269600 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3088 7210952 1% /mnt/lustre mkdir: cannot create directory '/mnt/lustre/adir': File exists Failing mds1 on oleg106-server Stopping /mnt/lustre-mds1 (opts:) on oleg106-server 04:52:36 (1769075556) shut down facet: mds1 facet_host: oleg106-server facet_failover_host: oleg106-server Failover mds1 to oleg106-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg106-server: oleg106-server.virtnet: executing set_default_debug -1 all pdsh@oleg106-client: oleg106-server: ssh exited with exit code 1 Started lustre-MDT0000 04:53:13 (1769075593) targets are mounted 04:53:13 (1769075593) facet_failover done oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 4 (74s) == replay-dual test 5: open, unlink |X| close ============ 04:53:33 (1769075613) multiop /mnt/lustre2/a vo_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.8207 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1788 1269516 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1668 1269636 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3088 7210952 1% /mnt/lustre Failing mds1 on oleg106-server Stopping /mnt/lustre-mds1 (opts:) on oleg106-server 04:53:51 (1769075631) shut down facet: mds1 facet_host: oleg106-server facet_failover_host: oleg106-server Failover mds1 to oleg106-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg106-server: oleg106-server.virtnet: executing set_default_debug -1 all pdsh@oleg106-client: oleg106-server: ssh exited with exit code 1 Started lustre-MDT0000 04:54:25 (1769075665) targets are mounted 04:54:25 (1769075665) facet_failover done oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 5 (70s) == replay-dual test 6: open1, open2, unlink |X| close1 [fail mds1] close2 ========================================================== 04:54:42 (1769075682) multiop /mnt/lustre2/a vo_c TMPPIPE=/tmp/multiop_open_wait_pipe.8207 multiop /mnt/lustre/a vo_c TMPPIPE=/tmp/multiop_open_wait_pipe.8207 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1788 1269516 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1668 1269636 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3088 7210952 1% /mnt/lustre Failing mds1 on oleg106-server Stopping /mnt/lustre-mds1 (opts:) on oleg106-server 04:54:59 (1769075699) shut down facet: mds1 facet_host: oleg106-server facet_failover_host: oleg106-server Failover mds1 to oleg106-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg106-server: oleg106-server.virtnet: executing set_default_debug -1 all pdsh@oleg106-client: oleg106-server: ssh exited with exit code 1 Started lustre-MDT0000 04:55:33 (1769075733) targets are mounted 04:55:33 (1769075733) facet_failover done oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 6 (69s) == replay-dual test 8: replay of resent request ========== 04:55:51 (1769075751) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1788 1269516 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1668 1269636 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3088 7210952 1% /mnt/lustre fail_loc=0x119 fail_loc=0 Failing mds1 on oleg106-server Stopping /mnt/lustre-mds1 (opts:) on oleg106-server 04:56:28 (1769075788) shut down facet: mds1 facet_host: oleg106-server facet_failover_host: oleg106-server Failover mds1 to oleg106-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg106-server: oleg106-server.virtnet: executing set_default_debug -1 all pdsh@oleg106-client: oleg106-server: ssh exited with exit code 1 Started lustre-MDT0000 04:57:07 (1769075827) targets are mounted 04:57:07 (1769075827) facet_failover done oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 8 (93s) == replay-dual test 9: resending a replayed create ======= 04:57:24 (1769075844) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1788 1269516 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1668 1269636 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3088 7210952 1% /mnt/lustre fail_loc=0x80000119 Failing mds1 on oleg106-server Stopping /mnt/lustre-mds1 (opts:) on oleg106-server 04:57:42 (1769075862) shut down facet: mds1 facet_host: oleg106-server facet_failover_host: oleg106-server Failover mds1 to oleg106-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg106-server: oleg106-server.virtnet: executing set_default_debug -1 all pdsh@oleg106-client: oleg106-server: ssh exited with exit code 1 Started lustre-MDT0000 04:58:16 (1769075896) targets are mounted 04:58:16 (1769075896) facet_failover done oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec fail_loc=0 PASS 9 (76s) == replay-dual test 10: resending a replayed unlink ====== 04:58:40 (1769075920) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1788 1269516 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1668 1269636 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3088 7210952 1% /mnt/lustre fail_loc=0x80000119 Failing mds1 on oleg106-server Stopping /mnt/lustre-mds1 (opts:) on oleg106-server 04:58:56 (1769075936) shut down facet: mds1 facet_host: oleg106-server facet_failover_host: oleg106-server Failover mds1 to oleg106-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg106-server: oleg106-server.virtnet: executing set_default_debug -1 all pdsh@oleg106-client: oleg106-server: ssh exited with exit code 1 Started lustre-MDT0000 04:59:33 (1769075973) targets are mounted 04:59:34 (1769075974) facet_failover done oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec fail_loc=0 PASS 10 (82s) == replay-dual test 11: both clients timeout during replay ========================================================== 05:00:02 (1769076002) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1788 1269516 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1668 1269636 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3088 7210952 1% /mnt/lustre fail_loc=0x0119 Failing mds1 on oleg106-server Stopping /mnt/lustre-mds1 (opts:) on oleg106-server 05:00:21 (1769076021) shut down facet: mds1 facet_host: oleg106-server facet_failover_host: oleg106-server Failover mds1 to oleg106-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg106-server: oleg106-server.virtnet: executing set_default_debug -1 all pdsh@oleg106-client: oleg106-server: ssh exited with exit code 1 Started lustre-MDT0000 05:00:57 (1769076057) targets are mounted 05:00:57 (1769076057) facet_failover done oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 7 sec fail_loc=0 PASS 11 (75s) == replay-dual test 12: open resend timeout ============== 05:01:17 (1769076077) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1788 1269516 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1668 1269636 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3088 7210952 1% /mnt/lustre multiop /mnt/lustre/f12.replay-dual vmo_c TMPPIPE=/tmp/multiop_open_wait_pipe.8207 fail_loc=0x80000302 Failing mds1 on oleg106-server Stopping /mnt/lustre-mds1 (opts:) on oleg106-server 05:01:33 (1769076093) shut down facet: mds1 facet_host: oleg106-server facet_failover_host: oleg106-server Failover mds1 to oleg106-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg106-server: oleg106-server.virtnet: executing set_default_debug -1 all pdsh@oleg106-client: oleg106-server: ssh exited with exit code 1 Started lustre-MDT0000 05:02:06 (1769076126) targets are mounted 05:02:06 (1769076126) facet_failover done fail_loc=0 /mnt/lustre/f12.replay-dual /mnt/lustre/f12.replay-dual has type file OK PASS 12 (69s) == replay-dual test 13: close resend timeout ============= 05:02:27 (1769076147) multiop /mnt/lustre/f13.replay-dual vmo_c TMPPIPE=/tmp/multiop_open_wait_pipe.8207 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1788 1269516 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1668 1269636 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3088 7210952 1% /mnt/lustre fail_loc=0x80000115 Failing mds1 on oleg106-server Stopping /mnt/lustre-mds1 (opts:) on oleg106-server 05:02:43 (1769076163) shut down facet: mds1 facet_host: oleg106-server facet_failover_host: oleg106-server Failover mds1 to oleg106-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg106-server: oleg106-server.virtnet: executing set_default_debug -1 all pdsh@oleg106-client: oleg106-server: ssh exited with exit code 1 Started lustre-MDT0000 05:03:19 (1769076199) targets are mounted 05:03:19 (1769076199) facet_failover done fail_loc=0 /mnt/lustre/f13.replay-dual /mnt/lustre/f13.replay-dual has type file OK PASS 13 (73s) SKIP: replay-dual test_14b skipping ALWAYS excluded test 14b == replay-dual test 15a: timeout waiting for lost client during replay, 1 client completes ========================================================== 05:03:42 (1769076222) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1788 1269516 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1668 1269636 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3088 7210952 1% /mnt/lustre total: 25 open/close in 1.28 seconds: 19.49 ops/second total: 1 open/close in 0.06 seconds: 17.85 ops/second Failing mds1 on oleg106-server Stopping /mnt/lustre-mds1 (opts:) on oleg106-server 05:04:00 (1769076240) shut down facet: mds1 facet_host: oleg106-server facet_failover_host: oleg106-server Failover mds1 to oleg106-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg106-server: oleg106-server.virtnet: executing set_default_debug -1 all pdsh@oleg106-client: oleg106-server: ssh exited with exit code 1 Started lustre-MDT0000 05:04:36 (1769076276) targets are mounted 05:04:36 (1769076276) facet_failover done oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec - unlinked 0 (time 1769076349 ; total 0 ; last 0) total: 25 unlinks in 2 seconds: 12.500000 unlinks/second Starting client: oleg106-client.virtnet: -o user_xattr,flock 192.168.201.106@tcp:/lustre /mnt/lustre2 PASS 15a (136s) == replay-dual test 15c: remove multiple OST orphans ===== 05:05:59 (1769076359) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1788 1269516 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1668 1269636 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3088 7210952 1% /mnt/lustre Failing mds1 on oleg106-server Stopping /mnt/lustre-mds1 (opts:) on oleg106-server 05:08:22 (1769076502) shut down facet: mds1 facet_host: oleg106-server facet_failover_host: oleg106-server Failover mds1 to oleg106-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg106-server: oleg106-server.virtnet: executing set_default_debug -1 all pdsh@oleg106-client: oleg106-server: ssh exited with exit code 1 Started lustre-MDT0000 05:08:58 (1769076538) targets are mounted 05:08:58 (1769076538) facet_failover done oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Starting client: oleg106-client.virtnet: -o user_xattr,flock 192.168.201.106@tcp:/lustre /mnt/lustre2 PASS 15c (262s) == replay-dual test 16: fail MDS during recovery (3571) == 05:10:20 (1769076620) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1788 1269516 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1668 1269636 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3088 7210952 1% /mnt/lustre total: 25 open/close in 1.53 seconds: 16.39 ops/second total: 1 open/close in 0.04 seconds: 27.25 ops/second Failing mds1 on oleg106-server Stopping /mnt/lustre-mds1 (opts:) on oleg106-server 05:10:43 (1769076643) shut down facet: mds1 facet_host: oleg106-server facet_failover_host: oleg106-server Failover mds1 to oleg106-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg106-server: oleg106-server.virtnet: executing set_default_debug -1 all pdsh@oleg106-client: oleg106-server: ssh exited with exit code 1 Started lustre-MDT0000 05:11:18 (1769076678) targets are mounted 05:11:18 (1769076678) facet_failover done Failing mds1 on oleg106-server Stopping /mnt/lustre-mds1 (opts:) on oleg106-server 05:11:45 (1769076705) shut down facet: mds1 facet_host: oleg106-server facet_failover_host: oleg106-server Failover mds1 to oleg106-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg106-server: oleg106-server.virtnet: executing set_default_debug -1 all pdsh@oleg106-client: oleg106-server: ssh exited with exit code 1 Started lustre-MDT0000 05:12:22 (1769076742) targets are mounted 05:12:22 (1769076742) facet_failover done oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec - unlinked 0 (time 1769076817 ; total 0 ; last 0) total: 25 unlinks in 1 seconds: 25.000000 unlinks/second Starting client: oleg106-client.virtnet: -o user_xattr,flock 192.168.201.106@tcp:/lustre /mnt/lustre2 PASS 16 (205s) == replay-dual test 17: fail OST during recovery (3571) == 05:13:45 (1769076825) total: 25 open/close in 1.02 seconds: 24.39 ops/second total: 1 open/close in 0.06 seconds: 15.40 ops/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1788 1269516 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1668 1269636 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3088 7210952 1% /mnt/lustre Failing ost1 on oleg106-server Stopping /mnt/lustre-ost1 (opts:) on oleg106-server 05:14:04 (1769076844) shut down facet: ost1 facet_host: oleg106-server facet_failover_host: oleg106-server Failover ost1 to oleg106-server mount facets: ost1 Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg106-server: oleg106-server.virtnet: executing set_default_debug -1 all pdsh@oleg106-client: oleg106-server: ssh exited with exit code 1 Started lustre-OST0000 05:14:40 (1769076880) targets are mounted 05:14:40 (1769076880) facet_failover done Failing ost1 on oleg106-server Stopping /mnt/lustre-ost1 (opts:) on oleg106-server 05:15:07 (1769076907) shut down facet: ost1 facet_host: oleg106-server facet_failover_host: oleg106-server Failover ost1 to oleg106-server mount facets: ost1 Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg106-server: oleg106-server.virtnet: executing set_default_debug -1 all pdsh@oleg106-client: oleg106-server: ssh exited with exit code 1 Started lustre-OST0000 05:15:37 (1769076937) targets are mounted 05:15:37 (1769076937) facet_failover done oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec - unlinked 0 (time 1769077009 ; total 0 ; last 0) total: 25 unlinks in 0 seconds: inf unlinks/second Starting client: oleg106-client.virtnet: -o user_xattr,flock 192.168.201.106@tcp:/lustre /mnt/lustre2 PASS 17 (192s) == replay-dual test 18: ldlm_handle_enqueue succeeds on evicted export (3822) ========================================================== 05:16:57 (1769077017) debug=+dlmtrace fail_loc=0x8000030b using seed 1545518232 running for 500 iterations total: 500 stats in 2 seconds: 250.000000 stats/second ldlm.namespaces.MGC192.168.201.106@tcp.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0000-mdc-ffff9cf7c351a800.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0000-mdc-ffff9cf7e075b000.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0001-mdc-ffff9cf7c351a800.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0001-mdc-ffff9cf7e075b000.early_lock_cancel=0 ldlm.namespaces.lustre-OST0000-osc-ffff9cf7c351a800.early_lock_cancel=0 ldlm.namespaces.lustre-OST0000-osc-ffff9cf7e075b000.early_lock_cancel=0 ldlm.namespaces.lustre-OST0001-osc-ffff9cf7c351a800.early_lock_cancel=0 ldlm.namespaces.lustre-OST0001-osc-ffff9cf7e075b000.early_lock_cancel=0 fail_loc=0x80000305 Error in opening file "/mnt/lustre2/d18.replay-dual/f18.replay-dual"(flags=O_RDONLY) 2: No such file or directory ldlm.namespaces.MGC192.168.201.106@tcp.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0000-mdc-ffff9cf7c351a800.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0000-mdc-ffff9cf7e075b000.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0001-mdc-ffff9cf7c351a800.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0001-mdc-ffff9cf7e075b000.early_lock_cancel=1 ldlm.namespaces.lustre-OST0000-osc-ffff9cf7c351a800.early_lock_cancel=1 ldlm.namespaces.lustre-OST0000-osc-ffff9cf7e075b000.early_lock_cancel=1 ldlm.namespaces.lustre-OST0001-osc-ffff9cf7c351a800.early_lock_cancel=1 ldlm.namespaces.lustre-OST0001-osc-ffff9cf7e075b000.early_lock_cancel=1 fail_loc=0 fail_loc=0 PASS 18 (64s) == replay-dual test 19: resend of open request =========== 05:18:01 (1769077081) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1792 1269512 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1668 1269636 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1560 3605460 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3104 7210936 1% /mnt/lustre fail_loc=0x157 - open/close 0 (time 1769077181.11 total 85.66 last 0.00) total: 1 open/close in 85.66 seconds: 0.01 ops/second fail_loc=0 Failing mds1 on oleg106-server Stopping /mnt/lustre-mds1 (opts:) on oleg106-server 05:19:48 (1769077188) shut down facet: mds1 facet_host: oleg106-server facet_failover_host: oleg106-server Failover mds1 to oleg106-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg106-server: oleg106-server.virtnet: executing set_default_debug -1 all pdsh@oleg106-client: oleg106-server: ssh exited with exit code 1 Started lustre-MDT0000 05:20:25 (1769077225) targets are mounted 05:20:25 (1769077225) facet_failover done oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 19 (161s) == replay-dual test 20: recovery time is not increasing == 05:20:42 (1769077242) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1792 1269512 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1668 1269636 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1560 3605460 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3104 7210936 1% /mnt/lustre Failing mds1 on oleg106-server Stopping /mnt/lustre-mds1 (opts:) on oleg106-server 05:21:00 (1769077260) shut down facet: mds1 facet_host: oleg106-server facet_failover_host: oleg106-server Failover mds1 to oleg106-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg106-server: oleg106-server.virtnet: executing set_default_debug -1 all pdsh@oleg106-client: oleg106-server: ssh exited with exit code 1 Started lustre-MDT0000 05:21:39 (1769077299) targets are mounted 05:21:39 (1769077299) facet_failover done oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Starting client: oleg106-client.virtnet: -o user_xattr,flock 192.168.201.106@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1792 1269512 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1668 1269636 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1560 3605460 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3104 7210936 1% /mnt/lustre Failing mds1 on oleg106-server Stopping /mnt/lustre-mds1 (opts:) on oleg106-server 05:24:20 (1769077460) shut down facet: mds1 facet_host: oleg106-server facet_failover_host: oleg106-server Failover mds1 to oleg106-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg106-server: oleg106-server.virtnet: executing set_default_debug -1 all pdsh@oleg106-client: oleg106-server: ssh exited with exit code 1 Started lustre-MDT0000 05:24:54 (1769077494) targets are mounted 05:24:54 (1769077494) facet_failover done oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Starting client: oleg106-client.virtnet: -o user_xattr,flock 192.168.201.106@tcp:/lustre /mnt/lustre2 PASS 20 (404s)