-----============= acceptance-small: replay-dual ============----- Tue Sep 8 07:22:55 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg454-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/replay-dual.*ex': No such file or directory excepting tests: 14b 21b skipping tests SLOW=no: 21b === replay-dual: start setup 07:23:20 (1788866600) === Starting client oleg454-client.virtnet: -o user_xattr,flock 192.168.204.154@tcp:/lustre /mnt/lustre2 Mount client oleg454-client.virtnet: mount -t lustre -o user_xattr,flock 192.168.204.154@tcp:/lustre /mnt/lustre2 Started clients oleg454-client.virtnet: 192.168.204.154@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) oleg454-client.virtnet: executing check_config_client /mnt/lustre oleg454-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg454-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff96d083779800.idle_timeout=debug osc.lustre-OST0000-osc-ffff96d099be5000.idle_timeout=debug osc.lustre-OST0001-osc-ffff96d083779800.idle_timeout=debug osc.lustre-OST0001-osc-ffff96d099be5000.idle_timeout=debug disable quota as required oleg454-server: oleg454-server.virtnet: executing set_default_debug -1 all osd-ldiskfs.track_declares_assert=1 === replay-dual: finish setup 07:24:08 (1788866648) === == replay-dual test 0a: expired recovery with lost client ========================================================== 07:24:10 (1788866650) Check file is LU482_FAILED=/tmp/replay-dual.lu482.pxnNPY UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1708 1269596 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1532 1269772 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3088 7210952 1% /mnt/lustre total: 50 open/close in 3.14 seconds: 15.94 ops/second fail_loc=0x80000514 Failing mds1 on oleg454-server Stopping /mnt/lustre-mds1 (opts:) on oleg454-server 07:24:32 (1788866672) shut down facet: mds1 facet_host: oleg454-server facet_failover_host: oleg454-server Failover mds1 to oleg454-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg454-server: oleg454-server.virtnet: executing set_default_debug -1 all pdsh@oleg454-client: oleg454-server: ssh exited with exit code 1 Started lustre-MDT0000 07:25:08 (1788866708) targets are mounted 07:25:08 (1788866708) facet_failover done Starting client: oleg454-client.virtnet: -o user_xattr,flock 192.168.204.154@tcp:/lustre /mnt/lustre2 - unlinked 0 (time 1788866821 ; total 0 ; last 0) total: 50 unlinks in 2 seconds: 25.000000 unlinks/second PASS 0a (181s) == replay-dual test 0b: lost client during waiting for next transno ========================================================== 07:27:12 (1788866832) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1676 1269628 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1532 1269772 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3088 7210952 1% /mnt/lustre Failing mds1 on oleg454-server Stopping /mnt/lustre-mds1 (opts:) on oleg454-server 07:27:28 (1788866848) shut down facet: mds1 facet_host: oleg454-server facet_failover_host: oleg454-server Failover mds1 to oleg454-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg454-server: oleg454-server.virtnet: executing set_default_debug -1 all pdsh@oleg454-client: oleg454-server: ssh exited with exit code 1 Started lustre-MDT0000 07:28:03 (1788866883) targets are mounted 07:28:03 (1788866883) facet_failover done Starting client: oleg454-client.virtnet: -o user_xattr,flock 192.168.204.154@tcp:/lustre /mnt/lustre Starting client: oleg454-client.virtnet: -o user_xattr,flock 192.168.204.154@tcp:/lustre /mnt/lustre2 PASS 0b (226s) == replay-dual test 1: |X| simple create ================= 07:30:57 (1788867057) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1676 1269628 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1532 1269772 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3088 7210952 1% /mnt/lustre mcreate: cannot create `/mnt/lustre/fsa-oleg454-client.virtnet' with mode 0100644: File exists Failing mds1 on oleg454-server Stopping /mnt/lustre-mds1 (opts:) on oleg454-server 07:31:13 (1788867073) shut down facet: mds1 facet_host: oleg454-server facet_failover_host: oleg454-server Failover mds1 to oleg454-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg454-server: oleg454-server.virtnet: executing set_default_debug -1 all pdsh@oleg454-client: oleg454-server: ssh exited with exit code 1 Started lustre-MDT0000 07:31:49 (1788867109) targets are mounted 07:31:49 (1788867109) facet_failover done oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 1 (72s) == replay-dual test 2: |X| mkdir adir ==================== 07:32:09 (1788867129) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1676 1269628 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1532 1269772 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3088 7210952 1% /mnt/lustre Failing mds1 on oleg454-server Stopping /mnt/lustre-mds1 (opts:) on oleg454-server 07:32:24 (1788867144) shut down facet: mds1 facet_host: oleg454-server facet_failover_host: oleg454-server Failover mds1 to oleg454-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg454-server: oleg454-server.virtnet: executing set_default_debug -1 all pdsh@oleg454-client: oleg454-server: ssh exited with exit code 1 Started lustre-MDT0000 07:32:58 (1788867178) targets are mounted 07:32:58 (1788867178) facet_failover done oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 2 (67s) == replay-dual test 3: |X| mkdir adir, mkdir adir/bdir === 07:33:16 (1788867196) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1676 1269628 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1532 1269772 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3088 7210952 1% /mnt/lustre Failing mds1 on oleg454-server Stopping /mnt/lustre-mds1 (opts:) on oleg454-server 07:33:30 (1788867210) shut down facet: mds1 facet_host: oleg454-server facet_failover_host: oleg454-server Failover mds1 to oleg454-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg454-server: oleg454-server.virtnet: executing set_default_debug -1 all pdsh@oleg454-client: oleg454-server: ssh exited with exit code 1 Started lustre-MDT0000 07:34:06 (1788867246) targets are mounted 07:34:06 (1788867246) facet_failover done oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 3 (71s) == replay-dual test 4: |X| mkdir adir (-EEXIST), mkdir adir/bdir ========================================================== 07:34:27 (1788867267) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1852 1269452 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1740 1269564 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3088 7210952 1% /mnt/lustre mkdir: cannot create directory '/mnt/lustre/adir': File exists Failing mds1 on oleg454-server Stopping /mnt/lustre-mds1 (opts:) on oleg454-server 07:34:43 (1788867283) shut down facet: mds1 facet_host: oleg454-server facet_failover_host: oleg454-server Failover mds1 to oleg454-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg454-server: oleg454-server.virtnet: executing set_default_debug -1 all pdsh@oleg454-client: oleg454-server: ssh exited with exit code 1 Started lustre-MDT0000 07:35:20 (1788867320) targets are mounted 07:35:20 (1788867320) facet_failover done oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 4 (74s) == replay-dual test 5: open, unlink |X| close ============ 07:35:41 (1788867341) multiop /mnt/lustre2/a vo_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.8222 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1788 1269516 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1668 1269636 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3088 7210952 1% /mnt/lustre Failing mds1 on oleg454-server Stopping /mnt/lustre-mds1 (opts:) on oleg454-server 07:35:58 (1788867358) shut down facet: mds1 facet_host: oleg454-server facet_failover_host: oleg454-server Failover mds1 to oleg454-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg454-server: oleg454-server.virtnet: executing set_default_debug -1 all pdsh@oleg454-client: oleg454-server: ssh exited with exit code 1 Started lustre-MDT0000 07:36:31 (1788867391) targets are mounted 07:36:31 (1788867391) facet_failover done oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 5 (65s) == replay-dual test 6: open1, open2, unlink |X| close1 [fail mds1] close2 ========================================================== 07:36:46 (1788867406) multiop /mnt/lustre2/a vo_c TMPPIPE=/tmp/multiop_open_wait_pipe.8222 multiop /mnt/lustre/a vo_c TMPPIPE=/tmp/multiop_open_wait_pipe.8222 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1788 1269516 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1668 1269636 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3088 7210952 1% /mnt/lustre Failing mds1 on oleg454-server Stopping /mnt/lustre-mds1 (opts:) on oleg454-server 07:36:59 (1788867419) shut down facet: mds1 facet_host: oleg454-server facet_failover_host: oleg454-server Failover mds1 to oleg454-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg454-server: oleg454-server.virtnet: executing set_default_debug -1 all pdsh@oleg454-client: oleg454-server: ssh exited with exit code 1 Started lustre-MDT0000 07:37:34 (1788867454) targets are mounted 07:37:34 (1788867454) facet_failover done oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 6 (62s) == replay-dual test 8: replay of resent request ========== 07:37:48 (1788867468) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1788 1269516 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1668 1269636 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3088 7210952 1% /mnt/lustre fail_loc=0x119 fail_loc=0 Failing mds1 on oleg454-server Stopping /mnt/lustre-mds1 (opts:) on oleg454-server 07:38:19 (1788867499) shut down facet: mds1 facet_host: oleg454-server facet_failover_host: oleg454-server Failover mds1 to oleg454-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg454-server: oleg454-server.virtnet: executing set_default_debug -1 all pdsh@oleg454-client: oleg454-server: ssh exited with exit code 1 Started lustre-MDT0000 07:38:51 (1788867531) targets are mounted 07:38:51 (1788867531) facet_failover done oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 8 (77s) == replay-dual test 9: resending a replayed create ======= 07:39:05 (1788867545) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1788 1269516 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1668 1269636 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3088 7210952 1% /mnt/lustre fail_loc=0x80000119 Failing mds1 on oleg454-server Stopping /mnt/lustre-mds1 (opts:) on oleg454-server 07:39:20 (1788867560) shut down facet: mds1 facet_host: oleg454-server facet_failover_host: oleg454-server Failover mds1 to oleg454-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg454-server: oleg454-server.virtnet: executing set_default_debug -1 all pdsh@oleg454-client: oleg454-server: ssh exited with exit code 1 Started lustre-MDT0000 07:39:55 (1788867595) targets are mounted 07:39:55 (1788867595) facet_failover done oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec fail_loc=0 PASS 9 (76s) == replay-dual test 10: resending a replayed unlink ====== 07:40:21 (1788867621) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1788 1269516 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1668 1269636 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3088 7210952 1% /mnt/lustre fail_loc=0x80000119 Failing mds1 on oleg454-server Stopping /mnt/lustre-mds1 (opts:) on oleg454-server 07:40:34 (1788867634) shut down facet: mds1 facet_host: oleg454-server facet_failover_host: oleg454-server Failover mds1 to oleg454-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg454-server: oleg454-server.virtnet: executing set_default_debug -1 all pdsh@oleg454-client: oleg454-server: ssh exited with exit code 1 Started lustre-MDT0000 07:41:09 (1788867669) targets are mounted 07:41:09 (1788867669) facet_failover done oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec fail_loc=0 PASS 10 (73s) == replay-dual test 11: both clients timeout during replay ========================================================== 07:41:34 (1788867694) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1788 1269516 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1668 1269636 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3088 7210952 1% /mnt/lustre fail_loc=0x0119 Failing mds1 on oleg454-server Stopping /mnt/lustre-mds1 (opts:) on oleg454-server 07:41:49 (1788867709) shut down facet: mds1 facet_host: oleg454-server facet_failover_host: oleg454-server Failover mds1 to oleg454-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg454-server: oleg454-server.virtnet: executing set_default_debug -1 all pdsh@oleg454-client: oleg454-server: ssh exited with exit code 1 Started lustre-MDT0000 07:42:21 (1788867741) targets are mounted 07:42:21 (1788867741) facet_failover done oleg454-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 5 sec fail_loc=0 PASS 11 (65s) == replay-dual test 12: open resend timeout ============== 07:42:39 (1788867759) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1788 1269516 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1668 1269636 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3088 7210952 1% /mnt/lustre multiop /mnt/lustre/f12.replay-dual vmo_c TMPPIPE=/tmp/multiop_open_wait_pipe.8222 fail_loc=0x80000302 Failing mds1 on oleg454-server Stopping /mnt/lustre-mds1 (opts:) on oleg454-server 07:42:54 (1788867774) shut down facet: mds1 facet_host: oleg454-server facet_failover_host: oleg454-server Failover mds1 to oleg454-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg454-server: oleg454-server.virtnet: executing set_default_debug -1 all pdsh@oleg454-client: oleg454-server: ssh exited with exit code 1 Started lustre-MDT0000 07:43:29 (1788867809) targets are mounted 07:43:29 (1788867809) facet_failover done fail_loc=0 /mnt/lustre/f12.replay-dual /mnt/lustre/f12.replay-dual has type file OK PASS 12 (68s) == replay-dual test 13: close resend timeout ============= 07:43:47 (1788867827) multiop /mnt/lustre/f13.replay-dual vmo_c TMPPIPE=/tmp/multiop_open_wait_pipe.8222 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1788 1269516 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1668 1269636 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3088 7210952 1% /mnt/lustre fail_loc=0x80000115 Failing mds1 on oleg454-server Stopping /mnt/lustre-mds1 (opts:) on oleg454-server 07:43:59 (1788867839) shut down facet: mds1 facet_host: oleg454-server facet_failover_host: oleg454-server Failover mds1 to oleg454-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg454-server: oleg454-server.virtnet: executing set_default_debug -1 all pdsh@oleg454-client: oleg454-server: ssh exited with exit code 1 Started lustre-MDT0000 07:44:29 (1788867869) targets are mounted 07:44:29 (1788867869) facet_failover done fail_loc=0 /mnt/lustre/f13.replay-dual /mnt/lustre/f13.replay-dual has type file OK PASS 13 (64s) SKIP: replay-dual test_14b skipping ALWAYS excluded test 14b == replay-dual test 15a: timeout waiting for lost client during replay, 1 client completes ========================================================== 07:44:52 (1788867892) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1788 1269516 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1668 1269636 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3088 7210952 1% /mnt/lustre total: 25 open/close in 0.87 seconds: 28.76 ops/second total: 1 open/close in 0.04 seconds: 26.73 ops/second Failing mds1 on oleg454-server Stopping /mnt/lustre-mds1 (opts:) on oleg454-server 07:45:07 (1788867907) shut down facet: mds1 facet_host: oleg454-server facet_failover_host: oleg454-server Failover mds1 to oleg454-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg454-server: oleg454-server.virtnet: executing set_default_debug -1 all pdsh@oleg454-client: oleg454-server: ssh exited with exit code 1 Started lustre-MDT0000 07:45:43 (1788867943) targets are mounted 07:45:43 (1788867943) facet_failover done oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec - unlinked 0 (time 1788868018 ; total 0 ; last 0) total: 25 unlinks in 1 seconds: 25.000000 unlinks/second Starting client: oleg454-client.virtnet: -o user_xattr,flock 192.168.204.154@tcp:/lustre /mnt/lustre2 PASS 15a (132s) == replay-dual test 15c: remove multiple OST orphans ===== 07:47:04 (1788868024) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1788 1269516 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1668 1269636 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3088 7210952 1% /mnt/lustre Failing mds1 on oleg454-server Stopping /mnt/lustre-mds1 (opts:) on oleg454-server 07:48:54 (1788868134) shut down facet: mds1 facet_host: oleg454-server facet_failover_host: oleg454-server Failover mds1 to oleg454-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg454-server: oleg454-server.virtnet: executing set_default_debug -1 all pdsh@oleg454-client: oleg454-server: ssh exited with exit code 1 Started lustre-MDT0000 07:49:20 (1788868160) targets are mounted 07:49:20 (1788868160) facet_failover done oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Starting client: oleg454-client.virtnet: -o user_xattr,flock 192.168.204.154@tcp:/lustre /mnt/lustre2 PASS 15c (216s) == replay-dual test 16: fail MDS during recovery (3571) == 07:50:40 (1788868240) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1788 1269516 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1668 1269636 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3088 7210952 1% /mnt/lustre total: 25 open/close in 0.86 seconds: 29.07 ops/second total: 1 open/close in 0.03 seconds: 34.60 ops/second Failing mds1 on oleg454-server Stopping /mnt/lustre-mds1 (opts:) on oleg454-server 07:50:54 (1788868254) shut down facet: mds1 facet_host: oleg454-server facet_failover_host: oleg454-server Failover mds1 to oleg454-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg454-server: oleg454-server.virtnet: executing set_default_debug -1 all pdsh@oleg454-client: oleg454-server: ssh exited with exit code 1 Started lustre-MDT0000 07:51:28 (1788868288) targets are mounted 07:51:28 (1788868288) facet_failover done Failing mds1 on oleg454-server Stopping /mnt/lustre-mds1 (opts:) on oleg454-server 07:51:54 (1788868314) shut down facet: mds1 facet_host: oleg454-server facet_failover_host: oleg454-server Failover mds1 to oleg454-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg454-server: oleg454-server.virtnet: executing set_default_debug -1 all pdsh@oleg454-client: oleg454-server: ssh exited with exit code 1 Started lustre-MDT0000 07:52:25 (1788868345) targets are mounted 07:52:25 (1788868345) facet_failover done oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec - unlinked 0 (time 1788868420 ; total 0 ; last 0) total: 25 unlinks in 1 seconds: 25.000000 unlinks/second Starting client: oleg454-client.virtnet: -o user_xattr,flock 192.168.204.154@tcp:/lustre /mnt/lustre2 PASS 16 (185s) == replay-dual test 17: fail OST during recovery (3571) == 07:53:45 (1788868425) total: 25 open/close in 0.86 seconds: 29.20 ops/second total: 1 open/close in 0.05 seconds: 19.43 ops/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1788 1269516 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1668 1269636 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3088 7210952 1% /mnt/lustre Failing ost1 on oleg454-server Stopping /mnt/lustre-ost1 (opts:) on oleg454-server 07:53:58 (1788868438) shut down facet: ost1 facet_host: oleg454-server facet_failover_host: oleg454-server Failover ost1 to oleg454-server mount facets: ost1 Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg454-server: oleg454-server.virtnet: executing set_default_debug -1 all pdsh@oleg454-client: oleg454-server: ssh exited with exit code 1 Started lustre-OST0000 07:54:26 (1788868466) targets are mounted 07:54:26 (1788868466) facet_failover done Failing ost1 on oleg454-server Stopping /mnt/lustre-ost1 (opts:) on oleg454-server 07:54:51 (1788868491) shut down facet: ost1 facet_host: oleg454-server facet_failover_host: oleg454-server Failover ost1 to oleg454-server mount facets: ost1 Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg454-server: oleg454-server.virtnet: executing set_default_debug -1 all pdsh@oleg454-client: oleg454-server: ssh exited with exit code 1 Started lustre-OST0000 07:55:16 (1788868516) targets are mounted 07:55:16 (1788868516) facet_failover done oleg454-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec - unlinked 0 (time 1788868588 ; total 0 ; last 0) total: 25 unlinks in 1 seconds: 25.000000 unlinks/second Starting client: oleg454-client.virtnet: -o user_xattr,flock 192.168.204.154@tcp:/lustre /mnt/lustre2 PASS 17 (169s) == replay-dual test 18: ldlm_handle_enqueue succeeds on evicted export (3822) ========================================================== 07:56:35 (1788868595) debug=+dlmtrace fail_loc=0x8000030b using seed 2561279824 running for 500 iterations total: 500 stats in 2 seconds: 250.000000 stats/second ldlm.namespaces.MGC192.168.204.154@tcp.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0000-mdc-ffff96d0873bd000.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0000-mdc-ffff96d099be6000.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0001-mdc-ffff96d0873bd000.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0001-mdc-ffff96d099be6000.early_lock_cancel=0 ldlm.namespaces.lustre-OST0000-osc-ffff96d0873bd000.early_lock_cancel=0 ldlm.namespaces.lustre-OST0000-osc-ffff96d099be6000.early_lock_cancel=0 ldlm.namespaces.lustre-OST0001-osc-ffff96d0873bd000.early_lock_cancel=0 ldlm.namespaces.lustre-OST0001-osc-ffff96d099be6000.early_lock_cancel=0 fail_loc=0x80000305 Error in opening file "/mnt/lustre2/d18.replay-dual/f18.replay-dual"(flags=O_RDONLY) 2: No such file or directory ldlm.namespaces.MGC192.168.204.154@tcp.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0000-mdc-ffff96d0873bd000.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0000-mdc-ffff96d099be6000.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0001-mdc-ffff96d0873bd000.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0001-mdc-ffff96d099be6000.early_lock_cancel=1 ldlm.namespaces.lustre-OST0000-osc-ffff96d0873bd000.early_lock_cancel=1 ldlm.namespaces.lustre-OST0000-osc-ffff96d099be6000.early_lock_cancel=1 ldlm.namespaces.lustre-OST0001-osc-ffff96d0873bd000.early_lock_cancel=1 ldlm.namespaces.lustre-OST0001-osc-ffff96d099be6000.early_lock_cancel=1 fail_loc=0 fail_loc=0 PASS 18 (61s) == replay-dual test 19: resend of open request =========== 07:57:36 (1788868656) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1792 1269512 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1668 1269636 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1560 3605460 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3104 7210936 1% /mnt/lustre fail_loc=0x157 - open/close 0 (time 1788868754.75 total 86.15 last 0.00) total: 1 open/close in 86.15 seconds: 0.01 ops/second fail_loc=0 Failing mds1 on oleg454-server Stopping /mnt/lustre-mds1 (opts:) on oleg454-server 07:59:20 (1788868760) shut down facet: mds1 facet_host: oleg454-server facet_failover_host: oleg454-server