-----============= acceptance-small: replay-dual ============----- Sun Feb 25 00:28:03 EST 2024 excepting tests: 14b 21b skipping tests SLOW=no: 21b === replay-dual: start setup 00:28:05 (1708838885) === Starting client oleg318-client.virtnet: -o user_xattr,flock oleg318-server@tcp:/lustre /mnt/lustre2 Started clients oleg318-client.virtnet: 192.168.203.118@tcp:/lustre on /mnt/lustre2 type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) oleg318-client.virtnet: executing check_config_client /mnt/lustre oleg318-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg318-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff88012aaaa800.idle_timeout=debug osc.lustre-OST0000-osc-ffff88012dc24800.idle_timeout=debug osc.lustre-OST0001-osc-ffff88012aaaa800.idle_timeout=debug osc.lustre-OST0001-osc-ffff88012dc24800.idle_timeout=debug disable quota as required oleg318-server: oleg318-server.virtnet: executing set_default_debug -1 all osd-ldiskfs.track_declares_assert=1 === replay-dual: finish setup 00:28:12 (1708838892) === osp.lustre-OST0000-osc-MDT0001.prealloc_force_new_seq=1 osp.lustre-OST0001-osc-MDT0000.prealloc_force_new_seq=1 osp.lustre-OST0001-osc-MDT0001.prealloc_force_new_seq=1 osp.lustre-OST0000-osc-MDT0000.prealloc_force_new_seq=1 Creating to objid 33 on ost lustre-OST0001... Creating to objid 33 on ost lustre-OST0000... Creating to objid 33 on ost lustre-OST0001... Creating to objid 33 on ost lustre-OST0000... total: 33 open/close in 0.22 seconds: 150.00 ops/second total: 33 open/close in 0.22 seconds: 150.34 ops/second total: 33 open/close in 0.22 seconds: 147.91 ops/second total: 33 open/close in 0.22 seconds: 147.51 ops/second osp.lustre-OST0001-osc-MDT0001.prealloc_force_new_seq=0 osp.lustre-OST0000-osc-MDT0001.prealloc_force_new_seq=0 osp.lustre-OST0000-osc-MDT0000.prealloc_force_new_seq=0 osp.lustre-OST0001-osc-MDT0000.prealloc_force_new_seq=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-dual test 0a: expired recovery with lost client ========================================================== 00:28:25 (1708838905) Check file is LU482_FAILED=/tmp/replay-dual.lu482.1RjTkV UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1840 1285848 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1688 1286000 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1796 3605224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1796 3605224 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3592 7210448 1% /mnt/lustre total: 50 open/close in 0.41 seconds: 121.56 ops/second fail_loc=0x80000514 Failing mds1 on oleg318-server Stopping /mnt/lustre-mds1 (opts:) on oleg318-server 00:28:29 (1708838909) shut down Failover mds1 to oleg318-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg318-server: oleg318-server.virtnet: executing set_default_debug -1 all pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-MDT0000 00:28:43 (1708838923) targets are mounted 00:28:43 (1708838923) facet_failover done Starting client: oleg318-client.virtnet: -o user_xattr,flock oleg318-server@tcp:/lustre /mnt/lustre2 - unlinked 0 (time 1708839006 ; total 0 ; last 0) total: 50 unlinks in 1 seconds: 50.000000 unlinks/second PASS 0a (103s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-dual test 0b: lost client during waiting for next transno ========================================================== 00:30:09 (1708839009) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1840 1285848 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1688 1286000 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1796 3605224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1796 3605224 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3592 7210448 1% /mnt/lustre Failing mds1 on oleg318-server Stopping /mnt/lustre-mds1 (opts:) on oleg318-server 00:30:13 (1708839013) shut down Failover mds1 to oleg318-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg318-server: oleg318-server.virtnet: executing set_default_debug -1 all pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-MDT0000 00:30:26 (1708839026) targets are mounted 00:30:26 (1708839026) facet_failover done Starting client: oleg318-client.virtnet: -o user_xattr,flock oleg318-server@tcp:/lustre /mnt/lustre Starting client: oleg318-client.virtnet: -o user_xattr,flock oleg318-server@tcp:/lustre /mnt/lustre2 PASS 0b (119s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-dual test 1: |X| simple create ================= 00:32:09 (1708839129) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1840 1285848 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1688 1286000 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1796 3605224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1796 3605224 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3592 7210448 1% /mnt/lustre mcreate: cannot create `/mnt/lustre/fsa-oleg318-client.virtnet' with mode 0100644: File exists Failing mds1 on oleg318-server Stopping /mnt/lustre-mds1 (opts:) on oleg318-server 00:32:13 (1708839133) shut down Failover mds1 to oleg318-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg318-server: oleg318-server.virtnet: executing set_default_debug -1 all pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-MDT0000 00:32:26 (1708839146) targets are mounted 00:32:26 (1708839146) facet_failover done oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 1 (24s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-dual test 2: |X| mkdir adir ==================== 00:32:35 (1708839155) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1840 1285848 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1688 1286000 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1796 3605224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1796 3605224 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3592 7210448 1% /mnt/lustre Failing mds1 on oleg318-server Stopping /mnt/lustre-mds1 (opts:) on oleg318-server 00:32:39 (1708839159) shut down Failover mds1 to oleg318-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg318-server: oleg318-server.virtnet: executing set_default_debug -1 all pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-MDT0000 00:32:53 (1708839173) targets are mounted 00:32:53 (1708839173) facet_failover done oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 2 (25s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-dual test 3: |X| mkdir adir, mkdir adir/bdir === 00:33:01 (1708839181) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1840 1285848 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1688 1286000 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1796 3605224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1796 3605224 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3592 7210448 1% /mnt/lustre Failing mds1 on oleg318-server Stopping /mnt/lustre-mds1 (opts:) on oleg318-server 00:33:05 (1708839185) shut down Failover mds1 to oleg318-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg318-server: oleg318-server.virtnet: executing set_default_debug -1 all pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-MDT0000 00:33:19 (1708839199) targets are mounted 00:33:19 (1708839199) facet_failover done oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 3 (25s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-dual test 4: |X| mkdir adir (-EEXIST), mkdir adir/bdir ========================================================== 00:33:28 (1708839208) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1916 1285772 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1796 1285892 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1796 3605224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1796 3605224 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3592 7210448 1% /mnt/lustre mkdir: cannot create directory '/mnt/lustre/adir': File exists Failing mds1 on oleg318-server Stopping /mnt/lustre-mds1 (opts:) on oleg318-server 00:33:36 (1708839216) shut down Failover mds1 to oleg318-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg318-server: oleg318-server.virtnet: executing set_default_debug -1 all pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-MDT0000 00:33:50 (1708839230) targets are mounted 00:33:50 (1708839230) facet_failover done oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 4 (29s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-dual test 5: open, unlink |X| close ============ 00:33:59 (1708839239) multiop /mnt/lustre2/a vo_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7303 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1916 1285772 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1800 1285888 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1796 3605224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1796 3605224 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3592 7210448 1% /mnt/lustre Failing mds1 on oleg318-server Stopping /mnt/lustre-mds1 (opts:) on oleg318-server 00:34:02 (1708839242) shut down Failover mds1 to oleg318-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg318-server: oleg318-server.virtnet: executing set_default_debug -1 all pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-MDT0000 00:34:16 (1708839256) targets are mounted 00:34:16 (1708839256) facet_failover done oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 5 (24s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-dual test 6: open1, open2, unlink |X| close1 [fail mds1] close2 ========================================================== 00:34:24 (1708839264) multiop /mnt/lustre2/a vo_c TMPPIPE=/tmp/multiop_open_wait_pipe.7303 multiop /mnt/lustre/a vo_c TMPPIPE=/tmp/multiop_open_wait_pipe.7303 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1916 1285772 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1800 1285888 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1796 3605224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1796 3605224 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3592 7210448 1% /mnt/lustre Failing mds1 on oleg318-server Stopping /mnt/lustre-mds1 (opts:) on oleg318-server 00:34:28 (1708839268) shut down Failover mds1 to oleg318-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg318-server: oleg318-server.virtnet: executing set_default_debug -1 all pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-MDT0000 00:34:41 (1708839281) targets are mounted 00:34:41 (1708839281) facet_failover done oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 6 (25s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-dual test 8: replay of resent request ========== 00:34:50 (1708839290) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1916 1285772 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1800 1285888 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1796 3605224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1796 3605224 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3592 7210448 1% /mnt/lustre fail_loc=0x119 fail_loc=0 Failing mds1 on oleg318-server Stopping /mnt/lustre-mds1 (opts:) on oleg318-server 00:35:10 (1708839310) shut down Failover mds1 to oleg318-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg318-server: oleg318-server.virtnet: executing set_default_debug -1 all pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-MDT0000 00:35:24 (1708839324) targets are mounted 00:35:24 (1708839324) facet_failover done oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 8 (41s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-dual test 9: resending a replayed create ======= 00:35:32 (1708839332) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1916 1285772 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1800 1285888 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1796 3605224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1796 3605224 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3592 7210448 1% /mnt/lustre fail_loc=0x80000119 Failing mds1 on oleg318-server Stopping /mnt/lustre-mds1 (opts:) on oleg318-server 00:35:36 (1708839336) shut down Failover mds1 to oleg318-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg318-server: oleg318-server.virtnet: executing set_default_debug -1 all pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-MDT0000 00:35:50 (1708839350) targets are mounted 00:35:50 (1708839350) facet_failover done oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec fail_loc=0 PASS 9 (41s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-dual test 10: resending a replayed unlink ====== 00:36:15 (1708839375) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1916 1285772 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1800 1285888 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1796 3605224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1796 3605224 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3592 7210448 1% /mnt/lustre fail_loc=0x80000119 Failing mds1 on oleg318-server Stopping /mnt/lustre-mds1 (opts:) on oleg318-server 00:36:18 (1708839378) shut down Failover mds1 to oleg318-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg318-server: oleg318-server.virtnet: executing set_default_debug -1 all pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-MDT0000 00:36:32 (1708839392) targets are mounted 00:36:32 (1708839392) facet_failover done oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec fail_loc=0 PASS 10 (38s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-dual test 11: both clients timeout during replay ========================================================== 00:36:55 (1708839415) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1916 1285772 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1800 1285888 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1796 3605224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1796 3605224 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3592 7210448 1% /mnt/lustre fail_loc=0x0119 Failing mds1 on oleg318-server Stopping /mnt/lustre-mds1 (opts:) on oleg318-server 00:36:59 (1708839419) shut down Failover mds1 to oleg318-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg318-server: oleg318-server.virtnet: executing set_default_debug -1 all pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-MDT0000 00:37:13 (1708839433) targets are mounted 00:37:13 (1708839433) facet_failover done oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 16 sec fail_loc=0 PASS 11 (37s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-dual test 12: open resend timeout ============== 00:37:34 (1708839454) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1916 1285772 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1800 1285888 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1796 3605224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1796 3605224 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3592 7210448 1% /mnt/lustre multiop /mnt/lustre/f12.replay-dual vmo_c TMPPIPE=/tmp/multiop_open_wait_pipe.7303 fail_loc=0x80000302 Failing mds1 on oleg318-server Stopping /mnt/lustre-mds1 (opts:) on oleg318-server 00:37:38 (1708839458) shut down Failover mds1 to oleg318-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg318-server: oleg318-server.virtnet: executing set_default_debug -1 all pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-MDT0000 00:37:53 (1708839473) targets are mounted 00:37:53 (1708839473) facet_failover done fail_loc=0 /mnt/lustre/f12.replay-dual /mnt/lustre/f12.replay-dual has type file OK PASS 12 (23s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-dual test 13: close resend timeout ============= 00:37:59 (1708839479) multiop /mnt/lustre/f13.replay-dual vmo_c TMPPIPE=/tmp/multiop_open_wait_pipe.7303 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1916 1285772 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1800 1285888 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1796 3605224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1796 3605224 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3592 7210448 1% /mnt/lustre fail_loc=0x80000115 Failing mds1 on oleg318-server Stopping /mnt/lustre-mds1 (opts:) on oleg318-server 00:38:04 (1708839484) shut down Failover mds1 to oleg318-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg318-server: oleg318-server.virtnet: executing set_default_debug -1 all pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-MDT0000 00:38:19 (1708839499) targets are mounted 00:38:19 (1708839499) facet_failover done fail_loc=0 /mnt/lustre/f13.replay-dual /mnt/lustre/f13.replay-dual has type file OK PASS 13 (24s) debug_raw_pointers=0 debug_raw_pointers=0 SKIP: replay-dual test_14b skipping ALWAYS excluded test 14b debug_raw_pointers=Y debug_raw_pointers=Y == replay-dual test 15a: timeout waiting for lost client during replay, 1 client completes ========================================================== 00:38:25 (1708839505) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1916 1285772 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1800 1285888 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1796 3605224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1796 3605224 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3592 7210448 1% /mnt/lustre total: 25 open/close in 0.18 seconds: 139.60 ops/second total: 1 open/close in 0.01 seconds: 133.25 ops/second Failing mds1 on oleg318-server Stopping /mnt/lustre-mds1 (opts:) on oleg318-server 00:38:29 (1708839509) shut down Failover mds1 to oleg318-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg318-server: oleg318-server.virtnet: executing set_default_debug -1 all pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-MDT0000 00:38:44 (1708839524) targets are mounted 00:38:44 (1708839524) facet_failover done oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec - unlinked 0 (time 1708839596 ; total 0 ; last 0) total: 25 unlinks in 0 seconds: inf unlinks/second Starting client: oleg318-client.virtnet: -o user_xattr,flock oleg318-server@tcp:/lustre /mnt/lustre2 PASS 15a (93s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-dual test 15c: remove multiple OST orphans ===== 00:40:00 (1708839600) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1916 1285772 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1800 1285888 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1796 3605224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1796 3605224 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3592 7210448 1% /mnt/lustre Failing mds1 on oleg318-server Stopping /mnt/lustre-mds1 (opts:) on oleg318-server 00:40:37 (1708839637) shut down Failover mds1 to oleg318-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg318-server: oleg318-server.virtnet: executing set_default_debug -1 all pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-MDT0000 00:40:52 (1708839652) targets are mounted 00:40:52 (1708839652) facet_failover done oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Starting client: oleg318-client.virtnet: -o user_xattr,flock oleg318-server@tcp:/lustre /mnt/lustre2 PASS 15c (125s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-dual test 16: fail MDS during recovery (3571) == 00:42:06 (1708839726) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1916 1285772 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1800 1285888 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1796 3605224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1796 3605224 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3592 7210448 1% /mnt/lustre total: 25 open/close in 0.24 seconds: 104.46 ops/second total: 1 open/close in 0.01 seconds: 86.37 ops/second Failing mds1 on oleg318-server Stopping /mnt/lustre-mds1 (opts:) on oleg318-server 00:42:11 (1708839731) shut down Failover mds1 to oleg318-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg318-server: oleg318-server.virtnet: executing set_default_debug -1 all pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-MDT0000 00:42:25 (1708839745) targets are mounted 00:42:25 (1708839745) facet_failover done Failing mds1 on oleg318-server Stopping /mnt/lustre-mds1 (opts:) on oleg318-server 00:42:47 (1708839767) shut down Failover mds1 to oleg318-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg318-server: oleg318-server.virtnet: executing set_default_debug -1 all pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-MDT0000 00:43:00 (1708839780) targets are mounted 00:43:00 (1708839780) facet_failover done oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec - unlinked 0 (time 1708839852 ; total 0 ; last 0) total: 25 unlinks in 1 seconds: 25.000000 unlinks/second Starting client: oleg318-client.virtnet: -o user_xattr,flock oleg318-server@tcp:/lustre /mnt/lustre2 PASS 16 (128s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-dual test 17: fail OST during recovery (3571) == 00:44:16 (1708839856) total: 25 open/close in 0.25 seconds: 98.35 ops/second total: 1 open/close in 0.01 seconds: 67.32 ops/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1988 1285700 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1800 1285888 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1796 3605224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1796 3605224 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3592 7210448 1% /mnt/lustre Failing ost1 on oleg318-server Stopping /mnt/lustre-ost1 (opts:) on oleg318-server 00:44:20 (1708839860) shut down Failover ost1 to oleg318-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg318-server: oleg318-server.virtnet: executing set_default_debug -1 all pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-OST0000 00:44:35 (1708839875) targets are mounted 00:44:35 (1708839875) facet_failover done Failing ost1 on oleg318-server Stopping /mnt/lustre-ost1 (opts:) on oleg318-server 00:44:56 (1708839896) shut down Failover ost1 to oleg318-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg318-server: oleg318-server.virtnet: executing set_default_debug -1 all pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-OST0000 00:45:11 (1708839911) targets are mounted 00:45:11 (1708839911) facet_failover done oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec - unlinked 0 (time 1708839986 ; total 0 ; last 0) total: 25 unlinks in 1 seconds: 25.000000 unlinks/second Starting client: oleg318-client.virtnet: -o user_xattr,flock oleg318-server@tcp:/lustre /mnt/lustre2 PASS 17 (132s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-dual test 18: ldlm_handle_enqueue succeeds on evicted export (3822) ========================================================== 00:46:30 (1708839990) debug=+dlmtrace using seed 3673538878 running for 500 iterations total: 500 stats in 0 seconds: inf stats/second fail_loc=0x8000030b ldlm.namespaces.MGC192.168.203.118@tcp.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0000-mdc-ffff8800aa2e6000.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0000-mdc-ffff8800aa3a0000.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0001-mdc-ffff8800aa2e6000.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0001-mdc-ffff8800aa3a0000.early_lock_cancel=0 ldlm.namespaces.lustre-OST0000-osc-ffff8800aa2e6000.early_lock_cancel=0 ldlm.namespaces.lustre-OST0000-osc-ffff8800aa3a0000.early_lock_cancel=0 ldlm.namespaces.lustre-OST0001-osc-ffff8800aa2e6000.early_lock_cancel=0 ldlm.namespaces.lustre-OST0001-osc-ffff8800aa3a0000.early_lock_cancel=0 fail_loc=0x80000305 Error in opening file "/mnt/lustre2/d18.replay-dual/f18.replay-dual"(flags=O_RDONLY) 2: No such file or directory ldlm.namespaces.MGC192.168.203.118@tcp.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0000-mdc-ffff8800aa2e6000.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0000-mdc-ffff8800aa3a0000.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0001-mdc-ffff8800aa2e6000.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0001-mdc-ffff8800aa3a0000.early_lock_cancel=1 ldlm.namespaces.lustre-OST0000-osc-ffff8800aa2e6000.early_lock_cancel=1 ldlm.namespaces.lustre-OST0000-osc-ffff8800aa3a0000.early_lock_cancel=1 ldlm.namespaces.lustre-OST0001-osc-ffff8800aa2e6000.early_lock_cancel=1 ldlm.namespaces.lustre-OST0001-osc-ffff8800aa3a0000.early_lock_cancel=1 fail_loc=0 fail_loc=0 PASS 18 (46s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-dual test 19: resend of open request =========== 00:47:17 (1708840037) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1992 1285696 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1800 1285888 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1812 3605208 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1796 3605224 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3608 7210432 1% /mnt/lustre fail_loc=0x157 - open/close 0 (time 1708840127.06 total 86.04 last 0.00) total: 1 open/close in 86.04 seconds: 0.01 ops/second fail_loc=0 Failing mds1 on oleg318-server Stopping /mnt/lustre-mds1 (opts:) on oleg318-server 00:48:48 (1708840128) shut down Failover mds1 to oleg318-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg318-server: oleg318-server.virtnet: executing set_default_debug -1 all pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-MDT0000 00:49:03 (1708840143) targets are mounted 00:49:03 (1708840143) facet_failover done oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 19 (114s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-dual test 20: recovery time is not increasing == 00:49:13 (1708840153) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1992 1285696 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1800 1285888 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1812 3605208 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1796 3605224 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3608 7210432 1% /mnt/lustre Failing mds1 on oleg318-server Stopping /mnt/lustre-mds1 (opts:) on oleg318-server 00:49:17 (1708840157) shut down Failover mds1 to oleg318-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg318-server: oleg318-server.virtnet: executing set_default_debug -1 all pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-MDT0000 00:49:32 (1708840172) targets are mounted 00:49:32 (1708840172) facet_failover done oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Starting client: oleg318-client.virtnet: -o user_xattr,flock oleg318-server@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1992 1285696 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1800 1285888 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1812 3605208 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1796 3605224 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3608 7210432 1% /mnt/lustre Failing mds1 on oleg318-server Stopping /mnt/lustre-mds1 (opts:) on oleg318-server 00:51:57 (1708840317) shut down Failover mds1 to oleg318-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg318-server: oleg318-server.virtnet: executing set_default_debug -1 all pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-MDT0000 00:52:11 (1708840331) targets are mounted 00:52:11 (1708840331) facet_failover done oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Starting client: oleg318-client.virtnet: -o user_xattr,flock oleg318-server@tcp:/lustre /mnt/lustre2 PASS 20 (323s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-dual test 21a: commit on sharing =============== 00:54:38 (1708840478) mdt.lustre-MDT0000.commit_on_sharing=1 mdt.lustre-MDT0001.commit_on_sharing=1 Replay barrier on lustre-MDT0000 Failing mds1 on oleg318-server Stopping /mnt/lustre-mds1 (opts:) on oleg318-server 00:54:42 (1708840482) shut down Failover mds1 to oleg318-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg318-server: oleg318-server.virtnet: executing set_default_debug -1 all pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-MDT0000 00:54:57 (1708840497) targets are mounted 00:54:57 (1708840497) facet_failover done Starting client: oleg318-client.virtnet: -o user_xattr,flock oleg318-server@tcp:/lustre /mnt/lustre2 mdt.lustre-MDT0000.commit_on_sharing=0 mdt.lustre-MDT0001.commit_on_sharing=0 PASS 21a (164s) debug_raw_pointers=0 debug_raw_pointers=0 SKIP: replay-dual test_21b skipping SLOW test 21b debug_raw_pointers=Y debug_raw_pointers=Y == replay-dual test 22a: c1 lfs mkdir -i 1 dir1, M1 drop reply & fail, c2 mkdir dir1/dir ========================================================== 00:57:24 (1708840644) fail_loc=0x119 Failing mds2 on oleg318-server Stopping /mnt/lustre-mds2 (opts:) on oleg318-server 00:57:27 (1708840647) shut down Failover mds2 to oleg318-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg318-server: oleg318-server.virtnet: executing set_default_debug -1 all pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-MDT0001 00:57:41 (1708840661) targets are mounted 00:57:41 (1708840661) facet_failover done oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2000 1285688 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1824 1285864 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1812 3605208 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1796 3605224 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3608 7210432 1% /mnt/lustre total: 2 open/close in 0.04 seconds: 53.79 ops/second total: 2 open/close in 0.02 seconds: 114.02 ops/second Failing mds1 on oleg318-server Stopping /mnt/lustre-mds1 (opts:) on oleg318-server 00:57:51 (1708840671) shut down Failover mds1 to oleg318-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg318-server: oleg318-server.virtnet: executing set_default_debug -1 all pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-MDT0000 00:58:05 (1708840685) targets are mounted 00:58:05 (1708840685) facet_failover done oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 22a (49s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-dual test 22b: c1 lfs mkdir -i 1 d1, M1 drop reply & fail M0/M1, c2 mkdir d1/dir ========================================================== 00:58:15 (1708840695) fail_loc=0x119 Failing mds1 on oleg318-server Stopping /mnt/lustre-mds1 (opts:) on oleg318-server Failing mds2 on oleg318-server Stopping /mnt/lustre-mds2 (opts:) on oleg318-server 00:58:20 (1708840700) shut down Failover mds1 to oleg318-server mount facets: mds1 Failover mds2 to oleg318-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg318-server: oleg318-server.virtnet: executing set_default_debug -1 all oleg318-server: oleg318-server.virtnet: executing set_default_debug -1 all pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-MDT0001 Started lustre-MDT0000 00:58:47 (1708840727) targets are mounted 00:58:47 (1708840727) facet_failover done oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1996 1285692 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1820 1285868 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1812 3605208 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1796 3605224 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3608 7210432 1% /mnt/lustre total: 2 open/close in 0.03 seconds: 67.61 ops/second total: 2 open/close in 0.02 seconds: 130.56 ops/second Failing mds1 on oleg318-server Stopping /mnt/lustre-mds1 (opts:) on oleg318-server 00:58:57 (1708840737) shut down Failover mds1 to oleg318-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg318-server: oleg318-server.virtnet: executing set_default_debug -1 all pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-MDT0000 00:59:11 (1708840751) targets are mounted 00:59:11 (1708840751) facet_failover done oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 22b (64s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-dual test 22c: c1 lfs mkdir -i 1 d1, M1 drop update & fail M1, c2 mkdir d1/dir ========================================================== 00:59:22 (1708840762) fail_loc=0x1701 fail_loc=0 Failing mds1 on oleg318-server Stopping /mnt/lustre-mds1 (opts:) on oleg318-server 00:59:25 (1708840765) shut down Failover mds1 to oleg318-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg318-server: oleg318-server.virtnet: executing set_default_debug -1 all pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-MDT0000 00:59:39 (1708840779) targets are mounted 00:59:39 (1708840779) facet_failover done oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1996 1285692 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1820 1285868 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1812 3605208 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1796 3605224 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3608 7210432 1% /mnt/lustre total: 2 open/close in 0.03 seconds: 57.81 ops/second total: 2 open/close in 0.02 seconds: 118.36 ops/second Failing mds1 on oleg318-server Stopping /mnt/lustre-mds1 (opts:) on oleg318-server 00:59:48 (1708840788) shut down Failover mds1 to oleg318-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg318-server: oleg318-server.virtnet: executing set_default_debug -1 all pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-MDT0000 01:00:03 (1708840803) targets are mounted 01:00:03 (1708840803) facet_failover done oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 22c (49s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-dual test 22d: c1 lfs mkdir -i 1 d1, M1 drop update & fail M0/M1,c2 mkdir d1/dir ========================================================== 01:00:13 (1708840813) fail_loc=0x1701 fail_loc=0 Failing mds1 on oleg318-server Stopping /mnt/lustre-mds1 (opts:) on oleg318-server Failing mds2 on oleg318-server Stopping /mnt/lustre-mds2 (opts:) on oleg318-server 01:00:26 (1708840826) shut down Failover mds1 to oleg318-server mount facets: mds1 Failover mds2 to oleg318-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg318-server: oleg318-server.virtnet: executing set_default_debug -1 all oleg318-server: oleg318-server.virtnet: executing set_default_debug -1 all pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-MDT0001 Started lustre-MDT0000 01:00:46 (1708840846) targets are mounted 01:00:46 (1708840846) facet_failover done oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1960 1285728 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1784 1285904 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1812 3605208 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1796 3605224 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3608 7210432 1% /mnt/lustre total: 2 open/close in 0.03 seconds: 57.15 ops/second total: 2 open/close in 0.01 seconds: 143.99 ops/second Failing mds1 on oleg318-server Stopping /mnt/lustre-mds1 (opts:) on oleg318-server 01:00:56 (1708840856) shut down Failover mds1 to oleg318-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg318-server: oleg318-server.virtnet: executing set_default_debug -1 all pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-MDT0000 01:01:11 (1708840871) targets are mounted 01:01:11 (1708840871) facet_failover done oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 22d (65s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-dual test 23a: c1 rmdir d1, M1 drop reply and fail, client2 mkdir d1 ========================================================== 01:01:20 (1708840880) fail_loc=0x119 fail_loc=0 Failing mds2 on oleg318-server Stopping /mnt/lustre-mds2 (opts:) on oleg318-server 01:01:24 (1708840884) shut down Failover mds2 to oleg318-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg318-server: oleg318-server.virtnet: executing set_default_debug -1 all pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-MDT0001 01:01:38 (1708840898) targets are mounted 01:01:38 (1708840898) facet_failover done oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1992 1285696 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1816 1285872 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1812 3605208 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1796 3605224 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3608 7210432 1% /mnt/lustre total: 2 open/close in 0.02 seconds: 96.78 ops/second Failing mds1 on oleg318-server Stopping /mnt/lustre-mds1 (opts:) on oleg318-server 01:01:47 (1708840907) shut down Failover mds1 to oleg318-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg318-server: oleg318-server.virtnet: executing set_default_debug -1 all pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-MDT0000 01:02:02 (1708840922) targets are mounted 01:02:02 (1708840922) facet_failover done oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 23a (49s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-dual test 23b: c1 rmdir d1, M1 drop reply and fail M0/M1, c2 mkdir d1 ========================================================== 01:02:11 (1708840931) fail_loc=0x119 fail_loc=0 Failing mds1 on oleg318-server Stopping /mnt/lustre-mds1 (opts:) on oleg318-server Failing mds2 on oleg318-server Stopping /mnt/lustre-mds2 (opts:) on oleg318-server 01:02:20 (1708840940) shut down Failover mds1 to oleg318-server mount facets: mds1 Failover mds2 to oleg318-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg318-server: oleg318-server.virtnet: executing set_default_debug -1 all oleg318-server: oleg318-server.virtnet: executing set_default_debug -1 all pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-MDT0001 Started lustre-MDT0000 01:02:41 (1708840961) targets are mounted 01:02:41 (1708840961) facet_failover done oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1992 1285696 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1816 1285872 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1812 3605208 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1796 3605224 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3608 7210432 1% /mnt/lustre total: 2 open/close in 0.02 seconds: 95.85 ops/second Failing mds1 on oleg318-server Stopping /mnt/lustre-mds1 (opts:) on oleg318-server 01:02:51 (1708840971) shut down Failover mds1 to oleg318-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg318-server: oleg318-server.virtnet: executing set_default_debug -1 all pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-MDT0000 01:03:05 (1708840985) targets are mounted 01:03:05 (1708840985) facet_failover done oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 23b (61s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-dual test 23c: c1 rmdir d1, M0 drop update reply and fail M0, c2 mkdir d1 ========================================================== 01:03:14 (1708840994) fail_loc=0x1701 fail_loc=0 Failing mds1 on oleg318-server Stopping /mnt/lustre-mds1 (opts:) on oleg318-server 01:03:17 (1708840997) shut down Failover mds1 to oleg318-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg318-server: oleg318-server.virtnet: executing set_default_debug -1 all pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-MDT0000 01:03:30 (1708841010) targets are mounted 01:03:30 (1708841010) facet_failover done oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1992 1285696 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1816 1285872 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1812 3605208 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1796 3605224 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3608 7210432 1% /mnt/lustre total: 2 open/close in 0.01 seconds: 155.33 ops/second Failing mds1 on oleg318-server Stopping /mnt/lustre-mds1 (opts:) on oleg318-server 01:03:39 (1708841019) shut down Failover mds1 to oleg318-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg318-server: oleg318-server.virtnet: executing set_default_debug -1 all pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-MDT0000 01:03:53 (1708841033) targets are mounted 01:03:53 (1708841033) facet_failover done oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 23c (46s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-dual test 23d: c1 rmdir d1, M0 drop update reply and fail M0/M1, c2 mkdir d1 ========================================================== 01:04:02 (1708841042) fail_loc=0x1701 fail_loc=0 Failing mds1 on oleg318-server Stopping /mnt/lustre-mds1 (opts:) on oleg318-server Failing mds2 on oleg318-server Stopping /mnt/lustre-mds2 (opts:) on oleg318-server 01:04:13 (1708841053) shut down Failover mds1 to oleg318-server mount facets: mds1 Failover mds2 to oleg318-server mount facets: mds2 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg318-server: oleg318-server.virtnet: executing set_default_debug -1 all oleg318-server: oleg318-server.virtnet: executing set_default_debug -1 all pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-MDT0001 Started lustre-MDT0000 01:04:34 (1708841074) targets are mounted 01:04:34 (1708841074) facet_failover done oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1956 1285732 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1780 1285908 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1812 3605208 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1796 3605224 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3608 7210432 1% /mnt/lustre total: 2 open/close in 0.02 seconds: 108.10 ops/second Failing mds1 on oleg318-server Stopping /mnt/lustre-mds1 (opts:) on oleg318-server 01:04:43 (1708841083) shut down Failover mds1 to oleg318-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg318-server: oleg318-server.virtnet: executing set_default_debug -1 all pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-MDT0000 01:04:57 (1708841097) targets are mounted 01:04:57 (1708841097) facet_failover done oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 23d (62s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-dual test 24: reconstruct on non-existing object ========================================================== 01:05:06 (1708841106) fail_loc=0x119 fail_loc=0 truncate: cannot truncate '/mnt/lustre/f24.replay-dual' to length 100: No such file or directory PASS 24 (87s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-dual test 25: replay|resend ==================== 01:06:35 (1708841195) 1+0 records in 1+0 records out 512 bytes (512 B) copied, 0.00454592 s, 113 kB/s fail_loc=0x304 fail_loc=0x80000325 Failing ost1 on oleg318-server Stopping /mnt/lustre-ost1 (opts:) on oleg318-server 01:06:39 (1708841199) shut down Failover ost1 to oleg318-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg318-server: oleg318-server.virtnet: executing set_default_debug -1 all pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-OST0000 01:06:53 (1708841213) targets are mounted 01:06:53 (1708841213) facet_failover done oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 6451: 4738 Terminated LUSTRE="/home/green/git/lustre-release/lustre" bash -c "multiop /mnt/lustre2/f25.replay-dual Ow512" fail_loc=0 PASS 25 (23s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-dual test 26: dbench and tar with mds failover ========================================================== 01:07:00 (1708841220) Starting client oleg318-client.virtnet: -o user_xattr,flock oleg318-server@tcp:/lustre /mnt/lustre Started clients oleg318-client.virtnet: 192.168.203.118@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) Started tar loop with pid 6359 Started dbench loop with 6360 striped dir -i0 -c2 -H crush /mnt/lustre2/d26.replay-dual/run_dbench striped dir -i0 -c2 -H crush /mnt/lustre/d26.replay-dual/run_tar looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Sun Feb 25 01:07:02 EST 2024 waiting for dbench pid 6401 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs failed to create barrier semaphore 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 164 6.80 MB/sec warmup 1 sec latency 38.079 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2220 1285468 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1992 1285696 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1816 3605204 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 26372 3580452 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 28188 7185656 1% /mnt/lustre 1 331 6.00 MB/sec warmup 2 sec latency 30.501 ms 1 454 5.30 MB/sec warmup 3 sec latency 426.961 ms 1 678 5.29 MB/sec warmup 4 sec latency 27.936 ms 1 863 4.31 MB/sec warmup 5 sec latency 25.235 ms test_26 fail mds1 1 times Failing mds1 on oleg318-server 1 1050 3.81 MB/sec warmup 6 sec latency 24.603 ms Stopping /mnt/lustre-mds1 (opts:) on oleg318-server 1 1190 3.29 MB/sec warmup 7 sec latency 532.449 ms 01:07:09 (1708841229) shut down 1 1190 2.88 MB/sec warmup 8 sec latency 1532.717 ms 1 1190 2.56 MB/sec warmup 9 sec latency 2532.920 ms 1 1190 2.30 MB/sec warmup 10 sec latency 3533.113 ms 1 1190 2.09 MB/sec warmup 11 sec latency 4533.261 ms 1 1190 1.92 MB/sec warmup 12 sec latency 5533.397 ms 1 1190 1.77 MB/sec warmup 13 sec latency 6533.580 ms 1 1190 1.65 MB/sec warmup 14 sec latency 7533.786 ms 1 1190 1.54 MB/sec warmup 15 sec latency 8534.000 ms 1 1190 1.44 MB/sec warmup 16 sec latency 9534.178 ms 1 1190 1.36 MB/sec warmup 17 sec latency 10534.374 ms Failover mds1 to oleg318-server mount facets: mds1 1 1190 1.28 MB/sec warmup 18 sec latency 11534.550 ms 1 1190 1.21 MB/sec warmup 19 sec latency 12534.697 ms Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 1190 0.00 MB/sec execute 1 sec latency 14534.990 ms oleg318-server: oleg318-server.virtnet: executing set_default_debug -1 all pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-MDT0000 01:07:24 (1708841244) targets are mounted 01:07:24 (1708841244) facet_failover done 1 1190 0.00 MB/sec execute 2 sec latency 15535.196 ms 1 1190 0.00 MB/sec execute 3 sec latency 16535.376 ms 1 1190 0.00 MB/sec execute 4 sec latency 17535.586 ms 1 1190 0.00 MB/sec execute 5 sec latency 18535.768 ms 1 1190 0.00 MB/sec execute 6 sec latency 19535.928 ms oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 1526 0.44 MB/sec execute 7 sec latency 19613.807 ms 1 1787 0.41 MB/sec execute 8 sec latency 19.110 ms 1 2058 0.40 MB/sec execute 9 sec latency 17.205 ms 1 2396 0.49 MB/sec execute 10 sec latency 16.410 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2924 1284764 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2612 1285076 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 7620 3590696 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 33388 3563424 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 41008 7154120 1% /mnt/lustre 1 2770 0.81 MB/sec execute 11 sec latency 19.945 ms 1 3197 0.94 MB/sec execute 12 sec latency 252.259 ms 1 3583 1.18 MB/sec execute 13 sec latency 459.801 ms 1 3828 1.19 MB/sec execute 14 sec latency 25.914 ms test_26 fail mds2 2 times Failing mds2 on oleg318-server 1 4029 1.13 MB/sec execute 15 sec latency 18.688 ms Stopping /mnt/lustre-mds2 (opts:) on oleg318-server 1 4141 1.07 MB/sec execute 16 sec latency 605.920 ms 01:07:38 (1708841258) shut down 1 4141 1.00 MB/sec execute 17 sec latency 1606.019 ms 1 4141 0.95 MB/sec execute 18 sec latency 2606.119 ms 1 4141 0.90 MB/sec execute 19 sec latency 3606.292 ms 1 4141 0.85 MB/sec execute 20 sec latency 4606.415 ms 1 4141 0.81 MB/sec execute 21 sec latency 5606.526 ms 1 4141 0.77 MB/sec execute 22 sec latency 6606.642 ms 1 4141 0.74 MB/sec execute 23 sec latency 7606.770 ms 1 4141 0.71 MB/sec execute 24 sec latency 8606.880 ms 1 4141 0.68 MB/sec execute 25 sec latency 9607.063 ms 1 4141 0.66 MB/sec execute 26 sec latency 10607.186 ms Failover mds2 to oleg318-server mount facets: mds2 1 4141 0.63 MB/sec execute 27 sec latency 11607.307 ms 1 4141 0.61 MB/sec execute 28 sec latency 12607.407 ms Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 1 4141 0.59 MB/sec execute 29 sec latency 13607.524 ms oleg318-server: oleg318-server.virtnet: executing set_default_debug -1 all 1 4141 0.57 MB/sec execute 30 sec latency 14607.629 ms pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-MDT0001 01:07:52 (1708841272) targets are mounted 01:07:52 (1708841272) facet_failover done 1 4141 0.55 MB/sec execute 31 sec latency 15607.759 ms 1 4141 0.53 MB/sec execute 32 sec latency 16607.874 ms 1 4141 0.52 MB/sec execute 33 sec latency 17607.983 ms 1 4141 0.50 MB/sec execute 34 sec latency 18608.105 ms oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1 4260 0.49 MB/sec execute 35 sec latency 19246.547 ms mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec 1 4488 0.50 MB/sec execute 36 sec latency 28.141 ms 1 4756 0.52 MB/sec execute 37 sec latency 21.388 ms 1 5116 0.58 MB/sec execute 38 sec latency 19.658 ms 1 5373 0.57 MB/sec execute 39 sec latency 19.903 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 4376 1283312 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2456 1285232 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12980 3582964 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 38160 3560412 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 51140 7143376 1% /mnt/lustre 1 5712 0.57 MB/sec execute 40 sec latency 12.600 ms 1 5986 0.59 MB/sec execute 41 sec latency 28.151 ms 1 6429 0.67 MB/sec execute 42 sec latency 19.071 ms 1 6920 0.73 MB/sec execute 43 sec latency 15.715 ms test_26 fail mds1 3 times Failing mds1 on oleg318-server 1 7239 0.79 MB/sec execute 44 sec latency 50.787 ms Stopping /mnt/lustre-mds1 (opts:) on oleg318-server 1 7422 0.80 MB/sec execute 45 sec latency 22.768 ms 01:08:07 (1708841287) shut down 1 7596 0.79 MB/sec execute 46 sec latency 20.180 ms 1 7627 0.77 MB/sec execute 47 sec latency 837.261 ms 1 7627 0.75 MB/sec execute 48 sec latency 1837.461 ms 1 7627 0.74 MB/sec execute 49 sec latency 2837.644 ms 1 7627 0.72 MB/sec execute 50 sec latency 3837.861 ms 1 7627 0.71 MB/sec execute 51 sec latency 4838.070 ms 1 7627 0.70 MB/sec execute 52 sec latency 5838.213 ms 1 7627 0.68 MB/sec execute 53 sec latency 6838.374 ms 1 7627 0.67 MB/sec execute 54 sec latency 7838.517 ms 1 7627 0.66 MB/sec execute 55 sec latency 8838.705 ms Failover mds1 to oleg318-server mount facets: mds1 1 7627 0.65 MB/sec execute 56 sec latency 9838.859 ms 1 7627 0.63 MB/sec execute 57 sec latency 10838.983 ms Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 7627 0.62 MB/sec execute 58 sec latency 11839.170 ms oleg318-server: oleg318-server.virtnet: executing set_default_debug -1 all 1 7627 0.61 MB/sec execute 59 sec latency 12839.303 ms pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-MDT0000 01:08:21 (1708841301) targets are mounted 01:08:21 (1708841301) facet_failover done 1 7627 0.60 MB/sec execute 60 sec latency 13839.562 ms 1 7627 0.59 MB/sec execute 61 sec latency 14839.754 ms 1 7627 0.58 MB/sec execute 62 sec latency 15839.935 ms 1 7627 0.57 MB/sec execute 63 sec latency 16840.178 ms 1 7627 0.57 MB/sec execute 64 sec latency 17840.381 ms 1 7639 0.56 MB/sec execute 65 sec latency 18797.573 ms oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 7986 0.56 MB/sec execute 66 sec latency 18.789 ms 1 8237 0.57 MB/sec execute 67 sec latency 21.017 ms 1 8585 0.61 MB/sec execute 68 sec latency 22.250 ms 1 8829 0.60 MB/sec execute 69 sec latency 17.304 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 4888 1282800 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2896 1284792 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 21252 3563532 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 37716 3553276 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 58968 7116808 1% /mnt/lustre 1 9186 0.60 MB/sec execute 70 sec latency 12.928 ms 1 9405 0.61 MB/sec execute 71 sec latency 406.767 ms 1 9679 0.61 MB/sec execute 72 sec latency 19.831 ms 1 10117 0.67 MB/sec execute 73 sec latency 23.097 ms test_26 fail mds2 4 times Failing mds2 on oleg318-server Stopping /mnt/lustre-mds2 (opts:) on oleg318-server 1 10679 0.72 MB/sec execute 74 sec latency 22.914 ms 01:08:37 (1708841317) shut down 1 10728 0.72 MB/sec execute 75 sec latency 860.747 ms 1 10728 0.71 MB/sec execute 76 sec latency 1861.055 ms 1 10728 0.70 MB/sec execute 77 sec latency 2861.248 ms 1 10728 0.69 MB/sec execute 78 sec latency 3861.388 ms 1 10728 0.68 MB/sec execute 79 sec latency 4861.545 ms 1 10728 0.67 MB/sec execute 80 sec latency 5861.845 ms 1 10728 0.66 MB/sec execute 81 sec latency 6862.098 ms 1 10728 0.66 MB/sec execute 82 sec latency 7862.308 ms 1 10728 0.65 MB/sec execute 83 sec latency 8862.547 ms 1 10728 0.64 MB/sec execute 84 sec latency 9862.857 ms Failover mds2 to oleg318-server mount facets: mds2 1 10728 0.63 MB/sec execute 85 sec latency 10862.978 ms 1 10728 0.63 MB/sec execute 86 sec latency 11863.112 ms 1 10728 0.62 MB/sec execute 87 sec latency 12863.233 ms Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 1 10728 0.61 MB/sec execute 88 sec latency 13863.380 ms oleg318-server: oleg318-server.virtnet: executing set_default_debug -1 all pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-MDT0001 01:08:51 (1708841331) targets are mounted 01:08:51 (1708841331) facet_failover done 1 10728 0.60 MB/sec execute 89 sec latency 14863.536 ms 1 10728 0.60 MB/sec execute 90 sec latency 15863.692 ms 1 10728 0.59 MB/sec execute 91 sec latency 16863.914 ms 1 10728 0.59 MB/sec execute 92 sec latency 17864.084 ms 1 10728 0.58 MB/sec execute 93 sec latency 18864.250 ms oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1 10879 0.58 MB/sec execute 94 sec latency 19216.821 ms mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec 1 11039 0.58 MB/sec execute 95 sec latency 22.650 ms 1 11193 0.58 MB/sec execute 96 sec latency 27.805 ms 1 11438 0.57 MB/sec execute 97 sec latency 28.842 ms 1 11686 0.58 MB/sec execute 98 sec latency 21.123 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 5404 1282284 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3504 1284184 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 31148 3570364 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 47076 3554716 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 78224 7125080 2% /mnt/lustre 1 11941 0.58 MB/sec execute 99 sec latency 20.591 ms 1 cleanup 100 sec 0 cleanup 101 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 1895 30.301 19613.793 Close 1392 1.973 15.886 Rename 81 13.749 23.074 Unlink 379 55.511 19246.506 Qpathinfo 1740 2.768 50.755 Qfileinfo 301 0.428 3.442 Qfsinfo 311 0.890 8.187 Sfileinfo 159 7.360 26.050 Find 662 30.916 18797.552 WriteX 934 2.155 9.739 ReadX 2999 0.092 10.611 LockX 6 1.890 2.460 UnlockX 6 1.938 2.631 Flush 130 15.079 459.790 Throughput 0.580337 MB/sec 1 clients 1 procs max_latency=19613.807 ms stopping dbench on /mnt/lustre at Sun Feb 25 01:09:03 EST 2024 with return code 0 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished striped dir -i0 -c2 -H crush2 /mnt/lustre2/d26.replay-dual/run_dbench looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt test_26 fail mds1 5 times '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Sun Feb 25 01:09:04 EST 2024 waiting for dbench pid 10975 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients Failing mds1 on oleg318-server Stopping /mnt/lustre-mds1 (opts:) on oleg318-server 1 173 7.06 MB/sec warmup 1 sec latency 257.981 ms 1 173 3.53 MB/sec warmup 2 sec latency 1258.154 ms 1 173 2.35 MB/sec warmup 3 sec latency 2258.322 ms 1 173 1.77 MB/sec warmup 4 sec latency 3258.621 ms 1 173 1.41 MB/sec warmup 5 sec latency 4258.914 ms 1 173 1.18 MB/sec warmup 6 sec latency 5259.196 ms 1 173 1.01 MB/sec warmup 7 sec latency 6259.378 ms 01:09:11 (1708841351) shut down 1 173 0.88 MB/sec warmup 8 sec latency 7259.598 ms 1 173 0.78 MB/sec warmup 9 sec latency 8259.823 ms 1 173 0.71 MB/sec warmup 10 sec latency 9260.007 ms 1 173 0.64 MB/sec warmup 11 sec latency 10260.211 ms 1 173 0.59 MB/sec warmup 12 sec latency 11260.480 ms 1 173 0.54 MB/sec warmup 13 sec latency 12260.786 ms 1 173 0.50 MB/sec warmup 14 sec latency 13261.003 ms 1 173 0.47 MB/sec warmup 15 sec latency 14261.269 ms 1 173 0.44 MB/sec warmup 16 sec latency 15261.610 ms 1 173 0.42 MB/sec warmup 17 sec latency 16261.985 ms Failover mds1 to oleg318-server mount facets: mds1 1 173 0.39 MB/sec warmup 18 sec latency 17262.170 ms 1 173 0.37 MB/sec warmup 19 sec latency 18262.319 ms Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 173 0.00 MB/sec execute 1 sec latency 20262.649 ms oleg318-server: oleg318-server.virtnet: executing set_default_debug -1 all pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-MDT0000 01:09:26 (1708841366) targets are mounted 01:09:26 (1708841366) facet_failover done 1 173 0.00 MB/sec execute 2 sec latency 21262.835 ms 1 173 0.00 MB/sec execute 3 sec latency 22262.994 ms 1 173 0.00 MB/sec execute 4 sec latency 23263.179 ms 1 173 0.00 MB/sec execute 5 sec latency 24263.312 ms 1 173 0.00 MB/sec execute 6 sec latency 25263.447 ms oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 268 0.43 MB/sec execute 7 sec latency 25921.464 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 486 1.24 MB/sec execute 8 sec latency 23.319 ms 1 744 1.58 MB/sec execute 9 sec latency 41.728 ms 1 988 1.48 MB/sec execute 10 sec latency 24.170 ms 1 1221 1.46 MB/sec execute 11 sec latency 18.876 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 4860 1282828 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3432 1284256 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 45576 3549636 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 13672 3582216 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 59248 7131852 1% /mnt/lustre 1 1508 1.58 MB/sec execute 12 sec latency 27.629 ms 1 1731 1.48 MB/sec execute 13 sec latency 25.076 ms 1 1982 1.39 MB/sec execute 14 sec latency 36.642 ms 1 2305 1.38 MB/sec execute 15 sec latency 15.848 ms test_26 fail mds2 6 times 1 2508 1.35 MB/sec execute 16 sec latency 21.607 ms Failing mds2 on oleg318-server Stopping /mnt/lustre-mds2 (opts:) on oleg318-server 1 2922 1.55 MB/sec execute 17 sec latency 90.366 ms 01:09:42 (1708841382) shut down 1 2922 1.47 MB/sec execute 18 sec latency 1090.565 ms 1 2922 1.39 MB/sec execute 19 sec latency 2090.784 ms 1 2922 1.32 MB/sec execute 20 sec latency 3090.953 ms 1 2922 1.26 MB/sec execute 21 sec latency 4091.121 ms 1 2922 1.20 MB/sec execute 22 sec latency 5091.312 ms 1 2922 1.15 MB/sec execute 23 sec latency 6091.437 ms 1 2922 1.10 MB/sec execute 24 sec latency 7091.563 ms 1 2922 1.06 MB/sec execute 25 sec latency 8091.697 ms 1 2922 1.02 MB/sec execute 26 sec latency 9091.855 ms 1 2922 0.98 MB/sec execute 27 sec latency 10092.003 ms Failover mds2 to oleg318-server mount facets: mds2 1 2922 0.94 MB/sec execute 28 sec latency 11092.137 ms 1 2922 0.91 MB/sec execute 29 sec latency 12092.403 ms 1 2922 0.88 MB/sec execute 30 sec latency 13092.591 ms Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 1 2922 0.85 MB/sec execute 31 sec latency 14092.797 ms oleg318-server: oleg318-server.virtnet: executing set_default_debug -1 all pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-MDT0001 01:09:56 (1708841396) targets are mounted 01:09:56 (1708841396) facet_failover done 1 2922 0.83 MB/sec execute 32 sec latency 15092.957 ms 1 2922 0.80 MB/sec execute 33 sec latency 16093.080 ms 1 2922 0.78 MB/sec execute 34 sec latency 17093.255 ms 1 2922 0.75 MB/sec execute 35 sec latency 18093.413 ms 1 2932 0.73 MB/sec execute 36 sec latency 19025.586 ms oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec 1 3501 0.85 MB/sec execute 37 sec latency 15.926 ms 1 3781 0.86 MB/sec execute 38 sec latency 16.883 ms 1 3958 0.85 MB/sec execute 39 sec latency 23.740 ms 1 4176 0.83 MB/sec execute 40 sec latency 20.771 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 3080 1284608 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2616 1285072 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 48092 3552276 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 15460 3580840 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 63552 7133116 1% /mnt/lustre 1 4428 0.82 MB/sec execute 41 sec latency 68.765 ms 1 4549 0.81 MB/sec execute 42 sec latency 392.636 ms 1 4780 0.82 MB/sec execute 43 sec latency 485.381 ms 1 5058 0.87 MB/sec execute 44 sec latency 37.094 ms test_26 fail mds1 7 times Failing mds1 on oleg318-server 1 5340 0.85 MB/sec execute 45 sec latency 15.743 ms Stopping /mnt/lustre-mds1 (opts:) on oleg318-server 01:10:10 (1708841410) shut down 1 5402 0.83 MB/sec execute 46 sec latency 709.574 ms 1 5402 0.82 MB/sec execute 47 sec latency 1709.759 ms 1 5402 0.80 MB/sec execute 48 sec latency 2709.933 ms 1 5402 0.78 MB/sec execute 49 sec latency 3710.156 ms 1 5402 0.77 MB/sec execute 50 sec latency 4710.365 ms 1 5402 0.75 MB/sec execute 51 sec latency 5710.598 ms 1 5402 0.74 MB/sec execute 52 sec latency 6710.819 ms 1 5402 0.72 MB/sec execute 53 sec latency 7711.024 ms 1 5402 0.71 MB/sec execute 54 sec latency 8711.183 ms 1 5402 0.70 MB/sec execute 55 sec latency 9711.410 ms 1 5402 0.69 MB/sec execute 56 sec latency 10711.589 ms Failover mds1 to oleg318-server mount facets: mds1 1 5402 0.67 MB/sec execute 57 sec latency 11711.796 ms 1 5402 0.66 MB/sec execute 58 sec latency 12712.008 ms Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 5402 0.65 MB/sec execute 59 sec latency 13712.190 ms oleg318-server: oleg318-server.virtnet: executing set_default_debug -1 all pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 1 5402 0.64 MB/sec execute 60 sec latency 14712.501 ms Started lustre-MDT0000 01:10:24 (1708841424) targets are mounted 01:10:24 (1708841424) facet_failover done 1 5402 0.63 MB/sec execute 61 sec latency 15712.687 ms 1 5402 0.62 MB/sec execute 62 sec latency 16712.931 ms 1 5402 0.61 MB/sec execute 63 sec latency 17713.140 ms 1 5402 0.60 MB/sec execute 64 sec latency 18713.320 ms oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 5551 0.59 MB/sec execute 65 sec latency 19297.247 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec striped dir -i0 -c2 -H all_char /mnt/lustre/d26.replay-dual/run_tar 1 5893 0.60 MB/sec execute 66 sec latency 17.395 ms 1 6115 0.61 MB/sec execute 67 sec latency 34.511 ms 1 6544 0.67 MB/sec execute 68 sec latency 19.507 ms tar: Unexpected EOF in archive tar: Unexpected EOF in archive tar: Error is not recoverable: exiting now dbench killed by signal 15 stopping dbench on /mnt/lustre at Sun Feb 25 01:10:33 EST 2024 with return code 0 10975 pts/0 S+ 0:00 dbench -c client.txt 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100 killed dbench main pid 10975 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished PASS 26 (216s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-dual test 28: lock replay should be ordered: waiting after granted ========================================================== 01:10:38 (1708841438) 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00308744 s, 1.3 MB/s fail_loc=0x80000324 fail_loc=0x32a Failing ost1 on oleg318-server Stopping /mnt/lustre-ost1 (opts:) on oleg318-server 01:10:42 (1708841442) shut down Failover ost1 to oleg318-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg318-server: oleg318-server.virtnet: executing set_default_debug -1 all pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-OST0000 01:10:56 (1708841456) targets are mounted 01:10:56 (1708841456) facet_failover done 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00502744 s, 815 kB/s oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec PASS 28 (25s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-dual test 29: replay vs update with the same xid ========================================================== 01:11:05 (1708841465) SKIP: replay-dual test_29 needs >= 2 clients SKIP 29 (1s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-dual test 30: layout lock replay is not blocked on IO ========================================================== 01:11:08 (1708841468) 10+0 records in 10+0 records out 40960 bytes (41 kB) copied, 0.0124361 s, 3.3 MB/s 10+0 records in 10+0 records out 40960 bytes (41 kB) copied, 0.0126804 s, 3.2 MB/s fail_loc=0x32e fail_val=4 Failing mds1 on oleg318-server Stopping /mnt/lustre-mds1 (opts:) on oleg318-server 01:11:11 (1708841471) shut down Failover mds1 to oleg318-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg318-server: oleg318-server.virtnet: executing set_default_debug -1 all pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-MDT0000 01:11:25 (1708841485) targets are mounted 01:11:25 (1708841485) facet_failover done 160+0 records in 160+0 records out 81920 bytes (82 kB) copied, 20.3999 s, 4.0 kB/s oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 30 (25s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-dual test 31: deadlock on file_remove_privs and occupied mod rpc slots ========================================================== 01:11:35 (1708841495) Failing ost1 on oleg318-server Stopping /mnt/lustre-ost1 (opts:) on oleg318-server 01:11:37 (1708841497) shut down Creating to objid 3041 on ost lustre-OST0000... total: 32 open/close in 0.24 seconds: 133.44 ops/second at_max=0 fail_loc=0x80001420 file /mnt/lustre2/d31.replay-dual/mdtdir/f31.replay-dual is ready Failover ost1 to oleg318-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg318-server: oleg318-server.virtnet: executing set_default_debug -1 all pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-OST0000 01:11:51 (1708841511) targets are mounted 01:11:51 (1708841511) facet_failover done oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE FULL state after 0 sec pids: 18772 18773 18779 18780 18781 18782 18783 18784 18785 at_max=600 PASS 31 (23s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-dual test 32: gap in update llog shouldn't break recovery ========================================================== 01:12:00 (1708841520) fail_loc=0x0000131d fail_val=10 fail_loc=0x726 Stopping /mnt/lustre-mds2 (opts:) on oleg318-server Stopping /mnt/lustre-mds1 (opts:) on oleg318-server fail_loc=0 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg318-server: oleg318-server.virtnet: executing set_default_debug -1 all pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-MDT0000 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg318-server: oleg318-server.virtnet: executing set_default_debug -1 all pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-MDT0001 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2436 1285224 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1996 1285692 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1820 3605200 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1796 3605224 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3616 7210424 1% /mnt/lustre PASS 32 (17s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-dual test 33: Check for OBD_INCOMPAT_MULTI_RPCS in last_rcvd after abort_recovery ========================================================== 01:12:19 (1708841539) at_min=60 Stopping /mnt/lustre-mds2 (opts:) on oleg318-server last_rcvd in /dev/mapper/mds2_flakey: incompat = 0x61c Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg318-server: oleg318-server.virtnet: executing set_default_debug -1 all pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg318-client.virtnet: executing wait_import_state_mount REPLAY_WAIT mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in REPLAY_WAIT state after 0 sec oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec affected facets: mds2 oleg318-server: oleg318-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0001.recovery_status 1475 oleg318-server: *.lustre-MDT0001.recovery_status status: COMPLETE Stopping /mnt/lustre-mds2 (opts:) on oleg318-server last_rcvd in /dev/mapper/mds2_flakey: incompat = 0x61c Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg318-server: oleg318-server.virtnet: executing set_default_debug -1 all pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-MDT0001 Starting client: oleg318-client.virtnet: -o user_xattr,flock oleg318-server@tcp:/lustre /mnt/lustre2 oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL DISCONN state after 3 sec affected facets: mds2 oleg318-server: oleg318-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0001.recovery_status 1475 oleg318-server: *.lustre-MDT0001.recovery_status status: COMPLETE at_min=5 PASS 33 (36s) debug_raw_pointers=0 debug_raw_pointers=0 == replay-dual test complete, duration 2692 sec ========== 01:12:55 (1708841575) === replay-dual: start cleanup 01:12:56 (1708841576) === Stopping clients: oleg318-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg318-client.virtnet /mnt/lustre2 opts: === replay-dual: finish cleanup 01:12:57 (1708841577) ===