-----============= acceptance-small: replay-dual ============----- Thu Apr 4 20:05:46 EDT 2024 excepting tests: 14b 21b skipping tests SLOW=no: 21b === replay-dual: start setup 20:05:48 (1712275548) === Starting client oleg354-client.virtnet: -o user_xattr,flock oleg354-server@tcp:/lustre /mnt/lustre2 Started clients oleg354-client.virtnet: 192.168.203.154@tcp:/lustre on /mnt/lustre2 type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) oleg354-client.virtnet: executing check_config_client /mnt/lustre oleg354-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg354-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800b7c0d800.idle_timeout=debug osc.lustre-OST0000-osc-ffff880136907800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800b7c0d800.idle_timeout=debug osc.lustre-OST0001-osc-ffff880136907800.idle_timeout=debug disable quota as required oleg354-server: oleg354-server.virtnet: executing set_default_debug -1 all osd-ldiskfs.track_declares_assert=1 === replay-dual: finish setup 20:05:56 (1712275556) === osp.lustre-OST0000-osc-MDT0000.prealloc_force_new_seq=1 osp.lustre-OST0000-osc-MDT0001.prealloc_force_new_seq=1 osp.lustre-OST0001-osc-MDT0000.prealloc_force_new_seq=1 osp.lustre-OST0001-osc-MDT0001.prealloc_force_new_seq=1 Creating to objid 33 on ost lustre-OST0000... Creating to objid 33 on ost lustre-OST0001... Creating to objid 33 on ost lustre-OST0000... Creating to objid 33 on ost lustre-OST0001... total: 33 open/close in 0.23 seconds: 146.62 ops/second total: 33 open/close in 0.23 seconds: 143.69 ops/second total: 33 open/close in 0.24 seconds: 138.83 ops/second total: 33 open/close in 0.23 seconds: 146.09 ops/second osp.lustre-OST0000-osc-MDT0000.prealloc_force_new_seq=0 osp.lustre-OST0001-osc-MDT0000.prealloc_force_new_seq=0 osp.lustre-OST0000-osc-MDT0001.prealloc_force_new_seq=0 osp.lustre-OST0001-osc-MDT0001.prealloc_force_new_seq=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-dual test 0a: expired recovery with lost client ========================================================== 20:06:09 (1712275569) Check file is LU482_FAILED=/tmp/replay-dual.lu482.bgJKeV UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1840 1285848 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1688 1286000 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1796 3605224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1796 3605224 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3592 7210448 1% /mnt/lustre total: 50 open/close in 0.38 seconds: 130.16 ops/second fail_loc=0x80000514 Failing mds1 on oleg354-server Stopping /mnt/lustre-mds1 (opts:) on oleg354-server 20:06:13 (1712275573) shut down Failover mds1 to oleg354-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg354-server: oleg354-server.virtnet: executing set_default_debug -1 all pdsh@oleg354-client: oleg354-server: ssh exited with exit code 1 Started lustre-MDT0000 20:06:27 (1712275587) targets are mounted 20:06:27 (1712275587) facet_failover done Starting client: oleg354-client.virtnet: -o user_xattr,flock oleg354-server@tcp:/lustre /mnt/lustre2 - unlinked 0 (time 1712275669 ; total 0 ; last 0) total: 50 unlinks in 1 seconds: 50.000000 unlinks/second PASS 0a (102s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-dual test 0b: lost client during waiting for next transno ========================================================== 20:07:53 (1712275673) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1840 1285848 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1688 1286000 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1796 3605224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1796 3605224 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3592 7210448 1% /mnt/lustre Failing mds1 on oleg354-server Stopping /mnt/lustre-mds1 (opts:) on oleg354-server 20:07:56 (1712275676) shut down Failover mds1 to oleg354-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg354-server: oleg354-server.virtnet: executing set_default_debug -1 all pdsh@oleg354-client: oleg354-server: ssh exited with exit code 1 Started lustre-MDT0000 20:08:11 (1712275691) targets are mounted 20:08:11 (1712275691) facet_failover done Starting client: oleg354-client.virtnet: -o user_xattr,flock oleg354-server@tcp:/lustre /mnt/lustre Starting client: oleg354-client.virtnet: -o user_xattr,flock oleg354-server@tcp:/lustre /mnt/lustre2 PASS 0b (119s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-dual test 1: |X| simple create ================= 20:09:54 (1712275794) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1840 1285848 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1688 1286000 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1796 3605224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1796 3605224 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3592 7210448 1% /mnt/lustre mcreate: cannot create `/mnt/lustre/fsa-oleg354-client.virtnet' with mode 0100644: File exists Failing mds1 on oleg354-server Stopping /mnt/lustre-mds1 (opts:) on oleg354-server 20:09:57 (1712275797) shut down Failover mds1 to oleg354-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg354-server: oleg354-server.virtnet: executing set_default_debug -1 all pdsh@oleg354-client: oleg354-server: ssh exited with exit code 1 Started lustre-MDT0000 20:10:11 (1712275811) targets are mounted 20:10:11 (1712275811) facet_failover done oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 1 (24s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-dual test 2: |X| mkdir adir ==================== 20:10:20 (1712275820) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1840 1285848 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1688 1286000 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1796 3605224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1796 3605224 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3592 7210448 1% /mnt/lustre Failing mds1 on oleg354-server Stopping /mnt/lustre-mds1 (opts:) on oleg354-server 20:10:23 (1712275823) shut down Failover mds1 to oleg354-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg354-server: oleg354-server.virtnet: executing set_default_debug -1 all pdsh@oleg354-client: oleg354-server: ssh exited with exit code 1 Started lustre-MDT0000 20:10:37 (1712275837) targets are mounted 20:10:37 (1712275837) facet_failover done oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 2 (24s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-dual test 3: |X| mkdir adir, mkdir adir/bdir === 20:10:46 (1712275846) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1840 1285848 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1688 1286000 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1796 3605224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1796 3605224 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3592 7210448 1% /mnt/lustre Failing mds1 on oleg354-server Stopping /mnt/lustre-mds1 (opts:) on oleg354-server 20:10:49 (1712275849) shut down Failover mds1 to oleg354-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg354-server: oleg354-server.virtnet: executing set_default_debug -1 all pdsh@oleg354-client: oleg354-server: ssh exited with exit code 1 Started lustre-MDT0000 20:11:03 (1712275863) targets are mounted 20:11:03 (1712275863) facet_failover done oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 3 (24s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-dual test 4: |X| mkdir adir (-EEXIST), mkdir adir/bdir ========================================================== 20:11:12 (1712275872) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1916 1285772 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1796 1285892 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1796 3605224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1796 3605224 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3592 7210448 1% /mnt/lustre mkdir: cannot create directory '/mnt/lustre/adir': File exists Failing mds1 on oleg354-server Stopping /mnt/lustre-mds1 (opts:) on oleg354-server 20:11:20 (1712275880) shut down Failover mds1 to oleg354-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg354-server: oleg354-server.virtnet: executing set_default_debug -1 all pdsh@oleg354-client: oleg354-server: ssh exited with exit code 1 Started lustre-MDT0000 20:11:34 (1712275894) targets are mounted 20:11:34 (1712275894) facet_failover done oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 4 (29s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-dual test 5: open, unlink |X| close ============ 20:11:43 (1712275903) multiop /mnt/lustre2/a vo_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7363 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1916 1285772 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1800 1285888 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1796 3605224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1796 3605224 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3592 7210448 1% /mnt/lustre Failing mds1 on oleg354-server Stopping /mnt/lustre-mds1 (opts:) on oleg354-server 20:11:46 (1712275906) shut down Failover mds1 to oleg354-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg354-server: oleg354-server.virtnet: executing set_default_debug -1 all pdsh@oleg354-client: oleg354-server: ssh exited with exit code 1 Started lustre-MDT0000 20:12:00 (1712275920) targets are mounted 20:12:00 (1712275920) facet_failover done oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 5 (24s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-dual test 6: open1, open2, unlink |X| close1 [fail mds1] close2 ========================================================== 20:12:09 (1712275929) multiop /mnt/lustre2/a vo_c TMPPIPE=/tmp/multiop_open_wait_pipe.7363 multiop /mnt/lustre/a vo_c TMPPIPE=/tmp/multiop_open_wait_pipe.7363 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1916 1285772 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1800 1285888 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1796 3605224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1796 3605224 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3592 7210448 1% /mnt/lustre Failing mds1 on oleg354-server Stopping /mnt/lustre-mds1 (opts:) on oleg354-server 20:12:12 (1712275932) shut down Failover mds1 to oleg354-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg354-server: oleg354-server.virtnet: executing set_default_debug -1 all pdsh@oleg354-client: oleg354-server: ssh exited with exit code 1 Started lustre-MDT0000 20:12:26 (1712275946) targets are mounted 20:12:26 (1712275946) facet_failover done oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 6 (24s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-dual test 8: replay of resent request ========== 20:12:34 (1712275954) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1916 1285772 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1800 1285888 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1796 3605224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1796 3605224 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3592 7210448 1% /mnt/lustre fail_loc=0x119 fail_loc=0 Failing mds1 on oleg354-server Stopping /mnt/lustre-mds1 (opts:) on oleg354-server 20:12:54 (1712275974) shut down Failover mds1 to oleg354-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg354-server: oleg354-server.virtnet: executing set_default_debug -1 all pdsh@oleg354-client: oleg354-server: ssh exited with exit code 1 Started lustre-MDT0000 20:13:08 (1712275988) targets are mounted 20:13:08 (1712275988) facet_failover done oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 8 (41s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-dual test 9: resending a replayed create ======= 20:13:16 (1712275996) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1916 1285772 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1800 1285888 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1796 3605224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1796 3605224 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3592 7210448 1% /mnt/lustre fail_loc=0x80000119 Failing mds1 on oleg354-server Stopping /mnt/lustre-mds1 (opts:) on oleg354-server 20:13:20 (1712276000) shut down Failover mds1 to oleg354-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg354-server: oleg354-server.virtnet: executing set_default_debug -1 all pdsh@oleg354-client: oleg354-server: ssh exited with exit code 1 Started lustre-MDT0000 20:13:34 (1712276014) targets are mounted 20:13:34 (1712276014) facet_failover done oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec fail_loc=0 PASS 9 (36s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-dual test 10: resending a replayed unlink ====== 20:13:54 (1712276034) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1916 1285772 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1800 1285888 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1796 3605224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1796 3605224 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3592 7210448 1% /mnt/lustre fail_loc=0x80000119 Failing mds1 on oleg354-server Stopping /mnt/lustre-mds1 (opts:) on oleg354-server 20:13:58 (1712276038) shut down Failover mds1 to oleg354-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg354-server: oleg354-server.virtnet: executing set_default_debug -1 all pdsh@oleg354-client: oleg354-server: ssh exited with exit code 1 Started lustre-MDT0000 20:14:11 (1712276051) targets are mounted 20:14:11 (1712276051) facet_failover done oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec fail_loc=0 PASS 10 (39s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-dual test 11: both clients timeout during replay ========================================================== 20:14:34 (1712276074) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1916 1285772 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1800 1285888 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1796 3605224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1796 3605224 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3592 7210448 1% /mnt/lustre fail_loc=0x0119 Failing mds1 on oleg354-server Stopping /mnt/lustre-mds1 (opts:) on oleg354-server 20:14:38 (1712276078) shut down Failover mds1 to oleg354-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg354-server: oleg354-server.virtnet: executing set_default_debug -1 all pdsh@oleg354-client: oleg354-server: ssh exited with exit code 1 Started lustre-MDT0000 20:14:52 (1712276092) targets are mounted 20:14:52 (1712276092) facet_failover done oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 17 sec fail_loc=0 PASS 11 (38s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-dual test 12: open resend timeout ============== 20:15:13 (1712276113) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1916 1285772 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1800 1285888 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1796 3605224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1796 3605224 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3592 7210448 1% /mnt/lustre multiop /mnt/lustre/f12.replay-dual vmo_c TMPPIPE=/tmp/multiop_open_wait_pipe.7363 fail_loc=0x80000302 Failing mds1 on oleg354-server Stopping /mnt/lustre-mds1 (opts:) on oleg354-server 20:15:17 (1712276117) shut down Failover mds1 to oleg354-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg354-server: oleg354-server.virtnet: executing set_default_debug -1 all pdsh@oleg354-client: oleg354-server: ssh exited with exit code 1 Started lustre-MDT0000 20:15:31 (1712276131) targets are mounted 20:15:31 (1712276131) facet_failover done fail_loc=0 /mnt/lustre/f12.replay-dual /mnt/lustre/f12.replay-dual has type file OK PASS 12 (23s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-dual test 13: close resend timeout ============= 20:15:37 (1712276137) multiop /mnt/lustre/f13.replay-dual vmo_c TMPPIPE=/tmp/multiop_open_wait_pipe.7363 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1916 1285772 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1800 1285888 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1796 3605224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1796 3605224 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3592 7210448 1% /mnt/lustre fail_loc=0x80000115 Failing mds1 on oleg354-server Stopping /mnt/lustre-mds1 (opts:) on oleg354-server 20:15:41 (1712276141) shut down Failover mds1 to oleg354-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg354-server: oleg354-server.virtnet: executing set_default_debug -1 all pdsh@oleg354-client: oleg354-server: ssh exited with exit code 1 Started lustre-MDT0000 20:15:55 (1712276155) targets are mounted 20:15:55 (1712276155) facet_failover done fail_loc=0 /mnt/lustre/f13.replay-dual /mnt/lustre/f13.replay-dual has type file OK PASS 13 (23s) debug_raw_pointers=0 debug_raw_pointers=0 SKIP: replay-dual test_14b skipping ALWAYS excluded test 14b debug_raw_pointers=Y debug_raw_pointers=Y == replay-dual test 15a: timeout waiting for lost client during replay, 1 client completes ========================================================== 20:16:02 (1712276162) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1916 1285772 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1800 1285888 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1796 3605224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1796 3605224 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3592 7210448 1% /mnt/lustre total: 25 open/close in 0.18 seconds: 140.72 ops/second total: 1 open/close in 0.01 seconds: 128.88 ops/second Failing mds1 on oleg354-server Stopping /mnt/lustre-mds1 (opts:) on oleg354-server 20:16:05 (1712276165) shut down Failover mds1 to oleg354-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg354-server: oleg354-server.virtnet: executing set_default_debug -1 all pdsh@oleg354-client: oleg354-server: ssh exited with exit code 1 Started lustre-MDT0000 20:16:19 (1712276179) targets are mounted 20:16:19 (1712276179) facet_failover done oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec - unlinked 0 (time 1712276250 ; total 0 ; last 0) total: 25 unlinks in 0 seconds: inf unlinks/second Starting client: oleg354-client.virtnet: -o user_xattr,flock oleg354-server@tcp:/lustre /mnt/lustre2 PASS 15a (90s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-dual test 15c: remove multiple OST orphans ===== 20:17:33 (1712276253) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1916 1285772 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1800 1285888 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1796 3605224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1796 3605224 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3592 7210448 1% /mnt/lustre Failing mds1 on oleg354-server Stopping /mnt/lustre-mds1 (opts:) on oleg354-server 20:18:06 (1712276286) shut down Failover mds1 to oleg354-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg354-server: oleg354-server.virtnet: executing set_default_debug -1 all pdsh@oleg354-client: oleg354-server: ssh exited with exit code 1 Started lustre-MDT0000 20:18:20 (1712276300) targets are mounted 20:18:20 (1712276300) facet_failover done oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Starting client: oleg354-client.virtnet: -o user_xattr,flock oleg354-server@tcp:/lustre /mnt/lustre2 PASS 15c (121s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-dual test 16: fail MDS during recovery (3571) == 20:19:36 (1712276376) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1916 1285772 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1800 1285888 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1796 3605224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1796 3605224 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3592 7210448 1% /mnt/lustre total: 25 open/close in 0.17 seconds: 148.01 ops/second total: 1 open/close in 0.01 seconds: 154.41 ops/second Failing mds1 on oleg354-server Stopping /mnt/lustre-mds1 (opts:) on oleg354-server 20:19:39 (1712276379) shut down Failover mds1 to oleg354-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg354-server: oleg354-server.virtnet: executing set_default_debug -1 all pdsh@oleg354-client: oleg354-server: ssh exited with exit code 1 Started lustre-MDT0000 20:19:53 (1712276393) targets are mounted 20:19:53 (1712276393) facet_failover done Failing mds1 on oleg354-server Stopping /mnt/lustre-mds1 (opts:) on oleg354-server 20:20:15 (1712276415) shut down Failover mds1 to oleg354-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg354-server: oleg354-server.virtnet: executing set_default_debug -1 all pdsh@oleg354-client: oleg354-server: ssh exited with exit code 1 Started lustre-MDT0000 20:20:28 (1712276428) targets are mounted 20:20:28 (1712276428) facet_failover done oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec - unlinked 0 (time 1712276502 ; total 0 ; last 0) total: 25 unlinks in 0 seconds: inf unlinks/second Starting client: oleg354-client.virtnet: -o user_xattr,flock oleg354-server@tcp:/lustre /mnt/lustre2 PASS 16 (127s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-dual test 17: fail OST during recovery (3571) == 20:21:45 (1712276505) total: 25 open/close in 0.20 seconds: 127.15 ops/second total: 1 open/close in 0.01 seconds: 107.37 ops/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1988 1285700 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1800 1285888 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1796 3605224 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1796 3605224 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3592 7210448 1% /mnt/lustre Failing ost1 on oleg354-server Stopping /mnt/lustre-ost1 (opts:) on oleg354-server 20:21:49 (1712276509) shut down Failover ost1 to oleg354-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg354-server: oleg354-server.virtnet: executing set_default_debug -1 all pdsh@oleg354-client: oleg354-server: ssh exited with exit code 1 Started lustre-OST0000 20:22:03 (1712276523) targets are mounted 20:22:03 (1712276523) facet_failover done Failing ost1 on oleg354-server Stopping /mnt/lustre-ost1 (opts:) on oleg354-server 20:22:24 (1712276544) shut down Failover ost1 to oleg354-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg354-server: oleg354-server.virtnet: executing set_default_debug -1 all pdsh@oleg354-client: oleg354-server: ssh exited with exit code 1 Started lustre-OST0000 20:22:38 (1712276558) targets are mounted 20:22:38 (1712276558) facet_failover done oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec - unlinked 0 (time 1712276633 ; total 0 ; last 0) total: 25 unlinks in 0 seconds: inf unlinks/second Starting client: oleg354-client.virtnet: -o user_xattr,flock oleg354-server@tcp:/lustre /mnt/lustre2 PASS 17 (130s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-dual test 18: ldlm_handle_enqueue succeeds on evicted export (3822) ========================================================== 20:23:57 (1712276637) debug=+dlmtrace using seed 1024837996 running for 500 iterations total: 500 stats in 0 seconds: inf stats/second fail_loc=0x8000030b ldlm.namespaces.MGC192.168.203.154@tcp.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8cbe000.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0000-mdc-ffff8800ab000800.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8cbe000.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0001-mdc-ffff8800ab000800.early_lock_cancel=0 ldlm.namespaces.lustre-OST0000-osc-ffff8800a8cbe000.early_lock_cancel=0 ldlm.namespaces.lustre-OST0000-osc-ffff8800ab000800.early_lock_cancel=0 ldlm.namespaces.lustre-OST0001-osc-ffff8800a8cbe000.early_lock_cancel=0 ldlm.namespaces.lustre-OST0001-osc-ffff8800ab000800.early_lock_cancel=0 fail_loc=0x80000305 Error in opening file "/mnt/lustre2/d18.replay-dual/f18.replay-dual"(flags=O_RDONLY) 2: No such file or directory ldlm.namespaces.MGC192.168.203.154@tcp.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a8cbe000.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0000-mdc-ffff8800ab000800.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0001-mdc-ffff8800a8cbe000.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0001-mdc-ffff8800ab000800.early_lock_cancel=1 ldlm.namespaces.lustre-OST0000-osc-ffff8800a8cbe000.early_lock_cancel=1 ldlm.namespaces.lustre-OST0000-osc-ffff8800ab000800.early_lock_cancel=1 ldlm.namespaces.lustre-OST0001-osc-ffff8800a8cbe000.early_lock_cancel=1 ldlm.namespaces.lustre-OST0001-osc-ffff8800ab000800.early_lock_cancel=1 fail_loc=0 fail_loc=0 PASS 18 (45s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-dual test 19: resend of open request =========== 20:24:44 (1712276684) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1992 1285696 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1800 1285888 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1812 3605208 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1796 3605224 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3608 7210432 1% /mnt/lustre fail_loc=0x157 - open/close 0 (time 1712276772.95 total 86.04 last 0.00) total: 1 open/close in 86.04 seconds: 0.01 ops/second fail_loc=0 Failing mds1 on oleg354-server Stopping /mnt/lustre-mds1 (opts:) on oleg354-server 20:26:14 (1712276774) shut down Failover mds1 to oleg354-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg354-server: oleg354-server.virtnet: executing set_default_debug -1 all pdsh@oleg354-client: oleg354-server: ssh exited with exit code 1 Started lustre-MDT0000 20:26:29 (1712276789) targets are mounted 20:26:29 (1712276789) facet_failover done oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 19 (113s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-dual test 20: recovery time is not increasing == 20:26:39 (1712276799) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1992 1285696 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1800 1285888 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1812 3605208 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1796 3605224 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3608 7210432 1% /mnt/lustre Failing mds1 on oleg354-server Stopping /mnt/lustre-mds1 (opts:) on oleg354-server 20:26:43 (1712276803) shut down Failover mds1 to oleg354-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg354-server: oleg354-server.virtnet: executing set_default_debug -1 all pdsh@oleg354-client: oleg354-server: ssh exited with exit code 1 Started lustre-MDT0000 20:26:58 (1712276818) targets are mounted 20:26:58 (1712276818) facet_failover done oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Starting client: oleg354-client.virtnet: -o user_xattr,flock oleg354-server@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1992 1285696 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1800 1285888 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1812 3605208 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1796 3605224 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3608 7210432 1% /mnt/lustre Failing mds1 on oleg354-server Stopping /mnt/lustre-mds1 (opts:) on oleg354-server 20:29:24 (1712276964) shut down Failover mds1 to oleg354-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg354-server: oleg354-server.virtnet: executing set_default_debug -1 all pdsh@oleg354-client: oleg354-server: ssh exited with exit code 1 Started lustre-MDT0000 20:29:40 (1712276980) targets are mounted 20:29:40 (1712276980) facet_failover done oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Starting client: oleg354-client.virtnet: -o user_xattr,flock oleg354-server@tcp:/lustre /mnt/lustre2 PASS 20 (325s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-dual test 21a: commit on sharing =============== 20:32:06 (1712277126) mdt.lustre-MDT0000.commit_on_sharing=1 mdt.lustre-MDT0001.commit_on_sharing=1 Replay barrier on lustre-MDT0000 Failing mds1 on oleg354-server Stopping /mnt/lustre-mds1 (opts:) on oleg354-server 20:32:11 (1712277131) shut down Failover mds1 to oleg354-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg354-server: oleg354-server.virtnet: executing set_default_debug -1 all pdsh@oleg354-client: oleg354-server: ssh exited with exit code 1 Started lustre-MDT0000 20:32:25 (1712277145) targets are mounted 20:32:25 (1712277145) facet_failover done Starting client: oleg354-client.virtnet: -o user_xattr,flock oleg354-server@tcp:/lustre /mnt/lustre2 mdt.lustre-MDT0000.commit_on_sharing=0 mdt.lustre-MDT0001.commit_on_sharing=0 PASS 21a (163s) debug_raw_pointers=0 debug_raw_pointers=0 SKIP: replay-dual test_21b skipping SLOW test 21b debug_raw_pointers=Y debug_raw_pointers=Y == replay-dual test 22a: c1 lfs mkdir -i 1 dir1, M1 drop reply & fail, c2 mkdir dir1/dir ========================================================== 20:34:51 (1712277291) fail_loc=0x119 Failing mds2 on oleg354-server Stopping /mnt/lustre-mds2 (opts:) on oleg354-server 20:34:54 (1712277294) shut down Failover mds2 to oleg354-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg354-server: oleg354-server.virtnet: executing set_default_debug -1 all pdsh@oleg354-client: oleg354-server: ssh exited with exit code 1 Started lustre-MDT0001 20:35:08 (1712277308) targets are mounted 20:35:08 (1712277308) facet_failover done oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2000 1285688 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1824 1285864 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1812 3605208 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1796 3605224 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3608 7210432 1% /mnt/lustre total: 2 open/close in 0.03 seconds: 64.34 ops/second total: 2 open/close in 0.02 seconds: 122.85 ops/second Failing mds1 on oleg354-server Stopping /mnt/lustre-mds1 (opts:) on oleg354-server 20:35:17 (1712277317) shut down Failover mds1 to oleg354-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg354-server: oleg354-server.virtnet: executing set_default_debug -1 all pdsh@oleg354-client: oleg354-server: ssh exited with exit code 1 Started lustre-MDT0000 20:35:32 (1712277332) targets are mounted 20:35:32 (1712277332) facet_failover done oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 22a (49s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-dual test 22b: c1 lfs mkdir -i 1 d1, M1 drop reply & fail M0/M1, c2 mkdir d1/dir ========================================================== 20:35:42 (1712277342) fail_loc=0x119 Failing mds1 on oleg354-server Stopping /mnt/lustre-mds1 (opts:) on oleg354-server Failing mds2 on oleg354-server Stopping /mnt/lustre-mds2 (opts:) on oleg354-server 20:35:46 (1712277346) shut down Failover mds1 to oleg354-server mount facets: mds1 Failover mds2 to oleg354-server mount facets: mds2 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg354-server: oleg354-server.virtnet: executing set_default_debug -1 all oleg354-server: oleg354-server.virtnet: executing set_default_debug -1 all pdsh@oleg354-client: oleg354-server: ssh exited with exit code 1 pdsh@oleg354-client: oleg354-server: ssh exited with exit code 1 Started lustre-MDT0001 Started lustre-MDT0000 20:36:12 (1712277372) targets are mounted 20:36:12 (1712277372) facet_failover done oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1996 1285692 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1820 1285868 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1812 3605208 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1796 3605224 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3608 7210432 1% /mnt/lustre total: 2 open/close in 0.04 seconds: 49.38 ops/second total: 2 open/close in 0.02 seconds: 117.03 ops/second Failing mds1 on oleg354-server Stopping /mnt/lustre-mds1 (opts:) on oleg354-server 20:36:23 (1712277383) shut down Failover mds1 to oleg354-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg354-server: oleg354-server.virtnet: executing set_default_debug -1 all pdsh@oleg354-client: oleg354-server: ssh exited with exit code 1 Started lustre-MDT0000 20:36:37 (1712277397) targets are mounted 20:36:37 (1712277397) facet_failover done oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 22b (63s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-dual test 22c: c1 lfs mkdir -i 1 d1, M1 drop update & fail M1, c2 mkdir d1/dir ========================================================== 20:36:48 (1712277408) fail_loc=0x1701 fail_loc=0 Failing mds1 on oleg354-server Stopping /mnt/lustre-mds1 (opts:) on oleg354-server 20:36:51 (1712277411) shut down Failover mds1 to oleg354-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg354-server: oleg354-server.virtnet: executing set_default_debug -1 all pdsh@oleg354-client: oleg354-server: ssh exited with exit code 1 Started lustre-MDT0000 20:37:06 (1712277426) targets are mounted 20:37:06 (1712277426) facet_failover done oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1996 1285692 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1820 1285868 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1812 3605208 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1796 3605224 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3608 7210432 1% /mnt/lustre total: 2 open/close in 0.04 seconds: 52.15 ops/second total: 2 open/close in 0.02 seconds: 121.43 ops/second Failing mds1 on oleg354-server Stopping /mnt/lustre-mds1 (opts:) on oleg354-server 20:37:16 (1712277436) shut down Failover mds1 to oleg354-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg354-server: oleg354-server.virtnet: executing set_default_debug -1 all pdsh@oleg354-client: oleg354-server: ssh exited with exit code 1 Started lustre-MDT0000 20:37:31 (1712277451) targets are mounted 20:37:31 (1712277451) facet_failover done oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 22c (51s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-dual test 22d: c1 lfs mkdir -i 1 d1, M1 drop update & fail M0/M1,c2 mkdir d1/dir ========================================================== 20:37:41 (1712277461) fail_loc=0x1701 fail_loc=0 Failing mds1 on oleg354-server Stopping /mnt/lustre-mds1 (opts:) on oleg354-server Failing mds2 on oleg354-server Stopping /mnt/lustre-mds2 (opts:) on oleg354-server 20:37:54 (1712277474) shut down Failover mds1 to oleg354-server mount facets: mds1 Failover mds2 to oleg354-server mount facets: mds2 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg354-server: oleg354-server.virtnet: executing set_default_debug -1 all oleg354-server: oleg354-server.virtnet: executing set_default_debug -1 all pdsh@oleg354-client: oleg354-server: ssh exited with exit code 1 pdsh@oleg354-client: oleg354-server: ssh exited with exit code 1 Started lustre-MDT0001 Started lustre-MDT0000 20:38:14 (1712277494) targets are mounted 20:38:14 (1712277494) facet_failover done oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1960 1285728 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1784 1285904 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1812 3605208 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1796 3605224 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3608 7210432 1% /mnt/lustre total: 2 open/close in 0.04 seconds: 47.23 ops/second total: 2 open/close in 0.02 seconds: 113.43 ops/second Failing mds1 on oleg354-server Stopping /mnt/lustre-mds1 (opts:) on oleg354-server 20:38:26 (1712277506) shut down Failover mds1 to oleg354-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg354-server: oleg354-server.virtnet: executing set_default_debug -1 all pdsh@oleg354-client: oleg354-server: ssh exited with exit code 1 Started lustre-MDT0000 20:38:41 (1712277521) targets are mounted 20:38:41 (1712277521) facet_failover done oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 22d (68s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-dual test 23a: c1 rmdir d1, M1 drop reply and fail, client2 mkdir d1 ========================================================== 20:38:51 (1712277531) fail_loc=0x119 fail_loc=0 Failing mds2 on oleg354-server Stopping /mnt/lustre-mds2 (opts:) on oleg354-server 20:38:54 (1712277534) shut down Failover mds2 to oleg354-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg354-server: oleg354-server.virtnet: executing set_default_debug -1 all pdsh@oleg354-client: oleg354-server: ssh exited with exit code 1 Started lustre-MDT0001 20:39:08 (1712277548) targets are mounted 20:39:08 (1712277548) facet_failover done oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1992 1285696 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1816 1285872 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1812 3605208 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1796 3605224 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3608 7210432 1% /mnt/lustre total: 2 open/close in 0.03 seconds: 71.97 ops/second Failing mds1 on oleg354-server Stopping /mnt/lustre-mds1 (opts:) on oleg354-server 20:39:19 (1712277559) shut down Failover mds1 to oleg354-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg354-server: oleg354-server.virtnet: executing set_default_debug -1 all pdsh@oleg354-client: oleg354-server: ssh exited with exit code 1 Started lustre-MDT0000 20:39:34 (1712277574) targets are mounted 20:39:34 (1712277574) facet_failover done oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 23a (50s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-dual test 23b: c1 rmdir d1, M1 drop reply and fail M0/M1, c2 mkdir d1 ========================================================== 20:39:43 (1712277583) fail_loc=0x119 fail_loc=0 Failing mds1 on oleg354-server Stopping /mnt/lustre-mds1 (opts:) on oleg354-server Failing mds2 on oleg354-server Stopping /mnt/lustre-mds2 (opts:) on oleg354-server 20:39:52 (1712277592) shut down Failover mds1 to oleg354-server mount facets: mds1 Failover mds2 to oleg354-server mount facets: mds2 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg354-server: oleg354-server.virtnet: executing set_default_debug -1 all oleg354-server: oleg354-server.virtnet: executing set_default_debug -1 all pdsh@oleg354-client: oleg354-server: ssh exited with exit code 1 pdsh@oleg354-client: oleg354-server: ssh exited with exit code 1 Started lustre-MDT0001 Started lustre-MDT0000 20:40:13 (1712277613) targets are mounted 20:40:13 (1712277613) facet_failover done oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1992 1285696 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1816 1285872 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1812 3605208 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1796 3605224 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3608 7210432 1% /mnt/lustre total: 2 open/close in 0.03 seconds: 71.42 ops/second Failing mds1 on oleg354-server Stopping /mnt/lustre-mds1 (opts:) on oleg354-server 20:40:24 (1712277624) shut down Failover mds1 to oleg354-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg354-server: oleg354-server.virtnet: executing set_default_debug -1 all pdsh@oleg354-client: oleg354-server: ssh exited with exit code 1 Started lustre-MDT0000 20:40:40 (1712277640) targets are mounted 20:40:40 (1712277640) facet_failover done oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 23b (64s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-dual test 23c: c1 rmdir d1, M0 drop update reply and fail M0, c2 mkdir d1 ========================================================== 20:40:49 (1712277649) fail_loc=0x1701 fail_loc=0 Failing mds1 on oleg354-server Stopping /mnt/lustre-mds1 (opts:) on oleg354-server 20:40:53 (1712277653) shut down Failover mds1 to oleg354-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg354-server: oleg354-server.virtnet: executing set_default_debug -1 all pdsh@oleg354-client: oleg354-server: ssh exited with exit code 1 Started lustre-MDT0000 20:41:07 (1712277667) targets are mounted 20:41:07 (1712277667) facet_failover done oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1992 1285696 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1816 1285872 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1812 3605208 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1796 3605224 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3608 7210432 1% /mnt/lustre total: 2 open/close in 0.03 seconds: 77.13 ops/second Failing mds1 on oleg354-server Stopping /mnt/lustre-mds1 (opts:) on oleg354-server 20:41:17 (1712277677) shut down Failover mds1 to oleg354-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg354-server: oleg354-server.virtnet: executing set_default_debug -1 all pdsh@oleg354-client: oleg354-server: ssh exited with exit code 1 Started lustre-MDT0000 20:41:33 (1712277693) targets are mounted 20:41:33 (1712277693) facet_failover done oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 23c (51s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-dual test 23d: c1 rmdir d1, M0 drop update reply and fail M0/M1, c2 mkdir d1 ========================================================== 20:41:43 (1712277703) fail_loc=0x1701 fail_loc=0 Failing mds1 on oleg354-server Stopping /mnt/lustre-mds1 (opts:) on oleg354-server Failing mds2 on oleg354-server Stopping /mnt/lustre-mds2 (opts:) on oleg354-server 20:41:55 (1712277715) shut down Failover mds1 to oleg354-server mount facets: mds1 Failover mds2 to oleg354-server mount facets: mds2 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg354-server: oleg354-server.virtnet: executing set_default_debug -1 all oleg354-server: oleg354-server.virtnet: executing set_default_debug -1 all pdsh@oleg354-client: oleg354-server: ssh exited with exit code 1 pdsh@oleg354-client: oleg354-server: ssh exited with exit code 1 Started lustre-MDT0000 Started lustre-MDT0001 20:42:17 (1712277737) targets are mounted 20:42:17 (1712277737) facet_failover done oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1956 1285732 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1780 1285908 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1812 3605208 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1796 3605224 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3608 7210432 1% /mnt/lustre total: 2 open/close in 0.03 seconds: 78.43 ops/second Failing mds1 on oleg354-server Stopping /mnt/lustre-mds1 (opts:) on oleg354-server 20:42:41 (1712277761) shut down Failover mds1 to oleg354-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg354-server: oleg354-server.virtnet: executing set_default_debug -1 all pdsh@oleg354-client: oleg354-server: ssh exited with exit code 1 Started lustre-MDT0000 20:42:56 (1712277776) targets are mounted 20:42:56 (1712277776) facet_failover done oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 23d (81s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-dual test 24: reconstruct on non-existing object ========================================================== 20:43:06 (1712277786) fail_loc=0x119 fail_loc=0 truncate: cannot truncate '/mnt/lustre/f24.replay-dual' to length 100: No such file or directory PASS 24 (87s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-dual test 25: replay|resend ==================== 20:44:35 (1712277875) 1+0 records in 1+0 records out 512 bytes (512 B) copied, 0.00509902 s, 100 kB/s fail_loc=0x304 fail_loc=0x80000325 Failing ost1 on oleg354-server Stopping /mnt/lustre-ost1 (opts:) on oleg354-server 20:44:38 (1712277878) shut down Failover ost1 to oleg354-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg354-server: oleg354-server.virtnet: executing set_default_debug -1 all pdsh@oleg354-client: oleg354-server: ssh exited with exit code 1 Started lustre-OST0000 20:44:53 (1712277893) targets are mounted 20:44:53 (1712277893) facet_failover done oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 6479: 4819 Terminated LUSTRE="/home/green/git/lustre-release/lustre" bash -c "multiop /mnt/lustre2/f25.replay-dual Ow512" fail_loc=0 PASS 25 (23s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-dual test 26: dbench and tar with mds failover ========================================================== 20:45:00 (1712277900) Starting client oleg354-client.virtnet: -o user_xattr,flock oleg354-server@tcp:/lustre /mnt/lustre Started clients oleg354-client.virtnet: 192.168.203.154@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) Started tar loop with pid 6441 Started dbench loop with 6442 striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre2/d26.replay-dual/run_dbench striped dir -i0 -c2 -H crush /mnt/lustre/d26.replay-dual/run_tar looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Thu Apr 4 20:45:01 EDT 2024 waiting for dbench pid 6483 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs failed to create barrier semaphore 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 181 7.32 MB/sec warmup 1 sec latency 30.479 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2264 1285424 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2036 1285652 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 26392 3555712 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1796 3605224 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 28188 7160936 1% /mnt/lustre 1 341 6.13 MB/sec warmup 2 sec latency 32.363 ms 1 522 6.02 MB/sec warmup 3 sec latency 26.150 ms 1 779 5.34 MB/sec warmup 4 sec latency 47.170 ms 1 959 4.36 MB/sec warmup 5 sec latency 23.437 ms test_26 fail mds1 1 times 1 1179 3.84 MB/sec warmup 6 sec latency 24.249 ms Failing mds1 on oleg354-server Stopping /mnt/lustre-mds1 (opts:) on oleg354-server 1 1405 3.72 MB/sec warmup 7 sec latency 263.463 ms 20:45:09 (1712277909) shut down 1 1405 3.25 MB/sec warmup 8 sec latency 1263.667 ms 1 1405 2.89 MB/sec warmup 9 sec latency 2263.943 ms 1 1405 2.60 MB/sec warmup 10 sec latency 3264.150 ms 1 1405 2.36 MB/sec warmup 11 sec latency 4264.386 ms 1 1405 2.17 MB/sec warmup 12 sec latency 5264.548 ms 1 1405 2.00 MB/sec warmup 13 sec latency 6264.840 ms 1 1405 1.86 MB/sec warmup 14 sec latency 7265.050 ms 1 1405 1.73 MB/sec warmup 15 sec latency 8265.224 ms 1 1405 1.63 MB/sec warmup 16 sec latency 9265.357 ms 1 1405 1.53 MB/sec warmup 17 sec latency 10265.527 ms Failover mds1 to oleg354-server mount facets: mds1 1 1405 1.45 MB/sec warmup 18 sec latency 11265.750 ms 1 1405 1.37 MB/sec warmup 19 sec latency 12265.981 ms Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 1405 0.00 MB/sec execute 1 sec latency 14266.313 ms oleg354-server: oleg354-server.virtnet: executing set_default_debug -1 all 1 1405 0.00 MB/sec execute 2 sec latency 15266.548 ms pdsh@oleg354-client: oleg354-server: ssh exited with exit code 1 Started lustre-MDT0000 20:45:24 (1712277924) targets are mounted 20:45:24 (1712277924) facet_failover done 1 1405 0.00 MB/sec execute 3 sec latency 16266.775 ms 1 1405 0.00 MB/sec execute 4 sec latency 17267.106 ms 1 1405 0.00 MB/sec execute 5 sec latency 18267.459 ms 1 1405 0.00 MB/sec execute 6 sec latency 19267.705 ms oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 1481 0.00 MB/sec execute 7 sec latency 19932.083 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 1762 0.03 MB/sec execute 8 sec latency 24.525 ms 1 2014 0.06 MB/sec execute 9 sec latency 16.658 ms 1 2337 0.18 MB/sec execute 10 sec latency 16.837 ms 1 2608 0.27 MB/sec execute 11 sec latency 17.278 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2916 1284772 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2600 1285088 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 29804 3567152 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 7120 3587072 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 36924 7154224 1% /mnt/lustre 1 3207 0.69 MB/sec execute 12 sec latency 17.552 ms 1 3673 0.98 MB/sec execute 13 sec latency 15.476 ms 1 3884 0.98 MB/sec execute 14 sec latency 23.364 ms 1 4062 0.94 MB/sec execute 15 sec latency 19.532 ms test_26 fail mds2 2 times Failing mds2 on oleg354-server Stopping /mnt/lustre-mds2 (opts:) on oleg354-server 1 4330 0.90 MB/sec execute 16 sec latency 48.554 ms 20:45:38 (1712277938) shut down 1 4557 0.88 MB/sec execute 17 sec latency 21.872 ms 1 4564 0.83 MB/sec execute 18 sec latency 964.614 ms 1 4564 0.79 MB/sec execute 19 sec latency 1964.715 ms 1 4564 0.75 MB/sec execute 20 sec latency 2964.801 ms 1 4564 0.71 MB/sec execute 21 sec latency 3964.885 ms 1 4564 0.68 MB/sec execute 22 sec latency 4964.971 ms 1 4564 0.65 MB/sec execute 23 sec latency 5965.127 ms 1 4564 0.63 MB/sec execute 24 sec latency 6965.236 ms 1 4564 0.60 MB/sec execute 25 sec latency 7965.318 ms 1 4564 0.58 MB/sec execute 26 sec latency 8965.407 ms Failover mds2 to oleg354-server mount facets: mds2 1 4564 0.56 MB/sec execute 27 sec latency 9965.509 ms 1 4564 0.54 MB/sec execute 28 sec latency 10965.608 ms 1 4564 0.52 MB/sec execute 29 sec latency 11965.703 ms Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 1 4564 0.50 MB/sec execute 30 sec latency 12965.800 ms oleg354-server: oleg354-server.virtnet: executing set_default_debug -1 all 1 4564 0.48 MB/sec execute 31 sec latency 13965.994 ms pdsh@oleg354-client: oleg354-server: ssh exited with exit code 1 Started lustre-MDT0001 20:45:53 (1712277953) targets are mounted 20:45:53 (1712277953) facet_failover done 1 4564 0.47 MB/sec execute 32 sec latency 14966.273 ms 1 4564 0.45 MB/sec execute 33 sec latency 15966.511 ms 1 4564 0.44 MB/sec execute 34 sec latency 16966.750 ms 1 4564 0.43 MB/sec execute 35 sec latency 17966.969 ms oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1 4722 0.45 MB/sec execute 36 sec latency 18463.656 ms mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec 1 4969 0.52 MB/sec execute 37 sec latency 22.087 ms 1 5137 0.51 MB/sec execute 38 sec latency 31.597 ms 1 5294 0.50 MB/sec execute 39 sec latency 28.085 ms 1 5475 0.49 MB/sec execute 40 sec latency 27.035 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 4132 1283556 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2436 1285252 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 34768 3562372 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 15980 3582004 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 50748 7144376 1% /mnt/lustre 1 5716 0.48 MB/sec execute 41 sec latency 18.680 ms 1 5937 0.50 MB/sec execute 42 sec latency 26.214 ms 1 6124 0.51 MB/sec execute 43 sec latency 25.691 ms 1 6486 0.60 MB/sec execute 44 sec latency 23.307 ms test_26 fail mds1 3 times 1 6847 0.63 MB/sec execute 45 sec latency 23.773 ms Failing mds1 on oleg354-server Stopping /mnt/lustre-mds1 (opts:) on oleg354-server 1 7169 0.69 MB/sec execute 46 sec latency 30.673 ms 20:46:08 (1712277968) shut down 1 7360 0.70 MB/sec execute 47 sec latency 25.772 ms 1 7507 0.69 MB/sec execute 48 sec latency 23.521 ms 1 7627 0.68 MB/sec execute 49 sec latency 92.813 ms 1 7627 0.66 MB/sec execute 50 sec latency 1093.053 ms 1 7627 0.65 MB/sec execute 51 sec latency 2093.315 ms 1 7627 0.64 MB/sec execute 52 sec latency 3093.550 ms 1 7627 0.63 MB/sec execute 53 sec latency 4093.808 ms 1 7627 0.61 MB/sec execute 54 sec latency 5094.047 ms 1 7627 0.60 MB/sec execute 55 sec latency 6094.288 ms 1 7627 0.59 MB/sec execute 56 sec latency 7094.543 ms 1 7627 0.58 MB/sec execute 57 sec latency 8094.722 ms Failover mds1 to oleg354-server mount facets: mds1 1 7627 0.57 MB/sec execute 58 sec latency 9094.920 ms 1 7627 0.56 MB/sec execute 59 sec latency 10095.149 ms 1 7627 0.55 MB/sec execute 60 sec latency 11095.361 ms Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 7627 0.54 MB/sec execute 61 sec latency 12095.530 ms oleg354-server: oleg354-server.virtnet: executing set_default_debug -1 all 1 7627 0.54 MB/sec execute 62 sec latency 13095.730 ms pdsh@oleg354-client: oleg354-server: ssh exited with exit code 1 Started lustre-MDT0000 20:46:24 (1712277984) targets are mounted 20:46:24 (1712277984) facet_failover done 1 7627 0.53 MB/sec execute 63 sec latency 14096.030 ms 1 7627 0.52 MB/sec execute 64 sec latency 15096.389 ms 1 7627 0.51 MB/sec execute 65 sec latency 16096.739 ms 1 7627 0.50 MB/sec execute 66 sec latency 17096.971 ms 1 7627 0.50 MB/sec execute 67 sec latency 18097.214 ms 1 7627 0.49 MB/sec execute 68 sec latency 19097.456 ms oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 7762 0.48 MB/sec execute 69 sec latency 19568.427 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 7918 0.48 MB/sec execute 70 sec latency 28.944 ms 1 8120 0.49 MB/sec execute 71 sec latency 23.268 ms 1 8334 0.49 MB/sec execute 72 sec latency 24.302 ms 1 8608 0.52 MB/sec execute 73 sec latency 43.261 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 4428 1283260 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2488 1285200 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 34928 3552260 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 15956 3573592 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 50884 7125852 1% /mnt/lustre 1 8768 0.52 MB/sec execute 74 sec latency 26.399 ms 1 8936 0.51 MB/sec execute 75 sec latency 32.243 ms 1 9140 0.51 MB/sec execute 76 sec latency 17.196 ms 1 9376 0.51 MB/sec execute 77 sec latency 19.171 ms 1 9594 0.52 MB/sec execute 78 sec latency 32.793 ms test_26 fail mds2 4 times Failing mds2 on oleg354-server 1 10018 0.58 MB/sec execute 79 sec latency 19.952 ms Stopping /mnt/lustre-mds2 (opts:) on oleg354-server 1 10111 0.57 MB/sec execute 80 sec latency 557.404 ms 20:46:42 (1712278002) shut down 1 10111 0.56 MB/sec execute 81 sec latency 1557.544 ms 1 10111 0.56 MB/sec execute 82 sec latency 2557.762 ms 1 10111 0.55 MB/sec execute 83 sec latency 3557.979 ms 1 10111 0.54 MB/sec execute 84 sec latency 4558.181 ms 1 10111 0.54 MB/sec execute 85 sec latency 5558.415 ms 1 10111 0.53 MB/sec execute 86 sec latency 6558.631 ms 1 10111 0.53 MB/sec execute 87 sec latency 7558.862 ms 1 10111 0.52 MB/sec execute 88 sec latency 8559.073 ms 1 10111 0.51 MB/sec execute 89 sec latency 9559.281 ms 1 10111 0.51 MB/sec execute 90 sec latency 10559.502 ms Failover mds2 to oleg354-server mount facets: mds2 1 10111 0.50 MB/sec execute 91 sec latency 11559.720 ms 1 10111 0.50 MB/sec execute 92 sec latency 12559.896 ms 1 10111 0.49 MB/sec execute 93 sec latency 13560.064 ms Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 1 10111 0.49 MB/sec execute 94 sec latency 14560.242 ms oleg354-server: oleg354-server.virtnet: executing set_default_debug -1 all 1 10111 0.48 MB/sec execute 95 sec latency 15560.456 ms pdsh@oleg354-client: oleg354-server: ssh exited with exit code 1 Started lustre-MDT0001 20:46:57 (1712278017) targets are mounted 20:46:57 (1712278017) facet_failover done 1 10111 0.48 MB/sec execute 96 sec latency 16560.635 ms 1 10111 0.47 MB/sec execute 97 sec latency 17560.830 ms 1 10111 0.47 MB/sec execute 98 sec latency 18561.053 ms 1 10111 0.46 MB/sec execute 99 sec latency 19561.275 ms 1 cleanup 100 sec oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec 0 cleanup 101 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 1507 37.506 19932.061 Close 1127 2.498 17.426 Rename 63 340.777 20446.172 Unlink 278 5.771 15.596 Qpathinfo 1422 3.230 30.659 Qfileinfo 259 0.563 3.514 Qfsinfo 227 1.108 11.760 Sfileinfo 116 9.136 20.895 Find 515 41.035 19568.298 WriteX 759 2.758 14.226 ReadX 2336 0.108 8.823 LockX 6 2.327 3.115 UnlockX 6 2.603 3.814 Flush 100 12.262 48.548 Throughput 0.462149 MB/sec 1 clients 1 procs max_latency=19932.083 ms stopping dbench on /mnt/lustre at Thu Apr 4 20:47:02 EDT 2024 with return code 0 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished striped dir -i0 -c2 -H crush2 /mnt/lustre2/d26.replay-dual/run_dbench looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Thu Apr 4 20:47:04 EDT 2024 waiting for dbench pid 10910 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 157 6.54 MB/sec warmup 1 sec latency 37.726 ms 1 306 5.61 MB/sec warmup 2 sec latency 33.824 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 5008 1282680 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3056 1284632 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 35108 3521472 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 2360 3580332 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 37468 7101804 1% /mnt/lustre 1 456 5.30 MB/sec warmup 3 sec latency 36.466 ms 1 664 5.29 MB/sec warmup 4 sec latency 101.422 ms 1 786 4.29 MB/sec warmup 5 sec latency 371.864 ms 1 977 3.65 MB/sec warmup 6 sec latency 23.996 ms test_26 fail mds1 5 times 1 1176 3.29 MB/sec warmup 7 sec latency 24.393 ms Failing mds1 on oleg354-server Stopping /mnt/lustre-mds1 (opts:) on oleg354-server 1 1428 3.25 MB/sec warmup 8 sec latency 23.878 ms 1 1428 2.89 MB/sec warmup 9 sec latency 1016.147 ms 1 1428 2.60 MB/sec warmup 10 sec latency 2016.402 ms 1 1428 2.36 MB/sec warmup 11 sec latency 3016.694 ms 1 1428 2.17 MB/sec warmup 12 sec latency 4016.941 ms 1 1428 2.00 MB/sec warmup 13 sec latency 5017.187 ms 1 1428 1.86 MB/sec warmup 14 sec latency 6017.419 ms 20:47:18 (1712278038) shut down 1 1428 1.73 MB/sec warmup 15 sec latency 7017.706 ms 1 1428 1.63 MB/sec warmup 16 sec latency 8017.937 ms 1 1428 1.53 MB/sec warmup 17 sec latency 9018.187 ms 1 1428 1.45 MB/sec warmup 18 sec latency 10018.479 ms 1 1428 1.37 MB/sec warmup 19 sec latency 11018.738 ms 1 1428 0.00 MB/sec execute 1 sec latency 13019.153 ms 1 1428 0.00 MB/sec execute 2 sec latency 14019.409 ms 1 1428 0.00 MB/sec execute 3 sec latency 15019.681 ms 1 1428 0.00 MB/sec execute 4 sec latency 16019.901 ms Failover mds1 to oleg354-server mount facets: mds1 1 1428 0.00 MB/sec execute 5 sec latency 17020.122 ms 1 1428 0.00 MB/sec execute 6 sec latency 18020.313 ms 1 1428 0.00 MB/sec execute 7 sec latency 19020.547 ms Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 1428 0.00 MB/sec execute 8 sec latency 20020.704 ms oleg354-server: oleg354-server.virtnet: executing set_default_debug -1 all 1 1428 0.00 MB/sec execute 9 sec latency 21020.886 ms pdsh@oleg354-client: oleg354-server: ssh exited with exit code 1 Started lustre-MDT0000 20:47:34 (1712278054) targets are mounted 20:47:34 (1712278054) facet_failover done 1 1428 0.00 MB/sec execute 10 sec latency 22021.113 ms 1 1428 0.00 MB/sec execute 11 sec latency 23021.383 ms 1 1428 0.00 MB/sec execute 12 sec latency 24021.634 ms 1 1428 0.00 MB/sec execute 13 sec latency 25021.866 ms 1 1428 0.00 MB/sec execute 14 sec latency 26022.023 ms oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 1684 0.01 MB/sec execute 15 sec latency 26142.855 ms 1 1903 0.03 MB/sec execute 16 sec latency 24.355 ms 1 2095 0.04 MB/sec execute 17 sec latency 20.903 ms 1 2345 0.10 MB/sec execute 18 sec latency 17.680 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 5432 1282256 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3464 1284224 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 58328 3547396 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 20412 3582792 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 78740 7130188 2% /mnt/lustre 1 2492 0.14 MB/sec execute 19 sec latency 24.967 ms 1 2929 0.37 MB/sec execute 20 sec latency 20.699 ms 1 3012 0.36 MB/sec execute 21 sec latency 637.445 ms 1 3374 0.43 MB/sec execute 22 sec latency 29.195 ms 1 3677 0.55 MB/sec execute 23 sec latency 19.802 ms test_26 fail mds2 6 times Failing mds2 on oleg354-server 1 3845 0.57 MB/sec execute 24 sec latency 24.651 ms Stopping /mnt/lustre-mds2 (opts:) on oleg354-server 1 3932 0.56 MB/sec execute 25 sec latency 545.784 ms 1 3932 0.54 MB/sec execute 26 sec latency 1545.971 ms 1 3932 0.52 MB/sec execute 27 sec latency 2546.262 ms 1 3932 0.50 MB/sec execute 28 sec latency 3546.514 ms 1 3932 0.48 MB/sec execute 29 sec latency 4546.742 ms 1 3932 0.47 MB/sec execute 30 sec latency 5546.965 ms 1 3932 0.45 MB/sec execute 31 sec latency 6547.217 ms 20:47:55 (1712278075) shut down 1 3932 0.44 MB/sec execute 32 sec latency 7547.444 ms 1 3932 0.43 MB/sec execute 33 sec latency 8547.681 ms 1 3932 0.41 MB/sec execute 34 sec latency 9547.942 ms 1 3932 0.40 MB/sec execute 35 sec latency 10548.257 ms 1 3932 0.39 MB/sec execute 36 sec latency 11548.537 ms 1 3932 0.38 MB/sec execute 37 sec latency 12548.770 ms 1 3932 0.37 MB/sec execute 38 sec latency 13549.102 ms 1 3932 0.36 MB/sec execute 39 sec latency 14549.373 ms 1 3932 0.35 MB/sec execute 40 sec latency 15549.632 ms 1 3932 0.34 MB/sec execute 41 sec latency 16549.879 ms Failover mds2 to oleg354-server mount facets: mds2 1 3932 0.33 MB/sec execute 42 sec latency 17550.103 ms 1 3932 0.33 MB/sec execute 43 sec latency 18550.338 ms 1 3932 0.32 MB/sec execute 44 sec latency 19550.520 ms Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 1 3932 0.31 MB/sec execute 45 sec latency 20550.716 ms oleg354-server: oleg354-server.virtnet: executing set_default_debug -1 all 1 3932 0.31 MB/sec execute 46 sec latency 21550.900 ms pdsh@oleg354-client: oleg354-server: ssh exited with exit code 1 Started lustre-MDT0001 20:48:11 (1712278091) targets are mounted 20:48:11 (1712278091) facet_failover done 1 3932 0.30 MB/sec execute 47 sec latency 22551.192 ms 1 3932 0.29 MB/sec execute 48 sec latency 23551.440 ms 1 3932 0.29 MB/sec execute 49 sec latency 24551.687 ms 1 3932 0.28 MB/sec execute 50 sec latency 25551.887 ms oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1 4049 0.28 MB/sec execute 51 sec latency 25903.252 ms mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec 1 4292 0.28 MB/sec execute 52 sec latency 26.954 ms 1 4534 0.28 MB/sec execute 53 sec latency 23.815 ms 1 4838 0.30 MB/sec execute 54 sec latency 17.964 ms 1 5190 0.35 MB/sec execute 55 sec latency 24.614 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 5008 1282680 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3032 1284656 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 58276 3547576 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 17408 3589192 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 75684 7136768 2% /mnt/lustre 1 5449 0.35 MB/sec execute 56 sec latency 29.520 ms 1 5700 0.35 MB/sec execute 57 sec latency 18.361 ms 1 5914 0.36 MB/sec execute 58 sec latency 22.567 ms 1 6052 0.37 MB/sec execute 59 sec latency 26.324 ms test_26 fail mds1 7 times Failing mds1 on oleg354-server 1 6437 0.42 MB/sec execute 60 sec latency 24.038 ms Stopping /mnt/lustre-mds1 (opts:) on oleg354-server 1 6800 0.46 MB/sec execute 61 sec latency 51.309 ms 20:48:26 (1712278106) shut down 1 7102 0.51 MB/sec execute 62 sec latency 21.798 ms 1 7297 0.52 MB/sec execute 63 sec latency 22.006 ms 1 7435 0.52 MB/sec execute 64 sec latency 25.503 ms 1 7570 0.51 MB/sec execute 65 sec latency 23.548 ms 1 7627 0.50 MB/sec execute 66 sec latency 684.124 ms 1 7627 0.50 MB/sec execute 67 sec latency 1684.316 ms 1 7627 0.49 MB/sec execute 68 sec latency 2684.495 ms 1 7627 0.48 MB/sec execute 69 sec latency 3684.829 ms 1 7627 0.47 MB/sec execute 70 sec latency 4685.007 ms 1 7627 0.47 MB/sec execute 71 sec latency 5685.212 ms Failover mds1 to oleg354-server mount facets: mds1 1 7627 0.46 MB/sec execute 72 sec latency 6685.382 ms 1 7627 0.45 MB/sec execute 73 sec latency 7685.499 ms Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 7627 0.45 MB/sec execute 74 sec latency 8685.598 ms oleg354-server: oleg354-server.virtnet: executing set_default_debug -1 all pdsh@oleg354-client: oleg354-server: ssh exited with exit code 1 1 7627 0.44 MB/sec execute 75 sec latency 9685.717 ms Started lustre-MDT0000 20:48:39 (1712278119) targets are mounted 20:48:39 (1712278119) facet_failover done 1 7627 0.44 MB/sec execute 76 sec latency 10685.892 ms 1 7627 0.43 MB/sec execute 77 sec latency 11686.072 ms 1 7627 0.43 MB/sec execute 78 sec latency 12686.304 ms 1 7627 0.42 MB/sec execute 79 sec latency 13686.522 ms 1 7627 0.41 MB/sec execute 80 sec latency 14686.738 ms oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 7766 0.41 MB/sec execute 81 sec latency 15087.443 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 7917 0.41 MB/sec execute 82 sec latency 43.003 ms 1 8080 0.41 MB/sec execute 83 sec latency 43.242 ms 1 8285 0.42 MB/sec execute 84 sec latency 23.649 ms 1 8660 0.45 MB/sec execute 85 sec latency 28.027 ms 1 8864 0.45 MB/sec execute 86 sec latency 21.264 ms 1 9129 0.45 MB/sec execute 87 sec latency 34.283 ms 1 9505 0.46 MB/sec execute 88 sec latency 17.330 ms 1 10031 0.51 MB/sec execute 89 sec latency 13.398 ms 1 10477 0.53 MB/sec execute 90 sec latency 23.296 ms 1 10783 0.56 MB/sec execute 91 sec latency 18.566 ms 1 10928 0.56 MB/sec execute 92 sec latency 60.404 ms 1 11065 0.56 MB/sec execute 93 sec latency 24.628 ms 1 11214 0.56 MB/sec execute 94 sec latency 22.937 ms 1 11407 0.55 MB/sec execute 95 sec latency 45.012 ms 1 11625 0.55 MB/sec execute 96 sec latency 26.151 ms 1 11902 0.56 MB/sec execute 97 sec latency 23.674 ms 1 12231 0.59 MB/sec execute 98 sec latency 23.046 ms 1 12542 0.58 MB/sec execute 99 sec latency 13.704 ms 1 cleanup 100 sec 0 cleanup 100 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 2032 23.976 25903.244 Close 1515 2.356 26.078 Rename 79 16.377 51.284 Unlink 385 6.094 23.082 Qpathinfo 1883 16.928 26142.834 Qfileinfo 323 0.535 4.282 Qfsinfo 308 1.148 10.249 Sfileinfo 150 8.883 26.139 Find 690 24.706 15087.423 WriteX 986 2.589 13.624 ReadX 3024 0.109 14.382 LockX 6 1.951 2.476 UnlockX 6 1.992 2.378 Flush 135 17.135 637.426 Throughput 0.58289 MB/sec 1 clients 1 procs max_latency=26142.855 ms stopping dbench on /mnt/lustre at Thu Apr 4 20:49:05 EDT 2024 with return code 0 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished striped dir -i0 -c2 -H all_char /mnt/lustre2/d26.replay-dual/run_dbench looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Thu Apr 4 20:49:06 EDT 2024 waiting for dbench pid 14389 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients dbench killed by signal 15 stopping dbench on /mnt/lustre at Thu Apr 4 20:49:07 EDT 2024 with return code 0 14389 pts/0 S+ 0:00 dbench -c client.txt 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100 killed dbench main pid 14389 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished PASS 26 (248s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-dual test 28: lock replay should be ordered: waiting after granted ========================================================== 20:49:10 (1712278150) 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00384691 s, 1.1 MB/s fail_loc=0x80000324 fail_loc=0x32a Failing ost1 on oleg354-server Stopping /mnt/lustre-ost1 (opts:) on oleg354-server 20:49:20 (1712278160) shut down Failover ost1 to oleg354-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg354-server: oleg354-server.virtnet: executing set_default_debug -1 all pdsh@oleg354-client: oleg354-server: ssh exited with exit code 1 Started lustre-OST0000 20:49:34 (1712278174) targets are mounted 20:49:34 (1712278174) facet_failover done 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00515422 s, 795 kB/s oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec PASS 28 (31s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-dual test 29: replay vs update with the same xid ========================================================== 20:49:43 (1712278183) SKIP: replay-dual test_29 needs >= 2 clients SKIP 29 (2s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-dual test 30: layout lock replay is not blocked on IO ========================================================== 20:49:47 (1712278187) 10+0 records in 10+0 records out 40960 bytes (41 kB) copied, 0.0127819 s, 3.2 MB/s 10+0 records in 10+0 records out 40960 bytes (41 kB) copied, 0.0129085 s, 3.2 MB/s fail_loc=0x32e fail_val=4 Failing mds1 on oleg354-server Stopping /mnt/lustre-mds1 (opts:) on oleg354-server 20:49:50 (1712278190) shut down Failover mds1 to oleg354-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg354-server: oleg354-server.virtnet: executing set_default_debug -1 all pdsh@oleg354-client: oleg354-server: ssh exited with exit code 1 Started lustre-MDT0000 20:50:02 (1712278202) targets are mounted 20:50:02 (1712278202) facet_failover done 160+0 records in 160+0 records out 81920 bytes (82 kB) copied, 21.7686 s, 3.8 kB/s oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 30 (26s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-dual test 31: deadlock on file_remove_privs and occupied mod rpc slots ========================================================== 20:50:14 (1712278214) Failing ost1 on oleg354-server Stopping /mnt/lustre-ost1 (opts:) on oleg354-server 20:50:16 (1712278216) shut down Creating to objid 3073 on ost lustre-OST0000... total: 32 open/close in 0.15 seconds: 215.80 ops/second at_max=0 fail_loc=0x80001420 file /mnt/lustre2/d31.replay-dual/mdtdir/f31.replay-dual is ready Failover ost1 to oleg354-server mount facets: ost1 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg354-server: oleg354-server.virtnet: executing set_default_debug -1 all pdsh@oleg354-client: oleg354-server: ssh exited with exit code 1 Started lustre-OST0000 20:50:29 (1712278229) targets are mounted 20:50:29 (1712278229) facet_failover done oleg354-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL IDLE state after 0 sec pids: 18916 18917 18923 18924 18925 18926 18927 18928 18929 at_max=600 PASS 31 (21s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-dual test 32: gap in update llog shouldn't break recovery ========================================================== 20:50:36 (1712278236) fail_loc=0x0000131d fail_val=10 fail_loc=0x726 Stopping /mnt/lustre-mds2 (opts:) on oleg354-server Stopping /mnt/lustre-mds1 (opts:) on oleg354-server fail_loc=0 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg354-server: oleg354-server.virtnet: executing set_default_debug -1 all pdsh@oleg354-client: oleg354-server: ssh exited with exit code 1 Started lustre-MDT0000 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg354-server: oleg354-server.virtnet: executing set_default_debug -1 all pdsh@oleg354-client: oleg354-server: ssh exited with exit code 1 Started lustre-MDT0001 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2452 1285208 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2036 1285652 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1820 3605200 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1828 3604764 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3648 7209964 1% /mnt/lustre PASS 32 (16s) debug_raw_pointers=0 debug_raw_pointers=0 debug_raw_pointers=Y debug_raw_pointers=Y == replay-dual test 33: Check for OBD_INCOMPAT_MULTI_RPCS in last_rcvd after abort_recovery ========================================================== 20:50:53 (1712278253) at_min=60 Stopping /mnt/lustre-mds2 (opts:) on oleg354-server last_rcvd in /dev/mapper/mds2_flakey: incompat = 0x61c Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg354-server: oleg354-server.virtnet: executing set_default_debug -1 all pdsh@oleg354-client: oleg354-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg354-client.virtnet: executing wait_import_state_mount REPLAY_WAIT mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in REPLAY_WAIT state after 0 sec oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec affected facets: mds2 oleg354-server: oleg354-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0001.recovery_status 1475 oleg354-server: *.lustre-MDT0001.recovery_status status: COMPLETE Stopping /mnt/lustre-mds2 (opts:) on oleg354-server last_rcvd in /dev/mapper/mds2_flakey: incompat = 0x61c Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg354-server: oleg354-server.virtnet: executing set_default_debug -1 all pdsh@oleg354-client: oleg354-server: ssh exited with exit code 1 Started lustre-MDT0001 Starting client: oleg354-client.virtnet: -o user_xattr,flock oleg354-server@tcp:/lustre /mnt/lustre2 oleg354-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 4 sec affected facets: mds2 oleg354-server: oleg354-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0001.recovery_status 1475 oleg354-server: *.lustre-MDT0001.recovery_status status: COMPLETE at_min=5 PASS 33 (34s) debug_raw_pointers=0 debug_raw_pointers=0 == replay-dual test complete, duration 2742 sec ========== 20:51:28 (1712278288) === replay-dual: start cleanup 20:51:28 (1712278288) === Stopping clients: oleg354-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg354-client.virtnet /mnt/lustre2 opts: === replay-dual: finish cleanup 20:51:30 (1712278290) ===