-----============= acceptance-small: replay-dual ============----- Wed Jul 24 23:55:37 EDT 2024 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 excepting tests: 14b 21b 21b skipping tests SLOW=no: 21b === replay-dual: start setup 23:55:42 (1721879742) === Starting client oleg435-client.virtnet: -o user_xattr,flock oleg435-server@tcp:/lustre /mnt/lustre2 Started clients oleg435-client.virtnet: 192.168.204.135@tcp:/lustre on /mnt/lustre2 type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) oleg435-client.virtnet: executing check_config_client /mnt/lustre oleg435-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg435-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800ab131000.idle_timeout=debug osc.lustre-OST0000-osc-ffff88012a531000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800ab131000.idle_timeout=debug osc.lustre-OST0001-osc-ffff88012a531000.idle_timeout=debug disable quota as required oleg435-server: oleg435-server.virtnet: executing set_default_debug -1 all === replay-dual: finish setup 23:55:48 (1721879748) === == replay-dual test 0a: expired recovery with lost client ========================================================== 23:55:48 (1721879748) Check file is LU482_FAILED=/tmp/replay-dual.lu482.Tc4ivv UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre total: 50 open/close in 0.80 seconds: 62.63 ops/second fail_loc=0x80000514 Failing mds1 on oleg435-server Stopping /mnt/lustre-mds1 (opts:) on oleg435-server 23:55:52 (1721879752) shut down Failover mds1 to oleg435-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg435-server: oleg435-server.virtnet: executing set_default_debug -1 all pdsh@oleg435-client: oleg435-server: ssh exited with exit code 1 Started lustre-MDT0000 23:56:05 (1721879765) targets are mounted 23:56:05 (1721879765) facet_failover done Starting client: oleg435-client.virtnet: -o user_xattr,flock oleg435-server@tcp:/lustre /mnt/lustre2 - unlinked 0 (time 1721879847 ; total 0 ; last 0) total: 50 unlinks in 1 seconds: 50.000000 unlinks/second PASS 0a (101s) == replay-dual test 0b: lost client during waiting for next transno ========================================================== 23:57:29 (1721879849) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg435-server Stopping /mnt/lustre-mds1 (opts:) on oleg435-server 23:57:32 (1721879852) shut down Failover mds1 to oleg435-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg435-server: oleg435-server.virtnet: executing set_default_debug -1 all pdsh@oleg435-client: oleg435-server: ssh exited with exit code 1 Started lustre-MDT0000 23:57:45 (1721879865) targets are mounted 23:57:45 (1721879865) facet_failover done Starting client: oleg435-client.virtnet: -o user_xattr,flock oleg435-server@tcp:/lustre /mnt/lustre Starting client: oleg435-client.virtnet: -o user_xattr,flock oleg435-server@tcp:/lustre /mnt/lustre2 PASS 0b (88s) == replay-dual test 1: |X| simple create ================= 23:58:57 (1721879937) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg435-server Stopping /mnt/lustre-mds1 (opts:) on oleg435-server 23:59:00 (1721879940) shut down Failover mds1 to oleg435-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg435-server: oleg435-server.virtnet: executing set_default_debug -1 all pdsh@oleg435-client: oleg435-server: ssh exited with exit code 1 Started lustre-MDT0000 23:59:14 (1721879954) targets are mounted 23:59:14 (1721879954) facet_failover done oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 1 (23s) == replay-dual test 2: |X| mkdir adir ==================== 23:59:20 (1721879960) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg435-server Stopping /mnt/lustre-mds1 (opts:) on oleg435-server 23:59:23 (1721879963) shut down Failover mds1 to oleg435-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg435-server: oleg435-server.virtnet: executing set_default_debug -1 all pdsh@oleg435-client: oleg435-server: ssh exited with exit code 1 Started lustre-MDT0000 23:59:36 (1721879976) targets are mounted 23:59:36 (1721879976) facet_failover done oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 2 (21s) == replay-dual test 3: |X| mkdir adir, mkdir adir/bdir === 23:59:41 (1721879981) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg435-server Stopping /mnt/lustre-mds1 (opts:) on oleg435-server 23:59:44 (1721879984) shut down Failover mds1 to oleg435-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg435-server: oleg435-server.virtnet: executing set_default_debug -1 all pdsh@oleg435-client: oleg435-server: ssh exited with exit code 1 Started lustre-MDT0000 23:59:57 (1721879997) targets are mounted 23:59:57 (1721879997) facet_failover done oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 3 (22s) == replay-dual test 4: |X| mkdir adir (-EEXIST), mkdir adir/bdir ========================================================== 00:00:03 (1721880003) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre mkdir: cannot create directory '/mnt/lustre/adir': File exists Failing mds1 on oleg435-server Stopping /mnt/lustre-mds1 (opts:) on oleg435-server 00:00:06 (1721880006) shut down Failover mds1 to oleg435-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg435-server: oleg435-server.virtnet: executing set_default_debug -1 all pdsh@oleg435-client: oleg435-server: ssh exited with exit code 1 Started lustre-MDT0000 00:00:19 (1721880019) targets are mounted 00:00:19 (1721880019) facet_failover done oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 4 (20s) == replay-dual test 5: open, unlink |X| close ============ 00:00:23 (1721880023) multiop /mnt/lustre2/a vo_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.6956 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg435-server Stopping /mnt/lustre-mds1 (opts:) on oleg435-server 00:00:26 (1721880026) shut down Failover mds1 to oleg435-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg435-server: oleg435-server.virtnet: executing set_default_debug -1 all pdsh@oleg435-client: oleg435-server: ssh exited with exit code 1 Started lustre-MDT0000 00:00:39 (1721880039) targets are mounted 00:00:39 (1721880039) facet_failover done oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 5 (21s) == replay-dual test 6: open1, open2, unlink |X| close1 [fail mds1] close2 ========================================================== 00:00:44 (1721880044) multiop /mnt/lustre2/a vo_c TMPPIPE=/tmp/multiop_open_wait_pipe.6956 multiop /mnt/lustre/a vo_c TMPPIPE=/tmp/multiop_open_wait_pipe.6956 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg435-server Stopping /mnt/lustre-mds1 (opts:) on oleg435-server 00:00:46 (1721880046) shut down Failover mds1 to oleg435-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg435-server: oleg435-server.virtnet: executing set_default_debug -1 all pdsh@oleg435-client: oleg435-server: ssh exited with exit code 1 Started lustre-MDT0000 00:01:00 (1721880060) targets are mounted 00:01:00 (1721880060) facet_failover done oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 6 (22s) == replay-dual test 8: replay of resent request ========== 00:01:06 (1721880066) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre fail_loc=0x119 fail_loc=0 Failing mds1 on oleg435-server Stopping /mnt/lustre-mds1 (opts:) on oleg435-server 00:01:26 (1721880086) shut down Failover mds1 to oleg435-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg435-server: oleg435-server.virtnet: executing set_default_debug -1 all pdsh@oleg435-client: oleg435-server: ssh exited with exit code 1 Started lustre-MDT0000 00:01:39 (1721880099) targets are mounted 00:01:39 (1721880099) facet_failover done oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 8 (37s) == replay-dual test 9: resending a replayed create ======= 00:01:43 (1721880103) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre fail_loc=0x80000119 Failing mds1 on oleg435-server Stopping /mnt/lustre-mds1 (opts:) on oleg435-server 00:01:47 (1721880107) shut down Failover mds1 to oleg435-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg435-server: oleg435-server.virtnet: executing set_default_debug -1 all pdsh@oleg435-client: oleg435-server: ssh exited with exit code 1 Started lustre-MDT0000 00:01:59 (1721880119) targets are mounted 00:01:59 (1721880119) facet_failover done oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec fail_loc=0 PASS 9 (36s) == replay-dual test 10: resending a replayed unlink ====== 00:02:19 (1721880139) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3200 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre fail_loc=0x80000119 Failing mds1 on oleg435-server Stopping /mnt/lustre-mds1 (opts:) on oleg435-server 00:02:22 (1721880142) shut down Failover mds1 to oleg435-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg435-server: oleg435-server.virtnet: executing set_default_debug -1 all pdsh@oleg435-client: oleg435-server: ssh exited with exit code 1 Started lustre-MDT0000 00:02:35 (1721880155) targets are mounted 00:02:35 (1721880155) facet_failover done oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec fail_loc=0 PASS 10 (35s) == replay-dual test 11: both clients timeout during replay ========================================================== 00:02:54 (1721880174) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre fail_loc=0x0119 Failing mds1 on oleg435-server Stopping /mnt/lustre-mds1 (opts:) on oleg435-server 00:02:57 (1721880177) shut down Failover mds1 to oleg435-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg435-server: oleg435-server.virtnet: executing set_default_debug -1 all pdsh@oleg435-client: oleg435-server: ssh exited with exit code 1 Started lustre-MDT0000 00:03:10 (1721880190) targets are mounted 00:03:10 (1721880190) facet_failover done oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 15 sec fail_loc=0 PASS 11 (34s) == replay-dual test 12: open resend timeout ============== 00:03:28 (1721880208) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre multiop /mnt/lustre/f12.replay-dual vmo_c TMPPIPE=/tmp/multiop_open_wait_pipe.6956 fail_loc=0x80000302 Failing mds1 on oleg435-server Stopping /mnt/lustre-mds1 (opts:) on oleg435-server 00:03:31 (1721880211) shut down Failover mds1 to oleg435-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg435-server: oleg435-server.virtnet: executing set_default_debug -1 all pdsh@oleg435-client: oleg435-server: ssh exited with exit code 1 Started lustre-MDT0000 00:03:44 (1721880224) targets are mounted 00:03:44 (1721880224) facet_failover done fail_loc=0 /mnt/lustre/f12.replay-dual /mnt/lustre/f12.replay-dual has type file OK PASS 12 (24s) == replay-dual test 13: close resend timeout ============= 00:03:52 (1721880232) multiop /mnt/lustre/f13.replay-dual vmo_c TMPPIPE=/tmp/multiop_open_wait_pipe.6956 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre fail_loc=0x80000115 Failing mds1 on oleg435-server Stopping /mnt/lustre-mds1 (opts:) on oleg435-server 00:03:55 (1721880235) shut down Failover mds1 to oleg435-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg435-server: oleg435-server.virtnet: executing set_default_debug -1 all pdsh@oleg435-client: oleg435-server: ssh exited with exit code 1 Started lustre-MDT0000 00:04:08 (1721880248) targets are mounted 00:04:08 (1721880248) facet_failover done fail_loc=0 /mnt/lustre/f13.replay-dual /mnt/lustre/f13.replay-dual has type file OK PASS 13 (20s) SKIP: replay-dual test_14b skipping ALWAYS excluded test 14b == replay-dual test 15a: timeout waiting for lost client during replay, 1 client completes ========================================================== 00:04:12 (1721880252) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre total: 25 open/close in 0.21 seconds: 121.60 ops/second total: 1 open/close in 0.01 seconds: 107.99 ops/second Failing mds1 on oleg435-server Stopping /mnt/lustre-mds1 (opts:) on oleg435-server 00:04:15 (1721880255) shut down Failover mds1 to oleg435-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg435-server: oleg435-server.virtnet: executing set_default_debug -1 all pdsh@oleg435-client: oleg435-server: ssh exited with exit code 1 Started lustre-MDT0000 00:04:28 (1721880268) targets are mounted 00:04:28 (1721880268) facet_failover done oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec - unlinked 0 (time 1721880340 ; total 0 ; last 0) total: 25 unlinks in 0 seconds: inf unlinks/second Starting client: oleg435-client.virtnet: -o user_xattr,flock oleg435-server@tcp:/lustre /mnt/lustre2 PASS 15a (89s) == replay-dual test 15c: remove multiple OST orphans ===== 00:05:41 (1721880341) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg435-server Stopping /mnt/lustre-mds1 (opts:) on oleg435-server 00:06:14 (1721880374) shut down Failover mds1 to oleg435-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg435-server: oleg435-server.virtnet: executing set_default_debug -1 all pdsh@oleg435-client: oleg435-server: ssh exited with exit code 1 Started lustre-MDT0000 00:06:26 (1721880386) targets are mounted 00:06:26 (1721880386) facet_failover done oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Starting client: oleg435-client.virtnet: -o user_xattr,flock oleg435-server@tcp:/lustre /mnt/lustre2 PASS 15c (117s) == replay-dual test 16: fail MDS during recovery (3571) == 00:07:38 (1721880458) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre total: 25 open/close in 0.22 seconds: 114.59 ops/second total: 1 open/close in 0.01 seconds: 103.49 ops/second Failing mds1 on oleg435-server Stopping /mnt/lustre-mds1 (opts:) on oleg435-server 00:07:41 (1721880461) shut down Failover mds1 to oleg435-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg435-server: oleg435-server.virtnet: executing set_default_debug -1 all pdsh@oleg435-client: oleg435-server: ssh exited with exit code 1 Started lustre-MDT0000 00:07:53 (1721880473) targets are mounted 00:07:53 (1721880473) facet_failover done Failing mds1 on oleg435-server Stopping /mnt/lustre-mds1 (opts:) on oleg435-server 00:08:15 (1721880495) shut down Failover mds1 to oleg435-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg435-server: oleg435-server.virtnet: executing set_default_debug -1 all pdsh@oleg435-client: oleg435-server: ssh exited with exit code 1 Started lustre-MDT0000 00:08:27 (1721880507) targets are mounted 00:08:27 (1721880507) facet_failover done oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec - unlinked 0 (time 1721880586 ; total 0 ; last 0) total: 25 unlinks in 0 seconds: inf unlinks/second Starting client: oleg435-client.virtnet: -o user_xattr,flock oleg435-server@tcp:/lustre /mnt/lustre2 PASS 16 (129s) == replay-dual test 17: fail OST during recovery (3571) == 00:09:47 (1721880587) total: 25 open/close in 0.23 seconds: 108.26 ops/second total: 1 open/close in 0.01 seconds: 76.69 ops/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing ost1 on oleg435-server Stopping /mnt/lustre-ost1 (opts:) on oleg435-server 00:09:50 (1721880590) shut down Failover ost1 to oleg435-server mount facets: ost1 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg435-server: oleg435-server.virtnet: executing set_default_debug -1 all pdsh@oleg435-client: oleg435-server: ssh exited with exit code 1 Started lustre-OST0000 00:10:03 (1721880603) targets are mounted 00:10:03 (1721880603) facet_failover done Failing ost1 on oleg435-server Stopping /mnt/lustre-ost1 (opts:) on oleg435-server 00:10:25 (1721880625) shut down Failover ost1 to oleg435-server mount facets: ost1 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg435-server: oleg435-server.virtnet: executing set_default_debug -1 all pdsh@oleg435-client: oleg435-server: ssh exited with exit code 1 Started lustre-OST0000 00:10:37 (1721880637) targets are mounted 00:10:37 (1721880637) facet_failover done oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec - unlinked 0 (time 1721880715 ; total 0 ; last 0) total: 25 unlinks in 0 seconds: inf unlinks/second Starting client: oleg435-client.virtnet: -o user_xattr,flock oleg435-server@tcp:/lustre /mnt/lustre2 PASS 17 (129s) == replay-dual test 18: ldlm_handle_enqueue succeeds on evicted export (3822) ========================================================== 00:11:56 (1721880716) debug=+dlmtrace fail_loc=0x8000030b using seed 3980478386 running for 500 iterations total: 500 stats in 0 seconds: inf stats/second ldlm.namespaces.MGC192.168.204.135@tcp.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a837a000.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a93f8800.early_lock_cancel=0 ldlm.namespaces.lustre-OST0000-osc-ffff8800a837a000.early_lock_cancel=0 ldlm.namespaces.lustre-OST0000-osc-ffff8800a93f8800.early_lock_cancel=0 ldlm.namespaces.lustre-OST0001-osc-ffff8800a837a000.early_lock_cancel=0 ldlm.namespaces.lustre-OST0001-osc-ffff8800a93f8800.early_lock_cancel=0 fail_loc=0x80000305 Error in opening file "/mnt/lustre2/d18.replay-dual/f18.replay-dual"(flags=O_RDONLY) 2: No such file or directory ldlm.namespaces.MGC192.168.204.135@tcp.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a837a000.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0000-mdc-ffff8800a93f8800.early_lock_cancel=1 ldlm.namespaces.lustre-OST0000-osc-ffff8800a837a000.early_lock_cancel=1 ldlm.namespaces.lustre-OST0000-osc-ffff8800a93f8800.early_lock_cancel=1 ldlm.namespaces.lustre-OST0001-osc-ffff8800a837a000.early_lock_cancel=1 ldlm.namespaces.lustre-OST0001-osc-ffff8800a93f8800.early_lock_cancel=1 fail_loc=0 fail_loc=0 PASS 18 (46s) == replay-dual test 19: resend of open request =========== 00:12:42 (1721880762) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre fail_loc=0x157 - open/close 0 (time 1721880850.14 total 86.02 last 0.00) total: 1 open/close in 86.02 seconds: 0.01 ops/second fail_loc=0 Failing mds1 on oleg435-server Stopping /mnt/lustre-mds1 (opts:) on oleg435-server 00:14:11 (1721880851) shut down Failover mds1 to oleg435-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg435-server: oleg435-server.virtnet: executing set_default_debug -1 all pdsh@oleg435-client: oleg435-server: ssh exited with exit code 1 Started lustre-MDT0000 00:14:24 (1721880864) targets are mounted 00:14:24 (1721880864) facet_failover done oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 19 (106s) == replay-dual test 20: recovery time is not increasing == 00:14:28 (1721880868) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg435-server Stopping /mnt/lustre-mds1 (opts:) on oleg435-server 00:14:31 (1721880871) shut down Failover mds1 to oleg435-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg435-server: oleg435-server.virtnet: executing set_default_debug -1 all pdsh@oleg435-client: oleg435-server: ssh exited with exit code 1 Started lustre-MDT0000 00:14:43 (1721880883) targets are mounted 00:14:43 (1721880883) facet_failover done oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Starting client: oleg435-client.virtnet: -o user_xattr,flock oleg435-server@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg435-server Stopping /mnt/lustre-mds1 (opts:) on oleg435-server 00:17:08 (1721881028) shut down Failover mds1 to oleg435-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg435-server: oleg435-server.virtnet: executing set_default_debug -1 all pdsh@oleg435-client: oleg435-server: ssh exited with exit code 1 Started lustre-MDT0000 00:17:21 (1721881041) targets are mounted 00:17:21 (1721881041) facet_failover done oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Starting client: oleg435-client.virtnet: -o user_xattr,flock oleg435-server@tcp:/lustre /mnt/lustre2 PASS 20 (316s) == replay-dual test 21a: commit on sharing =============== 00:19:44 (1721881184) mdt.lustre-MDT0000.commit_on_sharing=1 Replay barrier on lustre-MDT0000 Failing mds1 on oleg435-server Stopping /mnt/lustre-mds1 (opts:) on oleg435-server 00:19:47 (1721881187) shut down Failover mds1 to oleg435-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg435-server: oleg435-server.virtnet: executing set_default_debug -1 all pdsh@oleg435-client: oleg435-server: ssh exited with exit code 1 Started lustre-MDT0000 00:19:59 (1721881199) targets are mounted 00:19:59 (1721881199) facet_failover done Starting client: oleg435-client.virtnet: -o user_xattr,flock oleg435-server@tcp:/lustre /mnt/lustre2 mdt.lustre-MDT0000.commit_on_sharing=0 PASS 21a (156s) SKIP: replay-dual test_21b skipping SLOW test 21b == replay-dual test 22a: c1 lfs mkdir -i 1 dir1, M1 drop reply & fail, c2 mkdir dir1/dir ========================================================== 00:22:21 (1721881341) SKIP: replay-dual test_22a needs >= 2 MDTs SKIP 22a (0s) == replay-dual test 22b: c1 lfs mkdir -i 1 d1, M1 drop reply & fail M0/M1, c2 mkdir d1/dir ========================================================== 00:22:21 (1721881341) SKIP: replay-dual test_22b needs >= 2 MDTs SKIP 22b (1s) == replay-dual test 22c: c1 lfs mkdir -i 1 d1, M1 drop update & fail M1, c2 mkdir d1/dir ========================================================== 00:22:22 (1721881342) SKIP: replay-dual test_22c needs >= 2 MDTs SKIP 22c (1s) == replay-dual test 22d: c1 lfs mkdir -i 1 d1, M1 drop update & fail M0/M1,c2 mkdir d1/dir ========================================================== 00:22:23 (1721881343) SKIP: replay-dual test_22d needs >= 2 MDTs SKIP 22d (0s) == replay-dual test 23a: c1 rmdir d1, M1 drop reply and fail, client2 mkdir d1 ========================================================== 00:22:23 (1721881343) SKIP: replay-dual test_23a needs >= 2 MDTs SKIP 23a (1s) == replay-dual test 23b: c1 rmdir d1, M1 drop reply and fail M0/M1, c2 mkdir d1 ========================================================== 00:22:24 (1721881344) SKIP: replay-dual test_23b needs >= 2 MDTs SKIP 23b (1s) == replay-dual test 23c: c1 rmdir d1, M0 drop update reply and fail M0, c2 mkdir d1 ========================================================== 00:22:25 (1721881345) SKIP: replay-dual test_23c needs >= 2 MDTs SKIP 23c (1s) == replay-dual test 23d: c1 rmdir d1, M0 drop update reply and fail M0/M1, c2 mkdir d1 ========================================================== 00:22:26 (1721881346) SKIP: replay-dual test_23d needs >= 2 MDTs SKIP 23d (0s) == replay-dual test 24: reconstruct on non-existing object ========================================================== 00:22:26 (1721881346) fail_loc=0x119 fail_loc=0 truncate: cannot truncate '/mnt/lustre/f24.replay-dual' to length 100: No such file or directory PASS 24 (87s) == replay-dual test 25: replay|resend ==================== 00:23:53 (1721881433) 1+0 records in 1+0 records out 512 bytes (512 B) copied, 0.0035245 s, 145 kB/s fail_loc=0x304 fail_loc=0x80000325 Failing ost1 on oleg435-server Stopping /mnt/lustre-ost1 (opts:) on oleg435-server 00:23:56 (1721881436) shut down Failover ost1 to oleg435-server mount facets: ost1 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg435-server: oleg435-server.virtnet: executing set_default_debug -1 all pdsh@oleg435-client: oleg435-server: ssh exited with exit code 1 Started lustre-OST0000 00:24:09 (1721881449) targets are mounted 00:24:09 (1721881449) facet_failover done oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 6612: 12467 Terminated LUSTRE="/home/green/git/lustre-release/lustre" bash -c "multiop /mnt/lustre2/f25.replay-dual Ow512" fail_loc=0 PASS 25 (20s) == replay-dual test 26: dbench and tar with mds failover ========================================================== 00:24:13 (1721881453) Starting client oleg435-client.virtnet: -o user_xattr,flock oleg435-server@tcp:/lustre /mnt/lustre Started clients oleg435-client.virtnet: 192.168.204.135@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) Started tar loop with pid 13964 Started dbench loop with 13965 looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Thu Jul 25 00:24:15 EDT 2024 waiting for dbench pid 13996 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs failed to create barrier semaphore 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 255 9.66 MB/sec warmup 1 sec latency 27.344 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3679232 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3697664 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7376896 1% /mnt/lustre 1 500 8.73 MB/sec warmup 2 sec latency 20.986 ms 1 778 7.12 MB/sec warmup 3 sec latency 48.877 ms test_26 fail mds1 1 times 1 986 5.48 MB/sec warmup 4 sec latency 71.598 ms Failing mds1 on oleg435-server Stopping /mnt/lustre-mds1 (opts:) on oleg435-server 1 1252 4.62 MB/sec warmup 5 sec latency 136.617 ms 00:24:20 (1721881460) shut down 1 1252 3.85 MB/sec warmup 6 sec latency 1136.738 ms 1 1252 3.30 MB/sec warmup 7 sec latency 2136.923 ms 1 1252 2.89 MB/sec warmup 8 sec latency 3137.113 ms 1 1252 2.57 MB/sec warmup 9 sec latency 4137.300 ms 1 1252 2.31 MB/sec warmup 10 sec latency 5137.468 ms 1 1252 2.10 MB/sec warmup 11 sec latency 6137.638 ms 1 1252 1.92 MB/sec warmup 12 sec latency 7137.819 ms 1 1252 1.78 MB/sec warmup 13 sec latency 8137.978 ms 1 1252 1.65 MB/sec warmup 14 sec latency 9138.126 ms 1 1252 1.54 MB/sec warmup 15 sec latency 10138.301 ms Failover mds1 to oleg435-server mount facets: mds1 1 1252 1.44 MB/sec warmup 16 sec latency 11138.467 ms Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 1252 1.36 MB/sec warmup 17 sec latency 12138.636 ms oleg435-server: oleg435-server.virtnet: executing set_default_debug -1 all pdsh@oleg435-client: oleg435-server: ssh exited with exit code 1 Started lustre-MDT0000 00:24:33 (1721881473) targets are mounted 00:24:33 (1721881473) facet_failover done 1 1252 1.28 MB/sec warmup 18 sec latency 13138.761 ms 1 1252 1.22 MB/sec warmup 19 sec latency 14138.854 ms 1 1252 0.00 MB/sec execute 1 sec latency 16139.089 ms 1 1252 0.00 MB/sec execute 2 sec latency 17139.255 ms 1 1252 0.00 MB/sec execute 3 sec latency 18139.367 ms oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 1318 0.26 MB/sec execute 4 sec latency 18884.119 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 1603 0.61 MB/sec execute 5 sec latency 48.236 ms 1 1886 0.55 MB/sec execute 6 sec latency 48.821 ms 1 2191 0.52 MB/sec execute 7 sec latency 19.459 ms 1 2489 0.70 MB/sec execute 8 sec latency 49.023 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 4864 2203776 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 37888 3572736 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 19456 3586048 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 57344 7158784 1% /mnt/lustre 1 2990 1.16 MB/sec execute 9 sec latency 39.139 ms 1 3558 1.53 MB/sec execute 10 sec latency 28.473 ms 1 3868 1.52 MB/sec execute 11 sec latency 13.224 ms test_26 fail mds1 2 times Failing mds1 on oleg435-server Stopping /mnt/lustre-mds1 (opts:) on oleg435-server 1 4102 1.42 MB/sec execute 12 sec latency 42.102 ms 00:24:48 (1721881488) shut down 1 4102 1.31 MB/sec execute 13 sec latency 1042.279 ms 1 4102 1.21 MB/sec execute 14 sec latency 2042.584 ms 1 4102 1.13 MB/sec execute 15 sec latency 3042.731 ms 1 4102 1.06 MB/sec execute 16 sec latency 4042.929 ms 1 4102 1.00 MB/sec execute 17 sec latency 5043.085 ms 1 4102 0.94 MB/sec execute 18 sec latency 6043.233 ms 1 4102 0.89 MB/sec execute 19 sec latency 7043.387 ms 1 4102 0.85 MB/sec execute 20 sec latency 8043.545 ms 1 4102 0.81 MB/sec execute 21 sec latency 9043.746 ms 1 4102 0.77 MB/sec execute 22 sec latency 10043.908 ms Failover mds1 to oleg435-server mount facets: mds1 1 4102 0.74 MB/sec execute 23 sec latency 11044.088 ms Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 4102 0.71 MB/sec execute 24 sec latency 12044.277 ms oleg435-server: oleg435-server.virtnet: executing set_default_debug -1 all pdsh@oleg435-client: oleg435-server: ssh exited with exit code 1 1 4102 0.68 MB/sec execute 25 sec latency 13044.407 ms Started lustre-MDT0000 00:25:00 (1721881500) targets are mounted 00:25:00 (1721881500) facet_failover done 1 4102 0.65 MB/sec execute 26 sec latency 14044.605 ms 1 4102 0.63 MB/sec execute 27 sec latency 15044.772 ms 1 4102 0.61 MB/sec execute 28 sec latency 16044.944 ms 1 4102 0.59 MB/sec execute 29 sec latency 17045.120 ms 1 4102 0.57 MB/sec execute 30 sec latency 18045.311 ms 1 4102 0.55 MB/sec execute 31 sec latency 19045.555 ms 1 4102 0.53 MB/sec execute 32 sec latency 20045.792 ms 1 4102 0.52 MB/sec execute 33 sec latency 21045.932 ms 1 4114 0.50 MB/sec execute 34 sec latency 21966.904 ms oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 4309 0.50 MB/sec execute 35 sec latency 84.520 ms 1 4528 0.50 MB/sec execute 36 sec latency 40.591 ms 1 4795 0.52 MB/sec execute 37 sec latency 32.121 ms 1 5119 0.58 MB/sec execute 38 sec latency 59.104 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 6400 2202112 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 39936 3478528 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 24576 3489792 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 64512 6968320 1% /mnt/lustre 1 5342 0.57 MB/sec execute 39 sec latency 50.512 ms 1 5604 0.57 MB/sec execute 40 sec latency 28.406 ms 1 5936 0.59 MB/sec execute 41 sec latency 55.844 ms test_26 fail mds1 3 times Failing mds1 on oleg435-server 1 6308 0.67 MB/sec execute 42 sec latency 37.349 ms Stopping /mnt/lustre-mds1 (opts:) on oleg435-server 00:25:18 (1721881518) shut down 1 6457 0.67 MB/sec execute 43 sec latency 691.592 ms 1 6457 0.66 MB/sec execute 44 sec latency 1691.722 ms 1 6457 0.64 MB/sec execute 45 sec latency 2691.902 ms 1 6457 0.63 MB/sec execute 46 sec latency 3692.048 ms 1 6457 0.61 MB/sec execute 47 sec latency 4692.196 ms 1 6457 0.60 MB/sec execute 48 sec latency 5692.350 ms 1 6457 0.59 MB/sec execute 49 sec latency 6692.498 ms 1 6457 0.58 MB/sec execute 50 sec latency 7692.669 ms 1 6457 0.57 MB/sec execute 51 sec latency 8692.822 ms 1 6457 0.55 MB/sec execute 52 sec latency 9693.011 ms 1 6457 0.54 MB/sec execute 53 sec latency 10693.183 ms Failover mds1 to oleg435-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 6457 0.53 MB/sec execute 54 sec latency 11693.339 ms oleg435-server: oleg435-server.virtnet: executing set_default_debug -1 all 1 6457 0.52 MB/sec execute 55 sec latency 12693.555 ms pdsh@oleg435-client: oleg435-server: ssh exited with exit code 1 Started lustre-MDT0000 00:25:30 (1721881530) targets are mounted 00:25:30 (1721881530) facet_failover done 1 6457 0.52 MB/sec execute 56 sec latency 13693.795 ms 1 6457 0.51 MB/sec execute 57 sec latency 14694.006 ms 1 6457 0.50 MB/sec execute 58 sec latency 15694.203 ms 1 6457 0.49 MB/sec execute 59 sec latency 16694.356 ms 1 6457 0.48 MB/sec execute 60 sec latency 17694.518 ms 1 6457 0.47 MB/sec execute 61 sec latency 18694.719 ms 1 6457 0.47 MB/sec execute 62 sec latency 19694.955 ms 1 6457 0.46 MB/sec execute 63 sec latency 20695.084 ms 1 6498 0.46 MB/sec execute 64 sec latency 21462.743 ms oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 6934 0.48 MB/sec execute 65 sec latency 44.672 ms 1 7336 0.54 MB/sec execute 66 sec latency 18.274 ms 1 7557 0.54 MB/sec execute 67 sec latency 16.316 ms 1 7780 0.53 MB/sec execute 68 sec latency 112.085 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 8192 2200448 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 41984 3381248 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 24576 3436544 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 66560 6817792 1% /mnt/lustre 1 7975 0.53 MB/sec execute 69 sec latency 53.830 ms 1 8227 0.54 MB/sec execute 70 sec latency 35.526 ms 1 8572 0.58 MB/sec execute 71 sec latency 20.110 ms test_26 fail mds1 4 times Failing mds1 on oleg435-server Stopping /mnt/lustre-mds1 (opts:) on oleg435-server 1 8804 0.57 MB/sec execute 72 sec latency 64.057 ms 00:25:48 (1721881548) shut down 1 8851 0.57 MB/sec execute 73 sec latency 800.918 ms 1 8851 0.56 MB/sec execute 74 sec latency 1801.092 ms 1 8851 0.55 MB/sec execute 75 sec latency 2801.248 ms 1 8851 0.55 MB/sec execute 76 sec latency 3801.445 ms 1 8851 0.54 MB/sec execute 77 sec latency 4801.589 ms 1 8851 0.53 MB/sec execute 78 sec latency 5801.763 ms 1 8851 0.52 MB/sec execute 79 sec latency 6801.960 ms 1 8851 0.52 MB/sec execute 80 sec latency 7802.122 ms 1 8851 0.51 MB/sec execute 81 sec latency 8802.309 ms 1 8851 0.51 MB/sec execute 82 sec latency 9802.497 ms Failover mds1 to oleg435-server 1 8851 0.50 MB/sec execute 83 sec latency 10802.638 ms mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 8851 0.49 MB/sec execute 84 sec latency 11802.862 ms oleg435-server: oleg435-server.virtnet: executing set_default_debug -1 all 1 8851 0.49 MB/sec execute 85 sec latency 12803.057 ms pdsh@oleg435-client: oleg435-server: ssh exited with exit code 1 Started lustre-MDT0000 00:26:00 (1721881560) targets are mounted 00:26:00 (1721881560) facet_failover done 1 8851 0.48 MB/sec execute 86 sec latency 13803.222 ms 1 8851 0.48 MB/sec execute 87 sec latency 14803.479 ms 1 8851 0.47 MB/sec execute 88 sec latency 15803.632 ms 1 8851 0.47 MB/sec execute 89 sec latency 16803.816 ms 1 8851 0.46 MB/sec execute 90 sec latency 17804.006 ms 1 8851 0.46 MB/sec execute 91 sec latency 18804.236 ms 1 8851 0.45 MB/sec execute 92 sec latency 19804.469 ms 1 8851 0.45 MB/sec execute 93 sec latency 20804.576 ms 1 8895 0.44 MB/sec execute 94 sec latency 21609.145 ms oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 9166 0.44 MB/sec execute 95 sec latency 33.324 ms 1 9480 0.45 MB/sec execute 96 sec latency 61.843 ms 1 9777 0.47 MB/sec execute 97 sec latency 45.362 ms 1 10296 0.50 MB/sec execute 98 sec latency 47.757 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 9600 2199040 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 43008 3423232 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 26624 3386368 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 69632 6809600 2% /mnt/lustre 1 10684 0.54 MB/sec execute 99 sec latency 51.393 ms 1 cleanup 100 sec 0 cleanup 101 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 1636 33.944 21462.726 Close 1219 19.413 21609.136 Rename 68 14.714 21.379 Unlink 307 76.330 21966.887 Qpathinfo 1519 2.807 26.524 Qfileinfo 275 0.485 2.693 Qfsinfo 268 0.501 2.564 Sfileinfo 120 8.726 16.147 Find 585 1.292 24.330 WriteX 825 1.942 6.267 ReadX 2748 0.146 35.039 LockX 6 1.512 1.574 UnlockX 6 1.548 1.753 Flush 105 29.383 112.072 Throughput 0.541169 MB/sec 1 clients 1 procs max_latency=21966.904 ms stopping dbench on /mnt/lustre at Thu Jul 25 00:26:16 EDT 2024 with return code 0 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished test_26 fail mds1 5 times looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt Failing mds1 on oleg435-server Stopping /mnt/lustre-mds1 (opts:) on oleg435-server 00:26:18 (1721881578) shut down Failover mds1 to oleg435-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg435-server: oleg435-server.virtnet: executing set_default_debug -1 all pdsh@oleg435-client: oleg435-server: ssh exited with exit code 1 Started lustre-MDT0000 00:26:30 (1721881590) targets are mounted 00:26:30 (1721881590) facet_failover done '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Thu Jul 25 00:26:34 EDT 2024 waiting for dbench pid 18712 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs 0 of 1 processes prepared for launch 0 sec oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 of 1 processes prepared for launch 0 sec releasing clients mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 184 7.39 MB/sec warmup 1 sec latency 292.903 ms 1 421 7.43 MB/sec warmup 2 sec latency 353.029 ms 1 721 7.10 MB/sec warmup 3 sec latency 53.065 ms 1 934 5.44 MB/sec warmup 4 sec latency 42.767 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 10880 2197760 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 49152 3481600 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 22528 3533824 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 71680 7015424 2% /mnt/lustre 1 1215 4.61 MB/sec warmup 5 sec latency 46.463 ms 1 1588 4.36 MB/sec warmup 6 sec latency 42.678 ms 1 1887 3.77 MB/sec warmup 7 sec latency 39.744 ms test_26 fail mds1 6 times Failing mds1 on oleg435-server Stopping /mnt/lustre-mds1 (opts:) on oleg435-server 1 2204 3.35 MB/sec warmup 8 sec latency 59.575 ms 00:26:42 (1721881602) shut down 1 2204 2.97 MB/sec warmup 9 sec latency 1059.728 ms 1 2204 2.68 MB/sec warmup 10 sec latency 2059.937 ms 1 2204 2.43 MB/sec warmup 11 sec latency 3060.101 ms 1 2204 2.23 MB/sec warmup 12 sec latency 4060.266 ms 1 2204 2.06 MB/sec warmup 13 sec latency 5060.412 ms 1 2204 1.91 MB/sec warmup 14 sec latency 6060.558 ms 1 2204 1.78 MB/sec warmup 15 sec latency 7060.721 ms 1 2204 1.67 MB/sec warmup 16 sec latency 8060.874 ms 1 2204 1.57 MB/sec warmup 17 sec latency 9061.052 ms 1 2204 1.49 MB/sec warmup 18 sec latency 10061.168 ms Failover mds1 to oleg435-server mount facets: mds1 1 2204 1.41 MB/sec warmup 19 sec latency 11061.372 ms Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg435-server: oleg435-server.virtnet: executing set_default_debug -1 all pdsh@oleg435-client: oleg435-server: ssh exited with exit code 1 1 2204 0.00 MB/sec execute 1 sec latency 13061.572 ms Started lustre-MDT0000 00:26:55 (1721881615) targets are mounted 00:26:55 (1721881615) facet_failover done 1 2204 0.00 MB/sec execute 2 sec latency 14061.745 ms 1 2204 0.00 MB/sec execute 3 sec latency 15061.893 ms 1 2204 0.00 MB/sec execute 4 sec latency 16062.004 ms 1 2204 0.00 MB/sec execute 5 sec latency 17062.100 ms oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 2436 0.22 MB/sec execute 6 sec latency 17224.063 ms 1 2931 0.96 MB/sec execute 7 sec latency 46.537 ms 1 3382 1.09 MB/sec execute 8 sec latency 47.537 ms 1 3786 1.44 MB/sec execute 9 sec latency 17.582 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 11264 2197376 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 51200 3511296 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 23552 3563520 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 74752 7074816 2% /mnt/lustre 1 4016 1.33 MB/sec execute 10 sec latency 17.608 ms 1 4257 1.23 MB/sec execute 11 sec latency 44.412 ms 1 4455 1.17 MB/sec execute 12 sec latency 52.470 ms test_26 fail mds1 7 times Failing mds1 on oleg435-server 1 4703 1.18 MB/sec execute 13 sec latency 64.152 ms Stopping /mnt/lustre-mds1 (opts:) on oleg435-server 1 4834 1.11 MB/sec execute 14 sec latency 513.428 ms 1 4834 1.03 MB/sec execute 15 sec latency 1513.618 ms 1 4834 0.97 MB/sec execute 16 sec latency 2513.805 ms 1 4834 0.91 MB/sec execute 17 sec latency 3513.956 ms 1 4834 0.86 MB/sec execute 18 sec latency 4514.139 ms 1 4834 0.82 MB/sec execute 19 sec latency 5514.260 ms 1 4834 0.77 MB/sec execute 20 sec latency 6514.477 ms 00:27:14 (1721881634) shut down 1 4834 0.74 MB/sec execute 21 sec latency 7514.704 ms 1 4834 0.70 MB/sec execute 22 sec latency 8514.983 ms 1 4834 0.67 MB/sec execute 23 sec latency 9515.201 ms 1 4834 0.65 MB/sec execute 24 sec latency 10515.367 ms 1 4834 0.62 MB/sec execute 25 sec latency 11515.561 ms 1 4834 0.60 MB/sec execute 26 sec latency 12515.843 ms 1 4834 0.57 MB/sec execute 27 sec latency 13516.024 ms 1 4834 0.55 MB/sec execute 28 sec latency 14516.189 ms 1 4834 0.53 MB/sec execute 29 sec latency 15516.338 ms 1 4834 0.52 MB/sec execute 30 sec latency 16516.517 ms Failover mds1 to oleg435-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 4834 0.50 MB/sec execute 31 sec latency 17516.645 ms 1 4834 0.48 MB/sec execute 32 sec latency 18516.788 ms oleg435-server: oleg435-server.virtnet: executing set_default_debug -1 all pdsh@oleg435-client: oleg435-server: ssh exited with exit code 1 Started lustre-MDT0000 00:27:26 (1721881646) targets are mounted 00:27:26 (1721881646) facet_failover done 1 4834 0.47 MB/sec execute 33 sec latency 19516.936 ms 1 4834 0.46 MB/sec execute 34 sec latency 20517.128 ms 1 4834 0.44 MB/sec execute 35 sec latency 21517.279 ms 1 4834 0.43 MB/sec execute 36 sec latency 22517.426 ms 1 4834 0.42 MB/sec execute 37 sec latency 23517.649 ms 1 4834 0.41 MB/sec execute 38 sec latency 24517.795 ms 1 4834 0.40 MB/sec execute 39 sec latency 25517.978 ms 1 4834 0.39 MB/sec execute 40 sec latency 26518.119 ms 1 4834 0.38 MB/sec execute 41 sec latency 27518.277 ms 1 4834 0.37 MB/sec execute 42 sec latency 28518.429 ms 1 4834 0.36 MB/sec execute 43 sec latency 29518.631 ms 1 4834 0.35 MB/sec execute 44 sec latency 30518.801 ms 1 4834 0.34 MB/sec execute 45 sec latency 31518.929 ms 1 4834 0.34 MB/sec execute 46 sec latency 32519.081 ms 1 4834 0.33 MB/sec execute 47 sec latency 33519.256 ms 1 4834 0.32 MB/sec execute 48 sec latency 34519.423 ms 1 4834 0.32 MB/sec execute 49 sec latency 35519.547 ms 1 4836 0.31 MB/sec execute 50 sec latency 36452.364 ms oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 5151 0.36 MB/sec execute 51 sec latency 67.307 ms 1 5370 0.36 MB/sec execute 52 sec latency 88.615 ms 1 5631 0.36 MB/sec execute 53 sec latency 76.586 ms 1 5936 0.38 MB/sec execute 54 sec latency 70.983 ms 1 6274 0.44 MB/sec execute 55 sec latency 64.136 ms 1 6759 0.48 MB/sec execute 56 sec latency 68.876 ms 1 7190 0.54 MB/sec execute 57 sec latency 22.671 ms 1 7433 0.55 MB/sec execute 58 sec latency 18.910 ms 1 7655 0.55 MB/sec execute 59 sec latency 15.506 ms 1 7874 0.55 MB/sec execute 60 sec latency 90.006 ms 1 8086 0.55 MB/sec execute 61 sec latency 58.898 ms 1 8367 0.56 MB/sec execute 62 sec latency 43.491 ms 1 8690 0.60 MB/sec execute 63 sec latency 50.116 ms 1 8936 0.59 MB/sec execute 64 sec latency 60.367 ms 1 9260 0.59 MB/sec execute 65 sec latency 41.115 ms 1 9552 0.60 MB/sec execute 66 sec latency 48.544 ms 1 10021 0.67 MB/sec execute 67 sec latency 52.637 ms 1 10479 0.69 MB/sec execute 68 sec latency 49.084 ms dbench killed by signal 15 stopping dbench on /mnt/lustre at Thu Jul 25 00:28:03 EDT 2024 with return code 0 1 10882 0.74 MB/sec execute 69 sec latency 13.148 ms 18712 pts/0 S+ 0:00 dbench -c client.txt 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100 killed dbench main pid 18712 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished PASS 26 (232s) == replay-dual test 28: lock replay should be ordered: waiting after granted ========================================================== 00:28:05 (1721881685) 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00320064 s, 1.3 MB/s fail_loc=0x80000324 fail_loc=0x32a Failing ost1 on oleg435-server Stopping /mnt/lustre-ost1 (opts:) on oleg435-server 00:28:15 (1721881695) shut down Failover ost1 to oleg435-server mount facets: ost1 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg435-server: oleg435-server.virtnet: executing set_default_debug -1 all pdsh@oleg435-client: oleg435-server: ssh exited with exit code 1 Started lustre-OST0000 00:28:27 (1721881707) targets are mounted 00:28:27 (1721881707) facet_failover done 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00463955 s, 883 kB/s oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec PASS 28 (29s) == replay-dual test 29: replay vs update with the same xid ========================================================== 00:28:34 (1721881714) SKIP: replay-dual test_29 needs >= 2 MDTs SKIP 29 (0s) == replay-dual test 30: layout lock replay is not blocked on IO ========================================================== 00:28:35 (1721881715) 10+0 records in 10+0 records out 40960 bytes (41 kB) copied, 0.00798751 s, 5.1 MB/s 10+0 records in 10+0 records out 40960 bytes (41 kB) copied, 0.00760718 s, 5.4 MB/s fail_loc=0x32e fail_val=4 Failing mds1 on oleg435-server Stopping /mnt/lustre-mds1 (opts:) on oleg435-server 00:28:37 (1721881717) shut down Failover mds1 to oleg435-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg435-server: oleg435-server.virtnet: executing set_default_debug -1 all pdsh@oleg435-client: oleg435-server: ssh exited with exit code 1 Started lustre-MDT0000 00:28:50 (1721881730) targets are mounted 00:28:50 (1721881730) facet_failover done 160+0 records in 160+0 records out 81920 bytes (82 kB) copied, 21.6546 s, 3.8 kB/s oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 30 (25s) == replay-dual test 31: deadlock on file_remove_privs and occupied mod rpc slots ========================================================== 00:29:00 (1721881740) Failing ost1 on oleg435-server Stopping /mnt/lustre-ost1 (opts:) on oleg435-server 00:29:02 (1721881742) shut down Creating to objid 3169 on ost lustre-OST0000... total: 32 open/close in 0.18 seconds: 172.98 ops/second at_max=0 fail_loc=0x80001420 file /mnt/lustre2/d31.replay-dual/mdtdir/f31.replay-dual is ready Failover ost1 to oleg435-server mount facets: ost1 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg435-server: oleg435-server.virtnet: executing set_default_debug -1 all pdsh@oleg435-client: oleg435-server: ssh exited with exit code 1 Started lustre-OST0000 00:29:15 (1721881755) targets are mounted 00:29:15 (1721881755) facet_failover done oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE FULL state after 0 sec pids: 25357 25358 25363 25364 25365 25366 25367 25368 25369 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3712 2204800 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 20480 3703808 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 23552 7470080 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 493378 394 492984 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 118155 395 117760 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 117724 508 117216 1% /mnt/lustre[OST:1] filesystem_summary: 235370 394 234976 1% /mnt/lustre at_max=600 PASS 31 (19s) == replay-dual test 32: gap in update llog shouldn't break recovery ========================================================== 00:29:19 (1721881759) SKIP: replay-dual test_32 needs >= 2 MDTs SKIP 32 (1s) == replay-dual test 33: Check for OBD_INCOMPAT_MULTI_RPCS in last_rcvd after abort_recovery ========================================================== 00:29:20 (1721881760) SKIP: replay-dual test_33 ldiskfs only test SKIP 33 (1s) == replay-dual test complete, duration 2024 sec ========== 00:29:21 (1721881761) === replay-dual: start cleanup 00:29:21 (1721881761) === Stopping clients: oleg435-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg435-client.virtnet /mnt/lustre2 opts: === replay-dual: finish cleanup 00:29:22 (1721881762) ===