-----============= acceptance-small: replay-dual ============----- Tue Jun 17 12:45:09 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg203-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/replay-dual.*ex: No such file or directory excepting tests: 14b 21b 21b skipping tests SLOW=no: 21b === replay-dual: start setup 12:45:17 (1750178717) === Starting client oleg203-client.virtnet: -o user_xattr,flock 192.168.202.103@tcp:/lustre /mnt/lustre2 Started clients oleg203-client.virtnet: 192.168.202.103@tcp:/lustre on /mnt/lustre2 type lustre (rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) oleg203-client.virtnet: executing check_config_client /mnt/lustre oleg203-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg203-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800a9d1a000.idle_timeout=debug osc.lustre-OST0000-osc-ffff88012ab4c800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800a9d1a000.idle_timeout=debug osc.lustre-OST0001-osc-ffff88012ab4c800.idle_timeout=debug disable quota as required oleg203-server: oleg203-server.virtnet: executing set_default_debug -1 all === replay-dual: finish setup 12:45:25 (1750178725) === == replay-dual test 0a: expired recovery with lost client ========================================================== 12:45:26 (1750178726) Check file is LU482_FAILED=/tmp/replay-dual.lu482.gZptKa UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre total: 50 open/close in 0.51 seconds: 98.64 ops/second fail_loc=0x80000514 Failing mds1 on oleg203-server Stopping /mnt/lustre-mds1 (opts:) on oleg203-server 12:45:29 (1750178729) shut down facet: mds1 facet_host: oleg203-server facet_failover_host: oleg203-server Failover mds1 to oleg203-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg203-server: oleg203-server.virtnet: executing set_default_debug -1 all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Started lustre-MDT0000 12:45:43 (1750178743) targets are mounted 12:45:43 (1750178743) facet_failover done Starting client: oleg203-client.virtnet: -o user_xattr,flock 192.168.202.103@tcp:/lustre /mnt/lustre2 - unlinked 0 (time 1750178854 ; total 0 ; last 0) total: 50 unlinks in 1 seconds: 50.000000 unlinks/second PASS 0a (130s) == replay-dual test 0b: lost client during waiting for next transno ========================================================== 12:47:36 (1750178856) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg203-server Stopping /mnt/lustre-mds1 (opts:) on oleg203-server 12:47:40 (1750178860) shut down facet: mds1 facet_host: oleg203-server facet_failover_host: oleg203-server Failover mds1 to oleg203-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg203-server: oleg203-server.virtnet: executing set_default_debug -1 all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Started lustre-MDT0000 12:47:53 (1750178873) targets are mounted 12:47:53 (1750178873) facet_failover done Starting client: oleg203-client.virtnet: -o user_xattr,flock 192.168.202.103@tcp:/lustre /mnt/lustre Starting client: oleg203-client.virtnet: -o user_xattr,flock 192.168.202.103@tcp:/lustre /mnt/lustre2 PASS 0b (88s) == replay-dual test 1: |X| simple create ================= 12:49:04 (1750178944) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg203-server Stopping /mnt/lustre-mds1 (opts:) on oleg203-server 12:49:07 (1750178947) shut down facet: mds1 facet_host: oleg203-server facet_failover_host: oleg203-server Failover mds1 to oleg203-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg203-server: oleg203-server.virtnet: executing set_default_debug -1 all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Started lustre-MDT0000 12:49:19 (1750178959) targets are mounted 12:49:19 (1750178959) facet_failover done oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 1 (20s) == replay-dual test 2: |X| mkdir adir ==================== 12:49:24 (1750178964) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg203-server Stopping /mnt/lustre-mds1 (opts:) on oleg203-server 12:49:27 (1750178967) shut down facet: mds1 facet_host: oleg203-server facet_failover_host: oleg203-server Failover mds1 to oleg203-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg203-server: oleg203-server.virtnet: executing set_default_debug -1 all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Started lustre-MDT0000 12:49:40 (1750178980) targets are mounted 12:49:40 (1750178980) facet_failover done oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 2 (22s) == replay-dual test 3: |X| mkdir adir, mkdir adir/bdir === 12:49:46 (1750178986) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg203-server Stopping /mnt/lustre-mds1 (opts:) on oleg203-server 12:49:49 (1750178989) shut down facet: mds1 facet_host: oleg203-server facet_failover_host: oleg203-server Failover mds1 to oleg203-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg203-server: oleg203-server.virtnet: executing set_default_debug -1 all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Started lustre-MDT0000 12:50:02 (1750179002) targets are mounted 12:50:02 (1750179002) facet_failover done oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 3 (21s) == replay-dual test 4: |X| mkdir adir (-EEXIST), mkdir adir/bdir ========================================================== 12:50:07 (1750179007) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre mkdir: cannot create directory '/mnt/lustre/adir': File exists Failing mds1 on oleg203-server Stopping /mnt/lustre-mds1 (opts:) on oleg203-server 12:50:09 (1750179009) shut down facet: mds1 facet_host: oleg203-server facet_failover_host: oleg203-server Failover mds1 to oleg203-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg203-server: oleg203-server.virtnet: executing set_default_debug -1 all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Started lustre-MDT0000 12:50:22 (1750179022) targets are mounted 12:50:22 (1750179022) facet_failover done oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 4 (20s) == replay-dual test 5: open, unlink |X| close ============ 12:50:27 (1750179027) multiop /mnt/lustre2/a vo_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7265 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg203-server Stopping /mnt/lustre-mds1 (opts:) on oleg203-server 12:50:31 (1750179031) shut down facet: mds1 facet_host: oleg203-server facet_failover_host: oleg203-server Failover mds1 to oleg203-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg203-server: oleg203-server.virtnet: executing set_default_debug -1 all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Started lustre-MDT0000 12:50:44 (1750179044) targets are mounted 12:50:44 (1750179044) facet_failover done oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 5 (23s) == replay-dual test 6: open1, open2, unlink |X| close1 [fail mds1] close2 ========================================================== 12:50:50 (1750179050) multiop /mnt/lustre2/a vo_c TMPPIPE=/tmp/multiop_open_wait_pipe.7265 multiop /mnt/lustre/a vo_c TMPPIPE=/tmp/multiop_open_wait_pipe.7265 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg203-server Stopping /mnt/lustre-mds1 (opts:) on oleg203-server 12:50:53 (1750179053) shut down facet: mds1 facet_host: oleg203-server facet_failover_host: oleg203-server Failover mds1 to oleg203-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg203-server: oleg203-server.virtnet: executing set_default_debug -1 all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Started lustre-MDT0000 12:51:06 (1750179066) targets are mounted 12:51:06 (1750179066) facet_failover done oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 6 (21s) == replay-dual test 8: replay of resent request ========== 12:51:11 (1750179071) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre fail_loc=0x119 fail_loc=0 Failing mds1 on oleg203-server Stopping /mnt/lustre-mds1 (opts:) on oleg203-server 12:51:30 (1750179090) shut down facet: mds1 facet_host: oleg203-server facet_failover_host: oleg203-server Failover mds1 to oleg203-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg203-server: oleg203-server.virtnet: executing set_default_debug -1 all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Started lustre-MDT0000 12:51:43 (1750179103) targets are mounted 12:51:43 (1750179103) facet_failover done oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 8 (37s) == replay-dual test 9: resending a replayed create ======= 12:51:48 (1750179108) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre fail_loc=0x80000119 Failing mds1 on oleg203-server Stopping /mnt/lustre-mds1 (opts:) on oleg203-server 12:51:51 (1750179111) shut down facet: mds1 facet_host: oleg203-server facet_failover_host: oleg203-server Failover mds1 to oleg203-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg203-server: oleg203-server.virtnet: executing set_default_debug -1 all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Started lustre-MDT0000 12:52:04 (1750179124) targets are mounted 12:52:04 (1750179124) facet_failover done oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec fail_loc=0 PASS 9 (36s) == replay-dual test 10: resending a replayed unlink ====== 12:52:24 (1750179144) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre fail_loc=0x80000119 Failing mds1 on oleg203-server Stopping /mnt/lustre-mds1 (opts:) on oleg203-server 12:52:27 (1750179147) shut down facet: mds1 facet_host: oleg203-server facet_failover_host: oleg203-server Failover mds1 to oleg203-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg203-server: oleg203-server.virtnet: executing set_default_debug -1 all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Started lustre-MDT0000 12:52:40 (1750179160) targets are mounted 12:52:40 (1750179160) facet_failover done oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec fail_loc=0 PASS 10 (38s) == replay-dual test 11: both clients timeout during replay ========================================================== 12:53:02 (1750179182) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre fail_loc=0x0119 Failing mds1 on oleg203-server Stopping /mnt/lustre-mds1 (opts:) on oleg203-server 12:53:05 (1750179185) shut down facet: mds1 facet_host: oleg203-server facet_failover_host: oleg203-server Failover mds1 to oleg203-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg203-server: oleg203-server.virtnet: executing set_default_debug -1 all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Started lustre-MDT0000 12:53:22 (1750179202) targets are mounted 12:53:22 (1750179202) facet_failover done oleg203-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 10 sec fail_loc=0 PASS 11 (39s) == replay-dual test 12: open resend timeout ============== 12:53:41 (1750179221) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre multiop /mnt/lustre/f12.replay-dual vmo_c TMPPIPE=/tmp/multiop_open_wait_pipe.7265 fail_loc=0x80000302 Failing mds1 on oleg203-server Stopping /mnt/lustre-mds1 (opts:) on oleg203-server 12:53:47 (1750179227) shut down facet: mds1 facet_host: oleg203-server facet_failover_host: oleg203-server Failover mds1 to oleg203-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg203-server: oleg203-server.virtnet: executing set_default_debug -1 all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Started lustre-MDT0000 12:54:02 (1750179242) targets are mounted 12:54:02 (1750179242) facet_failover done fail_loc=0 /mnt/lustre/f12.replay-dual /mnt/lustre/f12.replay-dual has type file OK PASS 12 (24s) == replay-dual test 13: close resend timeout ============= 12:54:06 (1750179246) multiop /mnt/lustre/f13.replay-dual vmo_c TMPPIPE=/tmp/multiop_open_wait_pipe.7265 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre fail_loc=0x80000115 Failing mds1 on oleg203-server Stopping /mnt/lustre-mds1 (opts:) on oleg203-server 12:54:10 (1750179250) shut down facet: mds1 facet_host: oleg203-server facet_failover_host: oleg203-server Failover mds1 to oleg203-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg203-server: oleg203-server.virtnet: executing set_default_debug -1 all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Started lustre-MDT0000 12:54:26 (1750179266) targets are mounted 12:54:26 (1750179266) facet_failover done fail_loc=0 /mnt/lustre/f13.replay-dual /mnt/lustre/f13.replay-dual has type file OK PASS 13 (36s) SKIP: replay-dual test_14b skipping ALWAYS excluded test 14b == replay-dual test 15a: timeout waiting for lost client during replay, 1 client completes ========================================================== 12:54:43 (1750179283) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre total: 25 open/close in 0.50 seconds: 50.12 ops/second total: 1 open/close in 0.02 seconds: 49.23 ops/second Failing mds1 on oleg203-server Stopping /mnt/lustre-mds1 (opts:) on oleg203-server 12:54:49 (1750179289) shut down facet: mds1 facet_host: oleg203-server facet_failover_host: oleg203-server Failover mds1 to oleg203-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg203-server: oleg203-server.virtnet: executing set_default_debug -1 all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Started lustre-MDT0000 12:55:03 (1750179303) targets are mounted 12:55:03 (1750179303) facet_failover done oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec - unlinked 0 (time 1750179378 ; total 0 ; last 0) total: 25 unlinks in 0 seconds: inf unlinks/second Starting client: oleg203-client.virtnet: -o user_xattr,flock 192.168.202.103@tcp:/lustre /mnt/lustre2 PASS 15a (98s) == replay-dual test 15c: remove multiple OST orphans ===== 12:56:21 (1750179381) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg203-server Stopping /mnt/lustre-mds1 (opts:) on oleg203-server 12:57:45 (1750179465) shut down facet: mds1 facet_host: oleg203-server facet_failover_host: oleg203-server Failover mds1 to oleg203-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg203-server: oleg203-server.virtnet: executing set_default_debug -1 all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Started lustre-MDT0000 12:58:02 (1750179482) targets are mounted 12:58:02 (1750179482) facet_failover done oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Starting client: oleg203-client.virtnet: -o user_xattr,flock 192.168.202.103@tcp:/lustre /mnt/lustre2 PASS 15c (176s) == replay-dual test 16: fail MDS during recovery (3571) == 12:59:17 (1750179557) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre total: 25 open/close in 0.60 seconds: 41.72 ops/second total: 1 open/close in 0.02 seconds: 61.33 ops/second Failing mds1 on oleg203-server Stopping /mnt/lustre-mds1 (opts:) on oleg203-server 12:59:23 (1750179563) shut down facet: mds1 facet_host: oleg203-server facet_failover_host: oleg203-server Failover mds1 to oleg203-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg203-server: oleg203-server.virtnet: executing set_default_debug -1 all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Started lustre-MDT0000 12:59:39 (1750179579) targets are mounted 12:59:39 (1750179579) facet_failover done Failing mds1 on oleg203-server Stopping /mnt/lustre-mds1 (opts:) on oleg203-server 13:00:02 (1750179602) shut down facet: mds1 facet_host: oleg203-server facet_failover_host: oleg203-server Failover mds1 to oleg203-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg203-server: oleg203-server.virtnet: executing set_default_debug -1 all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Started lustre-MDT0000 13:00:17 (1750179617) targets are mounted 13:00:17 (1750179617) facet_failover done oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec - unlinked 0 (time 1750179694 ; total 0 ; last 0) total: 25 unlinks in 0 seconds: inf unlinks/second Starting client: oleg203-client.virtnet: -o user_xattr,flock 192.168.202.103@tcp:/lustre /mnt/lustre2 PASS 16 (140s) == replay-dual test 17: fail OST during recovery (3571) == 13:01:37 (1750179697) total: 25 open/close in 0.49 seconds: 51.28 ops/second total: 1 open/close in 0.03 seconds: 34.12 ops/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing ost1 on oleg203-server Stopping /mnt/lustre-ost1 (opts:) on oleg203-server 13:01:42 (1750179702) shut down facet: ost1 facet_host: oleg203-server facet_failover_host: oleg203-server Failover ost1 to oleg203-server mount facets: ost1 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg203-server: oleg203-server.virtnet: executing set_default_debug -1 all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Started lustre-OST0000 13:01:57 (1750179717) targets are mounted 13:01:57 (1750179717) facet_failover done Failing ost1 on oleg203-server Stopping /mnt/lustre-ost1 (opts:) on oleg203-server 13:02:19 (1750179739) shut down facet: ost1 facet_host: oleg203-server facet_failover_host: oleg203-server Failover ost1 to oleg203-server mount facets: ost1 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg203-server: oleg203-server.virtnet: executing set_default_debug -1 all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Started lustre-OST0000 13:02:33 (1750179753) targets are mounted 13:02:33 (1750179753) facet_failover done oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec - unlinked 0 (time 1750179826 ; total 0 ; last 0) total: 25 unlinks in 0 seconds: inf unlinks/second Starting client: oleg203-client.virtnet: -o user_xattr,flock 192.168.202.103@tcp:/lustre /mnt/lustre2 PASS 17 (132s) == replay-dual test 18: ldlm_handle_enqueue succeeds on evicted export (3822) ========================================================== 13:03:49 (1750179829) debug=+dlmtrace fail_loc=0x8000030b using seed 3349826783 running for 500 iterations total: 500 stats in 1 seconds: 500.000000 stats/second ldlm.namespaces.MGC192.168.202.103@tcp.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0000-mdc-ffff8800aaaff000.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0000-mdc-ffff88012a460000.early_lock_cancel=0 ldlm.namespaces.lustre-OST0000-osc-ffff8800aaaff000.early_lock_cancel=0 ldlm.namespaces.lustre-OST0000-osc-ffff88012a460000.early_lock_cancel=0 ldlm.namespaces.lustre-OST0001-osc-ffff8800aaaff000.early_lock_cancel=0 ldlm.namespaces.lustre-OST0001-osc-ffff88012a460000.early_lock_cancel=0 fail_loc=0x80000305 Error in opening file "/mnt/lustre2/d18.replay-dual/f18.replay-dual"(flags=O_RDONLY) 2: No such file or directory ldlm.namespaces.MGC192.168.202.103@tcp.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0000-mdc-ffff8800aaaff000.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0000-mdc-ffff88012a460000.early_lock_cancel=1 ldlm.namespaces.lustre-OST0000-osc-ffff8800aaaff000.early_lock_cancel=1 ldlm.namespaces.lustre-OST0000-osc-ffff88012a460000.early_lock_cancel=1 ldlm.namespaces.lustre-OST0001-osc-ffff8800aaaff000.early_lock_cancel=1 ldlm.namespaces.lustre-OST0001-osc-ffff88012a460000.early_lock_cancel=1 fail_loc=0 fail_loc=0 PASS 18 (48s) == replay-dual test 19: resend of open request =========== 13:04:37 (1750179877) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre fail_loc=0x157 - open/close 0 (time 1750179967.05 total 86.02 last 0.00) total: 1 open/close in 86.02 seconds: 0.01 ops/second fail_loc=0 Failing mds1 on oleg203-server Stopping /mnt/lustre-mds1 (opts:) on oleg203-server 13:06:08 (1750179968) shut down facet: mds1 facet_host: oleg203-server facet_failover_host: oleg203-server Failover mds1 to oleg203-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg203-server: oleg203-server.virtnet: executing set_default_debug -1 all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Started lustre-MDT0000 13:06:23 (1750179983) targets are mounted 13:06:23 (1750179983) facet_failover done oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 19 (113s) == replay-dual test 20: recovery time is not increasing == 13:06:30 (1750179990) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg203-server Stopping /mnt/lustre-mds1 (opts:) on oleg203-server 13:06:34 (1750179994) shut down facet: mds1 facet_host: oleg203-server facet_failover_host: oleg203-server Failover mds1 to oleg203-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg203-server: oleg203-server.virtnet: executing set_default_debug -1 all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Started lustre-MDT0000 13:06:48 (1750180008) targets are mounted 13:06:48 (1750180008) facet_failover done oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Starting client: oleg203-client.virtnet: -o user_xattr,flock 192.168.202.103@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg203-server Stopping /mnt/lustre-mds1 (opts:) on oleg203-server 13:09:16 (1750180156) shut down facet: mds1 facet_host: oleg203-server facet_failover_host: oleg203-server Failover mds1 to oleg203-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg203-server: oleg203-server.virtnet: executing set_default_debug -1 all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Started lustre-MDT0000 13:09:32 (1750180172) targets are mounted 13:09:32 (1750180172) facet_failover done oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Starting client: oleg203-client.virtnet: -o user_xattr,flock 192.168.202.103@tcp:/lustre /mnt/lustre2 PASS 20 (325s) == replay-dual test 21a: commit on sharing =============== 13:11:55 (1750180315) mdt.lustre-MDT0000.commit_on_sharing=1 Replay barrier on lustre-MDT0000 Failing mds1 on oleg203-server Stopping /mnt/lustre-mds1 (opts:) on oleg203-server 13:11:59 (1750180319) shut down facet: mds1 facet_host: oleg203-server facet_failover_host: oleg203-server Failover mds1 to oleg203-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg203-server: oleg203-server.virtnet: executing set_default_debug -1 all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Started lustre-MDT0000 13:12:14 (1750180334) targets are mounted 13:12:14 (1750180334) facet_failover done Starting client: oleg203-client.virtnet: -o user_xattr,flock 192.168.202.103@tcp:/lustre /mnt/lustre2 mdt.lustre-MDT0000.commit_on_sharing=0 PASS 21a (161s) SKIP: replay-dual test_21b skipping SLOW test 21b == replay-dual test 22a: c1 lfs mkdir -i 1 dir1, M1 drop reply & fail, c2 mkdir dir1/dir ========================================================== 13:14:37 (1750180477) SKIP: replay-dual test_22a needs >= 2 MDTs SKIP 22a (1s) == replay-dual test 22b: c1 lfs mkdir -i 1 d1, M1 drop reply & fail M0/M1, c2 mkdir d1/dir ========================================================== 13:14:38 (1750180478) SKIP: replay-dual test_22b needs >= 2 MDTs SKIP 22b (2s) == replay-dual test 22c: c1 lfs mkdir -i 1 d1, M1 drop update & fail M1, c2 mkdir d1/dir ========================================================== 13:14:40 (1750180480) SKIP: replay-dual test_22c needs >= 2 MDTs SKIP 22c (1s) == replay-dual test 22d: c1 lfs mkdir -i 1 d1, M1 drop update & fail M0/M1,c2 mkdir d1/dir ========================================================== 13:14:41 (1750180481) SKIP: replay-dual test_22d needs >= 2 MDTs SKIP 22d (2s) == replay-dual test 23a: c1 rmdir d1, M1 drop reply and fail, client2 mkdir d1 ========================================================== 13:14:43 (1750180483) SKIP: replay-dual test_23a needs >= 2 MDTs SKIP 23a (1s) == replay-dual test 23b: c1 rmdir d1, M1 drop reply and fail M0/M1, c2 mkdir d1 ========================================================== 13:14:44 (1750180484) SKIP: replay-dual test_23b needs >= 2 MDTs SKIP 23b (2s) == replay-dual test 23c: c1 rmdir d1, M0 drop update reply and fail M0, c2 mkdir d1 ========================================================== 13:14:46 (1750180486) SKIP: replay-dual test_23c needs >= 2 MDTs SKIP 23c (1s) == replay-dual test 23d: c1 rmdir d1, M0 drop update reply and fail M0/M1, c2 mkdir d1 ========================================================== 13:14:47 (1750180487) SKIP: replay-dual test_23d needs >= 2 MDTs SKIP 23d (2s) == replay-dual test 24: reconstruct on non-existing object ========================================================== 13:14:49 (1750180489) fail_loc=0x119 fail_loc=0 truncate: cannot truncate '/mnt/lustre/f24.replay-dual' to length 100: No such file or directory PASS 24 (88s) == replay-dual test 25: replay|resend ==================== 13:16:17 (1750180577) 1+0 records in 1+0 records out 512 bytes (512 B) copied, 0.00678538 s, 75.5 kB/s fail_loc=0x304 fail_loc=0x80000325 Failing ost1 on oleg203-server Stopping /mnt/lustre-ost1 (opts:) on oleg203-server 13:16:20 (1750180580) shut down facet: ost1 facet_host: oleg203-server facet_failover_host: oleg203-server Failover ost1 to oleg203-server mount facets: ost1 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg203-server: oleg203-server.virtnet: executing set_default_debug -1 all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Started lustre-OST0000 13:16:35 (1750180595) targets are mounted 13:16:35 (1750180595) facet_failover done oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 6834: 14115 Terminated LUSTRE="/home/green/git/lustre-release/lustre" bash -c "multiop /mnt/lustre2/f25.replay-dual Ow512" fail_loc=0 PASS 25 (25s) == replay-dual test 26: dbench and tar with mds failover ========================================================== 13:16:42 (1750180602) Starting client oleg203-client.virtnet: -o user_xattr,flock 192.168.202.103@tcp:/lustre /mnt/lustre Started clients oleg203-client.virtnet: 192.168.202.103@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) Started tar loop with pid 15642 Started dbench loop with 15643 looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Tue Jun 17 13:16:43 EDT 2025 waiting for dbench pid 15674 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs failed to create barrier semaphore 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3724288 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3737600 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7461888 1% /mnt/lustre 1 155 6.54 MB/sec warmup 1 sec latency 66.087 ms 1 293 5.38 MB/sec warmup 2 sec latency 46.034 ms 1 445 5.21 MB/sec warmup 3 sec latency 34.961 ms 1 639 5.26 MB/sec warmup 4 sec latency 41.199 ms test_26 fail mds1 1 times Failing mds1 on oleg203-server 1 721 4.26 MB/sec warmup 5 sec latency 124.195 ms Stopping /mnt/lustre-mds1 (opts:) on oleg203-server 1 759 3.56 MB/sec warmup 6 sec latency 477.396 ms 13:16:50 (1750180610) shut down facet: mds1 facet_host: oleg203-server facet_failover_host: oleg203-server 1 759 3.05 MB/sec warmup 7 sec latency 1477.562 ms 1 759 2.67 MB/sec warmup 8 sec latency 2477.782 ms 1 759 2.37 MB/sec warmup 9 sec latency 3477.992 ms 1 759 2.13 MB/sec warmup 10 sec latency 4478.209 ms 1 759 1.94 MB/sec warmup 11 sec latency 5478.461 ms 1 759 1.78 MB/sec warmup 12 sec latency 6478.691 ms 1 759 1.64 MB/sec warmup 13 sec latency 7478.893 ms 1 759 1.52 MB/sec warmup 14 sec latency 8479.111 ms 1 759 1.42 MB/sec warmup 15 sec latency 9479.337 ms 1 759 1.33 MB/sec warmup 16 sec latency 10479.515 ms Failover mds1 to oleg203-server mount facets: mds1 1 759 1.26 MB/sec warmup 17 sec latency 11479.826 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 759 1.19 MB/sec warmup 18 sec latency 12480.917 ms 1 759 1.12 MB/sec warmup 19 sec latency 13481.549 ms oleg203-server: oleg203-server.virtnet: executing set_default_debug -1 all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Started lustre-MDT0000 13:17:04 (1750180624) targets are mounted 13:17:04 (1750180624) facet_failover done 1 759 0.00 MB/sec execute 1 sec latency 15481.893 ms 1 759 0.00 MB/sec execute 2 sec latency 16482.089 ms 1 759 0.00 MB/sec execute 3 sec latency 17482.281 ms oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 884 0.08 MB/sec execute 4 sec latency 17649.311 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 996 0.11 MB/sec execute 5 sec latency 67.811 ms 1 1172 0.28 MB/sec execute 6 sec latency 41.766 ms 1 1450 0.67 MB/sec execute 7 sec latency 22.882 ms 1 1634 0.60 MB/sec execute 8 sec latency 58.124 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4352 2204160 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 36864 3597312 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 12288 3641344 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 49152 7238656 1% /mnt/lustre 1 1861 0.56 MB/sec execute 9 sec latency 26.231 ms 1 2060 0.53 MB/sec execute 10 sec latency 38.494 ms 1 2272 0.53 MB/sec execute 11 sec latency 26.632 ms test_26 fail mds1 2 times 1 2435 0.56 MB/sec execute 12 sec latency 86.700 ms Failing mds1 on oleg203-server Stopping /mnt/lustre-mds1 (opts:) on oleg203-server 1 2592 0.59 MB/sec execute 13 sec latency 95.091 ms 13:17:17 (1750180637) shut down facet: mds1 facet_host: oleg203-server facet_failover_host: oleg203-server 1 2592 0.55 MB/sec execute 14 sec latency 1095.376 ms 1 2592 0.51 MB/sec execute 15 sec latency 2095.586 ms 1 2592 0.48 MB/sec execute 16 sec latency 3095.784 ms 1 2592 0.45 MB/sec execute 17 sec latency 4096.418 ms 1 2592 0.43 MB/sec execute 18 sec latency 5096.864 ms 1 2592 0.40 MB/sec execute 19 sec latency 6097.291 ms 1 2592 0.38 MB/sec execute 20 sec latency 7097.541 ms 1 2592 0.36 MB/sec execute 21 sec latency 8097.835 ms 1 2592 0.35 MB/sec execute 22 sec latency 9098.019 ms 1 2592 0.33 MB/sec execute 23 sec latency 10098.208 ms Failover mds1 to oleg203-server mount facets: mds1 1 2592 0.32 MB/sec execute 24 sec latency 11098.717 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 2592 0.31 MB/sec execute 25 sec latency 12098.883 ms 1 2592 0.29 MB/sec execute 26 sec latency 13099.119 ms oleg203-server: oleg203-server.virtnet: executing set_default_debug -1 all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 1 2592 0.28 MB/sec execute 27 sec latency 14099.338 ms Started lustre-MDT0000 13:17:31 (1750180651) targets are mounted 13:17:31 (1750180651) facet_failover done 1 2592 0.27 MB/sec execute 28 sec latency 15100.234 ms 1 2592 0.26 MB/sec execute 29 sec latency 16100.483 ms 1 2592 0.26 MB/sec execute 30 sec latency 17100.859 ms 1 2592 0.25 MB/sec execute 31 sec latency 18101.207 ms 1 2592 0.24 MB/sec execute 32 sec latency 19101.410 ms 1 2592 0.23 MB/sec execute 33 sec latency 20101.629 ms 1 2764 0.31 MB/sec execute 34 sec latency 20580.244 ms oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 2993 0.35 MB/sec execute 35 sec latency 81.958 ms 1 3342 0.39 MB/sec execute 36 sec latency 67.910 ms 1 3626 0.47 MB/sec execute 37 sec latency 24.886 ms 1 3780 0.48 MB/sec execute 38 sec latency 37.306 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4864 2203648 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 43008 3535872 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 16384 3567616 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 59392 7103488 1% /mnt/lustre 1 3884 0.47 MB/sec execute 39 sec latency 42.339 ms 1 4006 0.47 MB/sec execute 40 sec latency 42.315 ms 1 4117 0.46 MB/sec execute 41 sec latency 36.665 ms 1 4245 0.45 MB/sec execute 42 sec latency 112.450 ms test_26 fail mds1 3 times Failing mds1 on oleg203-server 1 4330 0.44 MB/sec execute 43 sec latency 72.318 ms Stopping /mnt/lustre-mds1 (opts:) on oleg203-server 1 4370 0.44 MB/sec execute 44 sec latency 465.226 ms 13:17:48 (1750180668) shut down facet: mds1 facet_host: oleg203-server facet_failover_host: oleg203-server 1 4370 0.43 MB/sec execute 45 sec latency 1465.431 ms 1 4370 0.42 MB/sec execute 46 sec latency 2465.615 ms 1 4370 0.41 MB/sec execute 47 sec latency 3466.132 ms 1 4370 0.40 MB/sec execute 48 sec latency 4466.501 ms 1 4370 0.39 MB/sec execute 49 sec latency 5467.686 ms 1 4370 0.38 MB/sec execute 50 sec latency 6468.364 ms 1 4370 0.38 MB/sec execute 51 sec latency 7468.688 ms 1 4370 0.37 MB/sec execute 52 sec latency 8468.980 ms 1 4370 0.36 MB/sec execute 53 sec latency 9469.975 ms 1 4370 0.36 MB/sec execute 54 sec latency 10470.165 ms Failover mds1 to oleg203-server mount facets: mds1 1 4370 0.35 MB/sec execute 55 sec latency 11470.462 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 4370 0.34 MB/sec execute 56 sec latency 12471.436 ms 1 4370 0.34 MB/sec execute 57 sec latency 13471.657 ms 1 4370 0.33 MB/sec execute 58 sec latency 14471.853 ms oleg203-server: oleg203-server.virtnet: executing set_default_debug -1 all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Started lustre-MDT0000 13:18:02 (1750180682) targets are mounted 13:18:02 (1750180682) facet_failover done 1 4370 0.33 MB/sec execute 59 sec latency 15472.159 ms 1 4370 0.32 MB/sec execute 60 sec latency 16472.740 ms 1 4370 0.32 MB/sec execute 61 sec latency 17473.005 ms 1 4370 0.31 MB/sec execute 62 sec latency 18473.371 ms 1 4370 0.31 MB/sec execute 63 sec latency 19473.559 ms 1 4395 0.30 MB/sec execute 64 sec latency 20155.530 ms oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 4523 0.30 MB/sec execute 65 sec latency 74.063 ms 1 4680 0.31 MB/sec execute 66 sec latency 89.090 ms 1 4812 0.31 MB/sec execute 67 sec latency 49.815 ms 1 5011 0.35 MB/sec execute 68 sec latency 42.155 ms 1 5117 0.35 MB/sec execute 69 sec latency 105.775 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 6016 2202496 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 45056 3584000 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 20480 3555328 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 65536 7139328 1% /mnt/lustre 1 5199 0.34 MB/sec execute 70 sec latency 95.686 ms 1 5292 0.34 MB/sec execute 71 sec latency 57.760 ms 1 5385 0.33 MB/sec execute 72 sec latency 52.619 ms test_26 fail mds1 4 times 1 5478 0.33 MB/sec execute 73 sec latency 101.600 ms Failing mds1 on oleg203-server Stopping /mnt/lustre-mds1 (opts:) on oleg203-server 1 5587 0.33 MB/sec execute 74 sec latency 43.618 ms 1 5616 0.33 MB/sec execute 75 sec latency 890.896 ms 13:18:19 (1750180699) shut down facet: mds1 facet_host: oleg203-server facet_failover_host: oleg203-server 1 5616 0.32 MB/sec execute 76 sec latency 1892.824 ms 1 5616 0.32 MB/sec execute 77 sec latency 2893.784 ms 1 5616 0.31 MB/sec execute 78 sec latency 3894.076 ms 1 5616 0.31 MB/sec execute 79 sec latency 4894.505 ms 1 5616 0.30 MB/sec execute 80 sec latency 5894.726 ms 1 5616 0.30 MB/sec execute 81 sec latency 6894.900 ms 1 5616 0.30 MB/sec execute 82 sec latency 7895.308 ms 1 5616 0.29 MB/sec execute 83 sec latency 8895.573 ms 1 5616 0.29 MB/sec execute 84 sec latency 9895.764 ms 1 5616 0.29 MB/sec execute 85 sec latency 10895.944 ms Failover mds1 to oleg203-server mount facets: mds1 1 5616 0.28 MB/sec execute 86 sec latency 11896.061 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 5616 0.28 MB/sec execute 87 sec latency 12897.060 ms 1 5616 0.28 MB/sec execute 88 sec latency 13897.287 ms oleg203-server: oleg203-server.virtnet: executing set_default_debug -1 all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Started lustre-MDT0000 13:18:32 (1750180712) targets are mounted 13:18:32 (1750180712) facet_failover done 1 5616 0.27 MB/sec execute 89 sec latency 14897.548 ms 1 5616 0.27 MB/sec execute 90 sec latency 15897.832 ms 1 5616 0.27 MB/sec execute 91 sec latency 16898.129 ms 1 5616 0.27 MB/sec execute 92 sec latency 17898.575 ms 1 5616 0.26 MB/sec execute 93 sec latency 18898.861 ms oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 5718 0.26 MB/sec execute 94 sec latency 19157.495 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 5820 0.26 MB/sec execute 95 sec latency 48.240 ms 1 5928 0.27 MB/sec execute 96 sec latency 119.731 ms 1 6001 0.27 MB/sec execute 97 sec latency 131.002 ms 1 6129 0.27 MB/sec execute 98 sec latency 111.428 ms 1 6324 0.30 MB/sec execute 99 sec latency 99.968 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 6912 2201600 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 46080 3596288 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 21504 3609600 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 67584 7205888 1% /mnt/lustre 1 cleanup 100 sec 1 cleanup 101 sec 0 cleanup 101 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 971 36.764 17649.301 Close 751 3.816 16.517 Rename 43 27.080 56.504 Unlink 165 8.393 19.427 Qpathinfo 924 48.219 20580.218 Qfileinfo 171 0.638 6.178 Qfsinfo 144 1.112 16.996 Sfileinfo 94 232.127 20155.519 Find 333 1.674 18.989 WriteX 513 3.320 22.554 ReadX 1539 0.357 39.402 LockX 4 4.717 8.901 UnlockX 4 4.457 6.686 Flush 67 56.911 168.601 Throughput 0.300252 MB/sec 1 clients 1 procs max_latency=20580.244 ms stopping dbench on /mnt/lustre at Tue Jun 17 13:18:45 EDT 2025 with return code 0 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt test_26 fail mds1 5 times Failing mds1 on oleg203-server '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Tue Jun 17 13:18:47 EDT 2025 waiting for dbench pid 20127 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients Stopping /mnt/lustre-mds1 (opts:) on oleg203-server 1 90 4.19 MB/sec warmup 1 sec latency 304.268 ms 13:18:49 (1750180729) shut down facet: mds1 facet_host: oleg203-server facet_failover_host: oleg203-server 1 90 2.10 MB/sec warmup 2 sec latency 1304.491 ms 1 90 1.40 MB/sec warmup 3 sec latency 2304.679 ms 1 90 1.05 MB/sec warmup 4 sec latency 3304.894 ms 1 90 0.84 MB/sec warmup 5 sec latency 4305.077 ms 1 90 0.70 MB/sec warmup 6 sec latency 5306.706 ms 1 90 0.60 MB/sec warmup 7 sec latency 6307.371 ms 1 90 0.52 MB/sec warmup 8 sec latency 7307.586 ms 1 90 0.47 MB/sec warmup 9 sec latency 8308.074 ms 1 90 0.42 MB/sec warmup 10 sec latency 9309.421 ms 1 90 0.38 MB/sec warmup 11 sec latency 10309.625 ms 1 90 0.35 MB/sec warmup 12 sec latency 11309.840 ms Failover mds1 to oleg203-server mount facets: mds1 1 90 0.32 MB/sec warmup 13 sec latency 12310.247 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 90 0.30 MB/sec warmup 14 sec latency 13310.458 ms oleg203-server: oleg203-server.virtnet: executing set_default_debug -1 all 1 90 0.28 MB/sec warmup 15 sec latency 14310.762 ms pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Started lustre-MDT0000 13:19:03 (1750180743) targets are mounted 13:19:03 (1750180743) facet_failover done 1 90 0.26 MB/sec warmup 16 sec latency 15311.188 ms 1 90 0.25 MB/sec warmup 17 sec latency 16311.727 ms 1 90 0.23 MB/sec warmup 18 sec latency 17312.050 ms 1 90 0.22 MB/sec warmup 19 sec latency 18312.170 ms oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 289 3.64 MB/sec execute 1 sec latency 18606.045 ms 1 397 3.51 MB/sec execute 2 sec latency 47.750 ms 1 513 3.55 MB/sec execute 3 sec latency 47.749 ms 1 669 3.52 MB/sec execute 4 sec latency 108.998 ms 1 736 2.85 MB/sec execute 5 sec latency 102.356 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 7680 2200832 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 13312 3590144 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 40960 3569664 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 54272 7159808 1% /mnt/lustre 1 815 2.40 MB/sec execute 6 sec latency 75.187 ms 1 916 2.10 MB/sec execute 7 sec latency 99.049 ms 1 990 1.86 MB/sec execute 8 sec latency 99.916 ms test_26 fail mds1 6 times 1 1145 1.77 MB/sec execute 9 sec latency 46.667 ms Failing mds1 on oleg203-server Stopping /mnt/lustre-mds1 (opts:) on oleg203-server 1 1245 1.60 MB/sec execute 10 sec latency 247.592 ms 13:19:18 (1750180758) shut down facet: mds1 facet_host: oleg203-server facet_failover_host: oleg203-server 1 1245 1.46 MB/sec execute 11 sec latency 1247.963 ms 1 1245 1.33 MB/sec execute 12 sec latency 2248.574 ms 1 1245 1.23 MB/sec execute 13 sec latency 3248.823 ms 1 1245 1.14 MB/sec execute 14 sec latency 4249.042 ms 1 1245 1.07 MB/sec execute 15 sec latency 5249.275 ms 1 1245 1.00 MB/sec execute 16 sec latency 6249.495 ms 1 1245 0.94 MB/sec execute 17 sec latency 7249.744 ms 1 1245 0.89 MB/sec execute 18 sec latency 8250.106 ms 1 1245 0.84 MB/sec execute 19 sec latency 9251.727 ms 1 1245 0.80 MB/sec execute 20 sec latency 10251.928 ms Failover mds1 to oleg203-server mount facets: mds1 1 1245 0.76 MB/sec execute 21 sec latency 11252.145 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 1245 0.73 MB/sec execute 22 sec latency 12252.336 ms 1 1245 0.70 MB/sec execute 23 sec latency 13252.582 ms oleg203-server: oleg203-server.virtnet: executing set_default_debug -1 all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 1 1245 0.67 MB/sec execute 24 sec latency 14252.946 ms Started lustre-MDT0000 13:19:32 (1750180772) targets are mounted 13:19:32 (1750180772) facet_failover done 1 1245 0.64 MB/sec execute 25 sec latency 15253.157 ms 1 1245 0.62 MB/sec execute 26 sec latency 16253.347 ms 1 1245 0.59 MB/sec execute 27 sec latency 17253.624 ms 1 1245 0.57 MB/sec execute 28 sec latency 18253.828 ms 1 1245 0.55 MB/sec execute 29 sec latency 19254.482 ms 1 1277 0.54 MB/sec execute 30 sec latency 19922.432 ms oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 1492 0.61 MB/sec execute 31 sec latency 137.802 ms 1 1643 0.60 MB/sec execute 32 sec latency 84.589 ms 1 1834 0.58 MB/sec execute 33 sec latency 35.957 ms 1 1982 0.57 MB/sec execute 34 sec latency 85.236 ms 1 2150 0.56 MB/sec execute 35 sec latency 40.499 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 8448 2200064 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 16384 3562496 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 46080 3559424 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 62464 7121920 1% /mnt/lustre 1 2287 0.57 MB/sec execute 36 sec latency 44.058 ms 1 2389 0.56 MB/sec execute 37 sec latency 96.403 ms 1 2499 0.57 MB/sec execute 38 sec latency 85.132 ms test_26 fail mds1 7 times 1 2878 0.64 MB/sec execute 39 sec latency 65.801 ms Failing mds1 on oleg203-server Stopping /mnt/lustre-mds1 (opts:) on oleg203-server 1 3013 0.66 MB/sec execute 40 sec latency 131.856 ms 1 3013 0.65 MB/sec execute 41 sec latency 1132.328 ms 1 3013 0.63 MB/sec execute 42 sec latency 2132.525 ms 1 3013 0.62 MB/sec execute 43 sec latency 3133.309 ms 1 3013 0.60 MB/sec execute 44 sec latency 4133.494 ms 1 3013 0.59 MB/sec execute 45 sec latency 5136.302 ms 1 3013 0.58 MB/sec execute 46 sec latency 6137.490 ms 13:19:54 (1750180794) shut down facet: mds1 facet_host: oleg203-server facet_failover_host: oleg203-server 1 3013 0.56 MB/sec execute 47 sec latency 7137.724 ms 1 3013 0.55 MB/sec execute 48 sec latency 8137.890 ms 1 3013 0.54 MB/sec execute 49 sec latency 9139.249 ms 1 3013 0.53 MB/sec execute 50 sec latency 10139.887 ms 1 3013 0.52 MB/sec execute 51 sec latency 11140.160 ms 1 3013 0.51 MB/sec execute 52 sec latency 12143.219 ms 1 3013 0.50 MB/sec execute 53 sec latency 13143.466 ms 1 3013 0.49 MB/sec execute 54 sec latency 14143.689 ms 1 3013 0.48 MB/sec execute 55 sec latency 15143.882 ms 1 3013 0.47 MB/sec execute 56 sec latency 16144.082 ms Failover mds1 to oleg203-server mount facets: mds1 1 3013 0.46 MB/sec execute 57 sec latency 17144.434 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 3013 0.46 MB/sec execute 58 sec latency 18144.613 ms 1 3013 0.45 MB/sec execute 59 sec latency 19144.779 ms oleg203-server: oleg203-server.virtnet: executing set_default_debug -1 all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 1 3013 0.44 MB/sec execute 60 sec latency 20145.324 ms Started lustre-MDT0000 13:20:08 (1750180808) targets are mounted 13:20:08 (1750180808) facet_failover done 1 3013 0.43 MB/sec execute 61 sec latency 21145.588 ms 1 3013 0.43 MB/sec execute 62 sec latency 22145.772 ms 1 3013 0.42 MB/sec execute 63 sec latency 23145.955 ms 1 3013 0.41 MB/sec execute 64 sec latency 24146.187 ms 1 3013 0.41 MB/sec execute 65 sec latency 25146.375 ms 1 3013 0.40 MB/sec execute 66 sec latency 26147.437 ms 1 3013 0.39 MB/sec execute 67 sec latency 27147.760 ms 1 3013 0.39 MB/sec execute 68 sec latency 28147.912 ms 1 3013 0.38 MB/sec execute 69 sec latency 29148.142 ms 1 3024 0.38 MB/sec execute 70 sec latency 30003.324 ms oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 3326 0.40 MB/sec execute 71 sec latency 118.983 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 3537 0.43 MB/sec execute 72 sec latency 47.462 ms 1 3689 0.43 MB/sec execute 73 sec latency 31.456 ms 1 3820 0.44 MB/sec execute 74 sec latency 35.729 ms 1 3949 0.44 MB/sec execute 75 sec latency 26.348 ms 1 4054 0.43 MB/sec execute 76 sec latency 28.269 ms dbench killed by signal 15 stopping dbench on /mnt/lustre at Tue Jun 17 13:20:24 EDT 2025 with return code 0 20127 pts/0 S+ 0:00 dbench -c client.txt 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100 killed dbench main pid 20127 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished PASS 26 (267s) == replay-dual test 28: lock replay should be ordered: waiting after granted ========================================================== 13:21:09 (1750180869) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00534395 s, 766 kB/s fail_loc=0x80000324 fail_loc=0x32a Failing ost1 on oleg203-server Stopping /mnt/lustre-ost1 (opts:) on oleg203-server 13:21:33 (1750180893) shut down facet: ost1 facet_host: oleg203-server facet_failover_host: oleg203-server Failover ost1 to oleg203-server mount facets: ost1 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 1+0 records in 1+0 records out 4096 bytes (4.1 kB) copied, 0.00873649 s, 469 kB/s oleg203-server: oleg203-server.virtnet: executing set_default_debug -1 all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Started lustre-OST0000 13:21:48 (1750180908) targets are mounted 13:21:48 (1750180908) facet_failover done oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec PASS 28 (47s) == replay-dual test 29: replay vs update with the same xid ========================================================== 13:21:56 (1750180916) SKIP: replay-dual test_29 needs >= 2 MDTs SKIP 29 (1s) == replay-dual test 30: layout lock replay is not blocked on IO ========================================================== 13:21:57 (1750180917) 10+0 records in 10+0 records out 40960 bytes (41 kB) copied, 0.0159046 s, 2.6 MB/s 10+0 records in 10+0 records out 40960 bytes (41 kB) copied, 0.0194973 s, 2.1 MB/s fail_loc=0x32e fail_val=4 Failing mds1 on oleg203-server Stopping /mnt/lustre-mds1 (opts:) on oleg203-server 13:22:01 (1750180921) shut down facet: mds1 facet_host: oleg203-server facet_failover_host: oleg203-server Failover mds1 to oleg203-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg203-server: oleg203-server.virtnet: executing set_default_debug -1 all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Started lustre-MDT0000 13:22:15 (1750180935) targets are mounted 13:22:15 (1750180935) facet_failover done 160+0 records in 160+0 records out 81920 bytes (82 kB) copied, 21.8648 s, 3.7 kB/s oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 30 (29s) == replay-dual test 31: deadlock on file_remove_privs and occupied mod rpc slots ========================================================== 13:22:26 (1750180946) Failing ost1 on oleg203-server Stopping /mnt/lustre-ost1 (opts:) on oleg203-server Creating to objid 2753 on ost lustre-OST0000... 13:22:30 (1750180950) shut down facet: ost1 facet_host: oleg203-server facet_failover_host: oleg203-server total: 32 open/close in 0.50 seconds: 63.72 ops/second at_max=0 fail_loc=0x80001420 file /mnt/lustre2/d31.replay-dual/mdtdir/f31.replay-dual is ready Failover ost1 to oleg203-server mount facets: ost1 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg203-server: oleg203-server.virtnet: executing set_default_debug -1 all pdsh@oleg203-client: oleg203-server: ssh exited with exit code 1 Started lustre-OST0000 13:22:45 (1750180965) targets are mounted 13:22:45 (1750180965) facet_failover done oleg203-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE FULL state after 0 sec pids: 27608 27609 27615 27616 27617 27618 27619 27620 27621 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3712 2204800 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 496475 393 496082 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 118251 491 117760 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 118153 393 117760 1% /mnt/lustre[OST:1] filesystem_summary: 235913 393 235520 1% /mnt/lustre at_max=600 PASS 31 (27s) == replay-dual test 32: gap in update llog shouldn't break recovery ========================================================== 13:22:53 (1750180973) SKIP: replay-dual test_32 needs >= 2 MDTs SKIP 32 (2s) == replay-dual test 33: Check for OBD_INCOMPAT_MULTI_RPCS in last_rcvd after abort_recovery ========================================================== 13:22:55 (1750180975) SKIP: replay-dual test_33 ldiskfs only test SKIP 33 (1s) == replay-dual test complete, duration 2266 sec ========== 13:22:56 (1750180976) === replay-dual: start cleanup 13:22:57 (1750180977) === Stopping clients: oleg203-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg203-client.virtnet /mnt/lustre2 opts: === replay-dual: finish cleanup 13:23:00 (1750180980) ===