-----============= acceptance-small: replay-dual ============----- Sun Dec 7 05:24:26 EST 2025 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg459-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/replay-dual.*ex': No such file or directory excepting tests: 14b 21b 21b skipping tests SLOW=no: 21b === replay-dual: start setup 05:24:45 (1765103085) === Starting client oleg459-client.virtnet: -o user_xattr,flock 192.168.204.159@tcp:/lustre /mnt/lustre2 Mount client oleg459-client.virtnet: mount -t lustre -o user_xattr,flock 192.168.204.159@tcp:/lustre /mnt/lustre2 Started clients oleg459-client.virtnet: 192.168.204.159@tcp:/lustre on /mnt/lustre2 type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) oleg459-client.virtnet: executing check_config_client /mnt/lustre oleg459-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg459-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff9191c556f800.idle_timeout=debug osc.lustre-OST0000-osc-ffff9191c734e800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9191c556f800.idle_timeout=debug osc.lustre-OST0001-osc-ffff9191c734e800.idle_timeout=debug disable quota as required oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all === replay-dual: finish setup 05:25:08 (1765103108) === == replay-dual test 0a: expired recovery with lost client ========================================================== 05:25:09 (1765103109) Check file is LU482_FAILED=/tmp/replay-dual.lu482.pxINzE UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre total: 50 open/close in 1.61 seconds: 31.13 ops/second fail_loc=0x80000514 Failing mds1 on oleg459-server Stopping /mnt/lustre-mds1 (opts:) on oleg459-server 05:25:20 (1765103120) shut down facet: mds1 facet_host: oleg459-server facet_failover_host: oleg459-server Failover mds1 to oleg459-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 Started lustre-MDT0000 05:25:43 (1765103143) targets are mounted 05:25:43 (1765103143) facet_failover done Starting client: oleg459-client.virtnet: -o user_xattr,flock 192.168.204.159@tcp:/lustre /mnt/lustre2 - unlinked 0 (time 1765103253 ; total 0 ; last 0) total: 50 unlinks in 2 seconds: 25.000000 unlinks/second PASS 0a (151s) == replay-dual test 0b: lost client during waiting for next transno ========================================================== 05:27:41 (1765103261) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg459-server Stopping /mnt/lustre-mds1 (opts:) on oleg459-server 05:27:50 (1765103270) shut down facet: mds1 facet_host: oleg459-server facet_failover_host: oleg459-server Failover mds1 to oleg459-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 Started lustre-MDT0000 05:28:12 (1765103292) targets are mounted 05:28:12 (1765103292) facet_failover done Starting client: oleg459-client.virtnet: -o user_xattr,flock 192.168.204.159@tcp:/lustre /mnt/lustre Starting client: oleg459-client.virtnet: -o user_xattr,flock 192.168.204.159@tcp:/lustre /mnt/lustre2 PASS 0b (205s) == replay-dual test 1: |X| simple create ================= 05:31:06 (1765103466) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg459-server Stopping /mnt/lustre-mds1 (opts:) on oleg459-server 05:31:14 (1765103474) shut down facet: mds1 facet_host: oleg459-server facet_failover_host: oleg459-server Failover mds1 to oleg459-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 Started lustre-MDT0000 05:31:32 (1765103492) targets are mounted 05:31:32 (1765103492) facet_failover done oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 1 (36s) == replay-dual test 2: |X| mkdir adir ==================== 05:31:42 (1765103502) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg459-server Stopping /mnt/lustre-mds1 (opts:) on oleg459-server 05:31:49 (1765103509) shut down facet: mds1 facet_host: oleg459-server facet_failover_host: oleg459-server Failover mds1 to oleg459-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 Started lustre-MDT0000 05:32:08 (1765103528) targets are mounted 05:32:08 (1765103528) facet_failover done oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 2 (36s) == replay-dual test 3: |X| mkdir adir, mkdir adir/bdir === 05:32:18 (1765103538) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg459-server Stopping /mnt/lustre-mds1 (opts:) on oleg459-server 05:32:26 (1765103546) shut down facet: mds1 facet_host: oleg459-server facet_failover_host: oleg459-server Failover mds1 to oleg459-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 Started lustre-MDT0000 05:32:45 (1765103565) targets are mounted 05:32:45 (1765103565) facet_failover done oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 3 (37s) == replay-dual test 4: |X| mkdir adir (-EEXIST), mkdir adir/bdir ========================================================== 05:32:55 (1765103575) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre mkdir: cannot create directory '/mnt/lustre/adir': File exists Failing mds1 on oleg459-server Stopping /mnt/lustre-mds1 (opts:) on oleg459-server 05:33:02 (1765103582) shut down facet: mds1 facet_host: oleg459-server facet_failover_host: oleg459-server Failover mds1 to oleg459-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 Started lustre-MDT0000 05:33:23 (1765103603) targets are mounted 05:33:23 (1765103603) facet_failover done oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 4 (39s) == replay-dual test 5: open, unlink |X| close ============ 05:33:34 (1765103614) multiop /mnt/lustre2/a vo_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7559 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg459-server Stopping /mnt/lustre-mds1 (opts:) on oleg459-server 05:33:42 (1765103622) shut down facet: mds1 facet_host: oleg459-server facet_failover_host: oleg459-server Failover mds1 to oleg459-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 Started lustre-MDT0000 05:33:59 (1765103639) targets are mounted 05:33:59 (1765103639) facet_failover done oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 5 (35s) == replay-dual test 6: open1, open2, unlink |X| close1 [fail mds1] close2 ========================================================== 05:34:09 (1765103649) multiop /mnt/lustre2/a vo_c TMPPIPE=/tmp/multiop_open_wait_pipe.7559 multiop /mnt/lustre/a vo_c TMPPIPE=/tmp/multiop_open_wait_pipe.7559 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg459-server Stopping /mnt/lustre-mds1 (opts:) on oleg459-server 05:34:16 (1765103656) shut down facet: mds1 facet_host: oleg459-server facet_failover_host: oleg459-server Failover mds1 to oleg459-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 Started lustre-MDT0000 05:34:34 (1765103674) targets are mounted 05:34:34 (1765103674) facet_failover done oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 6 (35s) == replay-dual test 8: replay of resent request ========== 05:34:44 (1765103684) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3770368 3072 3765248 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7541760 6144 7531520 1% /mnt/lustre fail_loc=0x119 fail_loc=0 Failing mds1 on oleg459-server Stopping /mnt/lustre-mds1 (opts:) on oleg459-server 05:35:07 (1765103707) shut down facet: mds1 facet_host: oleg459-server facet_failover_host: oleg459-server Failover mds1 to oleg459-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 Started lustre-MDT0000 05:35:25 (1765103725) targets are mounted 05:35:25 (1765103725) facet_failover done oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 8 (49s) == replay-dual test 9: resending a replayed create ======= 05:35:33 (1765103733) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre fail_loc=0x80000119 Failing mds1 on oleg459-server Stopping /mnt/lustre-mds1 (opts:) on oleg459-server 05:35:39 (1765103739) shut down facet: mds1 facet_host: oleg459-server facet_failover_host: oleg459-server Failover mds1 to oleg459-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 Started lustre-MDT0000 05:35:56 (1765103756) targets are mounted 05:35:56 (1765103756) facet_failover done oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec fail_loc=0 PASS 9 (47s) == replay-dual test 10: resending a replayed unlink ====== 05:36:20 (1765103780) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre fail_loc=0x80000119 Failing mds1 on oleg459-server Stopping /mnt/lustre-mds1 (opts:) on oleg459-server 05:36:26 (1765103786) shut down facet: mds1 facet_host: oleg459-server facet_failover_host: oleg459-server Failover mds1 to oleg459-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 Started lustre-MDT0000 05:36:45 (1765103805) targets are mounted 05:36:45 (1765103805) facet_failover done oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec fail_loc=0 PASS 10 (48s) == replay-dual test 11: both clients timeout during replay ========================================================== 05:37:08 (1765103828) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre fail_loc=0x0119 Failing mds1 on oleg459-server Stopping /mnt/lustre-mds1 (opts:) on oleg459-server 05:37:17 (1765103837) shut down facet: mds1 facet_host: oleg459-server facet_failover_host: oleg459-server Failover mds1 to oleg459-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 Started lustre-MDT0000 05:37:34 (1765103854) targets are mounted 05:37:34 (1765103854) facet_failover done oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 14 sec fail_loc=0 PASS 11 (47s) == replay-dual test 12: open resend timeout ============== 05:37:55 (1765103875) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre multiop /mnt/lustre/f12.replay-dual vmo_c TMPPIPE=/tmp/multiop_open_wait_pipe.7559 fail_loc=0x80000302 Failing mds1 on oleg459-server Stopping /mnt/lustre-mds1 (opts:) on oleg459-server 05:38:00 (1765103880) shut down facet: mds1 facet_host: oleg459-server facet_failover_host: oleg459-server Failover mds1 to oleg459-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 Started lustre-MDT0000 05:38:18 (1765103898) targets are mounted 05:38:18 (1765103898) facet_failover done fail_loc=0 /mnt/lustre/f12.replay-dual /mnt/lustre/f12.replay-dual has type file OK PASS 12 (40s) == replay-dual test 13: close resend timeout ============= 05:38:36 (1765103916) multiop /mnt/lustre/f13.replay-dual vmo_c TMPPIPE=/tmp/multiop_open_wait_pipe.7559 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre fail_loc=0x80000115 Failing mds1 on oleg459-server Stopping /mnt/lustre-mds1 (opts:) on oleg459-server 05:38:43 (1765103923) shut down facet: mds1 facet_host: oleg459-server facet_failover_host: oleg459-server Failover mds1 to oleg459-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 Started lustre-MDT0000 05:38:59 (1765103939) targets are mounted 05:38:59 (1765103939) facet_failover done fail_loc=0 /mnt/lustre/f13.replay-dual /mnt/lustre/f13.replay-dual has type file OK PASS 13 (43s) SKIP: replay-dual test_14b skipping ALWAYS excluded test 14b == replay-dual test 15a: timeout waiting for lost client during replay, 1 client completes ========================================================== 05:39:20 (1765103960) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre total: 25 open/close in 0.55 seconds: 45.34 ops/second total: 1 open/close in 0.02 seconds: 53.98 ops/second Failing mds1 on oleg459-server Stopping /mnt/lustre-mds1 (opts:) on oleg459-server 05:39:26 (1765103966) shut down facet: mds1 facet_host: oleg459-server facet_failover_host: oleg459-server Failover mds1 to oleg459-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 Started lustre-MDT0000 05:39:43 (1765103983) targets are mounted 05:39:43 (1765103983) facet_failover done oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec - unlinked 0 (time 1765104056 ; total 0 ; last 0) total: 25 unlinks in 1 seconds: 25.000000 unlinks/second Starting client: oleg459-client.virtnet: -o user_xattr,flock 192.168.204.159@tcp:/lustre /mnt/lustre2 PASS 15a (99s) == replay-dual test 15c: remove multiple OST orphans ===== 05:40:59 (1765104059) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg459-server Stopping /mnt/lustre-mds1 (opts:) on oleg459-server 05:42:18 (1765104138) shut down facet: mds1 facet_host: oleg459-server facet_failover_host: oleg459-server Failover mds1 to oleg459-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 Started lustre-MDT0000 05:42:34 (1765104154) targets are mounted 05:42:34 (1765104154) facet_failover done oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Starting client: oleg459-client.virtnet: -o user_xattr,flock 192.168.204.159@tcp:/lustre /mnt/lustre2 PASS 15c (173s) == replay-dual test 16: fail MDS during recovery (3571) == 05:43:52 (1765104232) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3200 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre total: 25 open/close in 0.63 seconds: 39.61 ops/second total: 1 open/close in 0.02 seconds: 44.78 ops/second Failing mds1 on oleg459-server Stopping /mnt/lustre-mds1 (opts:) on oleg459-server 05:43:58 (1765104238) shut down facet: mds1 facet_host: oleg459-server facet_failover_host: oleg459-server Failover mds1 to oleg459-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 Started lustre-MDT0000 05:44:16 (1765104256) targets are mounted 05:44:16 (1765104256) facet_failover done Failing mds1 on oleg459-server Stopping /mnt/lustre-mds1 (opts:) on oleg459-server 05:44:39 (1765104279) shut down facet: mds1 facet_host: oleg459-server facet_failover_host: oleg459-server Failover mds1 to oleg459-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 Started lustre-MDT0000 05:44:56 (1765104296) targets are mounted 05:44:56 (1765104296) facet_failover done oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec - unlinked 0 (time 1765104372 ; total 0 ; last 0) total: 25 unlinks in 1 seconds: 25.000000 unlinks/second Starting client: oleg459-client.virtnet: -o user_xattr,flock 192.168.204.159@tcp:/lustre /mnt/lustre2 PASS 16 (143s) == replay-dual test 17: fail OST during recovery (3571) == 05:46:15 (1765104375) total: 25 open/close in 0.77 seconds: 32.56 ops/second total: 1 open/close in 0.04 seconds: 26.09 ops/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing ost1 on oleg459-server Stopping /mnt/lustre-ost1 (opts:) on oleg459-server 05:46:23 (1765104383) shut down facet: ost1 facet_host: oleg459-server facet_failover_host: oleg459-server Failover ost1 to oleg459-server mount facets: ost1 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 Started lustre-OST0000 05:46:40 (1765104400) targets are mounted 05:46:40 (1765104400) facet_failover done Failing ost1 on oleg459-server Stopping /mnt/lustre-ost1 (opts:) on oleg459-server 05:47:03 (1765104423) shut down facet: ost1 facet_host: oleg459-server facet_failover_host: oleg459-server Failover ost1 to oleg459-server mount facets: ost1 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 Started lustre-OST0000 05:47:20 (1765104440) targets are mounted 05:47:20 (1765104440) facet_failover done oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec - unlinked 0 (time 1765104512 ; total 0 ; last 0) total: 25 unlinks in 0 seconds: inf unlinks/second Starting client: oleg459-client.virtnet: -o user_xattr,flock 192.168.204.159@tcp:/lustre /mnt/lustre2 PASS 17 (140s) == replay-dual test 18: ldlm_handle_enqueue succeeds on evicted export (3822) ========================================================== 05:48:35 (1765104515) debug=+dlmtrace using seed 3786630274 running for 500 iterations total: 500 stats in 0 seconds: inf stats/second fail_loc=0x8000030b ldlm.namespaces.MGC192.168.204.159@tcp.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0000-mdc-ffff9191c86b5000.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0000-mdc-ffff9191d8250000.early_lock_cancel=0 ldlm.namespaces.lustre-OST0000-osc-ffff9191c86b5000.early_lock_cancel=0 ldlm.namespaces.lustre-OST0000-osc-ffff9191d8250000.early_lock_cancel=0 ldlm.namespaces.lustre-OST0001-osc-ffff9191c86b5000.early_lock_cancel=0 ldlm.namespaces.lustre-OST0001-osc-ffff9191d8250000.early_lock_cancel=0 fail_loc=0x80000305 Error in opening file "/mnt/lustre2/d18.replay-dual/f18.replay-dual"(flags=O_RDONLY) 2: No such file or directory ldlm.namespaces.MGC192.168.204.159@tcp.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0000-mdc-ffff9191c86b5000.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0000-mdc-ffff9191d8250000.early_lock_cancel=1 ldlm.namespaces.lustre-OST0000-osc-ffff9191c86b5000.early_lock_cancel=1 ldlm.namespaces.lustre-OST0000-osc-ffff9191d8250000.early_lock_cancel=1 ldlm.namespaces.lustre-OST0001-osc-ffff9191c86b5000.early_lock_cancel=1 ldlm.namespaces.lustre-OST0001-osc-ffff9191d8250000.early_lock_cancel=1 fail_loc=0 fail_loc=0 PASS 18 (49s) == replay-dual test 19: resend of open request =========== 05:49:24 (1765104564) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre fail_loc=0x157 - open/close 0 (time 1765104655.12 total 87.27 last 0.00) total: 1 open/close in 87.27 seconds: 0.01 ops/second fail_loc=0 Failing mds1 on oleg459-server Stopping /mnt/lustre-mds1 (opts:) on oleg459-server 05:50:59 (1765104659) shut down facet: mds1 facet_host: oleg459-server facet_failover_host: oleg459-server Failover mds1 to oleg459-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 Started lustre-MDT0000 05:51:16 (1765104676) targets are mounted 05:51:16 (1765104676) facet_failover done oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 19 (122s) == replay-dual test 20: recovery time is not increasing == 05:51:26 (1765104686) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg459-server Stopping /mnt/lustre-mds1 (opts:) on oleg459-server 05:51:32 (1765104692) shut down facet: mds1 facet_host: oleg459-server facet_failover_host: oleg459-server Failover mds1 to oleg459-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 Started lustre-MDT0000 05:51:50 (1765104710) targets are mounted 05:51:50 (1765104710) facet_failover done oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Starting client: oleg459-client.virtnet: -o user_xattr,flock 192.168.204.159@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre Failing mds1 on oleg459-server Stopping /mnt/lustre-mds1 (opts:) on oleg459-server 05:54:20 (1765104860) shut down facet: mds1 facet_host: oleg459-server facet_failover_host: oleg459-server Failover mds1 to oleg459-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 Started lustre-MDT0000 05:54:37 (1765104877) targets are mounted 05:54:37 (1765104877) facet_failover done oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Starting client: oleg459-client.virtnet: -o user_xattr,flock 192.168.204.159@tcp:/lustre /mnt/lustre2 PASS 20 (337s) == replay-dual test 21a: commit on sharing =============== 05:57:03 (1765105023) mdt.lustre-MDT0000.commit_on_sharing=1 Replay barrier on lustre-MDT0000 Failing mds1 on oleg459-server Stopping /mnt/lustre-mds1 (opts:) on oleg459-server 05:57:09 (1765105029) shut down facet: mds1 facet_host: oleg459-server facet_failover_host: oleg459-server Failover mds1 to oleg459-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 Started lustre-MDT0000 05:57:26 (1765105046) targets are mounted 05:57:26 (1765105046) facet_failover done Starting client: oleg459-client.virtnet: -o user_xattr,flock 192.168.204.159@tcp:/lustre /mnt/lustre2 mdt.lustre-MDT0000.commit_on_sharing=0 PASS 21a (167s) SKIP: replay-dual test_21b skipping SLOW test 21b == replay-dual test 22a: c1 lfs mkdir -i 1 dir1, M1 drop reply & fail, c2 mkdir dir1/dir ========================================================== 05:59:51 (1765105191) SKIP: replay-dual test_22a needs >= 2 MDTs SKIP 22a (1s) == replay-dual test 22b: c1 lfs mkdir -i 1 d1, M1 drop reply & fail M0/M1, c2 mkdir d1/dir ========================================================== 05:59:52 (1765105192) SKIP: replay-dual test_22b needs >= 2 MDTs SKIP 22b (2s) == replay-dual test 22c: c1 lfs mkdir -i 1 d1, M1 drop update & fail M1, c2 mkdir d1/dir ========================================================== 05:59:54 (1765105194) SKIP: replay-dual test_22c needs >= 2 MDTs SKIP 22c (1s) == replay-dual test 22d: c1 lfs mkdir -i 1 d1, M1 drop update & fail M0/M1,c2 mkdir d1/dir ========================================================== 05:59:56 (1765105196) SKIP: replay-dual test_22d needs >= 2 MDTs SKIP 22d (2s) == replay-dual test 23a: c1 rmdir d1, M1 drop reply and fail, client2 mkdir d1 ========================================================== 05:59:57 (1765105197) SKIP: replay-dual test_23a needs >= 2 MDTs SKIP 23a (2s) == replay-dual test 23b: c1 rmdir d1, M1 drop reply and fail M0/M1, c2 mkdir d1 ========================================================== 05:59:59 (1765105199) SKIP: replay-dual test_23b needs >= 2 MDTs SKIP 23b (1s) == replay-dual test 23c: c1 rmdir d1, M0 drop update reply and fail M0, c2 mkdir d1 ========================================================== 06:00:00 (1765105200) SKIP: replay-dual test_23c needs >= 2 MDTs SKIP 23c (2s) == replay-dual test 23d: c1 rmdir d1, M0 drop update reply and fail M0/M1, c2 mkdir d1 ========================================================== 06:00:02 (1765105202) SKIP: replay-dual test_23d needs >= 2 MDTs SKIP 23d (2s) == replay-dual test 24: reconstruct on non-existing object ========================================================== 06:00:04 (1765105204) fail_loc=0x119 fail_loc=0 truncate: cannot truncate '/mnt/lustre/f24.replay-dual' to length 100: No such file or directory PASS 24 (89s) == replay-dual test 25: replay|resend ==================== 06:01:33 (1765105293) 1+0 records in 1+0 records out 512 bytes copied, 0.00610387 s, 83.9 kB/s fail_loc=0x304 fail_loc=0x80000325 Failing ost1 on oleg459-server Stopping /mnt/lustre-ost1 (opts:) on oleg459-server 06:01:38 (1765105298) shut down facet: ost1 facet_host: oleg459-server facet_failover_host: oleg459-server Failover ost1 to oleg459-server mount facets: ost1 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 Started lustre-OST0000 06:01:55 (1765105315) targets are mounted 06:01:55 (1765105315) facet_failover done oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 6803: 43181 Terminated LUSTRE="/home/green/git/lustre-release/lustre" bash -c "multiop /mnt/lustre2/f25.replay-dual Ow512" fail_loc=0 PASS 25 (30s) == replay-dual test 26: dbench and tar with mds failover ========================================================== 06:02:03 (1765105323) Starting client oleg459-client.virtnet: -o user_xattr,flock 192.168.204.159@tcp:/lustre /mnt/lustre Started clients oleg459-client.virtnet: 192.168.204.159@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,encrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) Started tar loop with pid 44587 Started dbench loop with 44588 looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Sun Dec 7 06:02:06 EST 2025 waiting for dbench pid 44619 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs failed to create barrier semaphore 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3328 2205184 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 27648 3726336 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3753984 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 30720 7480320 1% /mnt/lustre 1 102 4.72 MB/sec warmup 1 sec latency 76.194 ms 1 181 3.66 MB/sec warmup 2 sec latency 66.462 ms 1 255 3.22 MB/sec warmup 3 sec latency 87.561 ms 1 338 3.06 MB/sec warmup 4 sec latency 67.624 ms test_26 fail mds1 1 times 1 421 2.97 MB/sec warmup 5 sec latency 57.427 ms Failing mds1 on oleg459-server Stopping /mnt/lustre-mds1 (opts:) on oleg459-server 1 497 2.88 MB/sec warmup 6 sec latency 61.958 ms 1 521 2.58 MB/sec warmup 7 sec latency 663.510 ms 06:02:14 (1765105334) shut down facet: mds1 facet_host: oleg459-server facet_failover_host: oleg459-server 1 521 2.26 MB/sec warmup 8 sec latency 1664.411 ms 1 521 2.01 MB/sec warmup 9 sec latency 2664.909 ms 1 521 1.80 MB/sec warmup 10 sec latency 3665.238 ms 1 521 1.64 MB/sec warmup 11 sec latency 4665.464 ms 1 521 1.50 MB/sec warmup 12 sec latency 5665.677 ms 1 521 1.39 MB/sec warmup 13 sec latency 6666.396 ms 1 521 1.29 MB/sec warmup 14 sec latency 7666.638 ms 1 521 1.20 MB/sec warmup 15 sec latency 8666.841 ms 1 521 1.13 MB/sec warmup 16 sec latency 9667.743 ms 1 521 1.06 MB/sec warmup 17 sec latency 10668.003 ms Failover mds1 to oleg459-server 1 521 1.00 MB/sec warmup 18 sec latency 11668.143 ms mount facets: mds1 1 521 0.95 MB/sec warmup 19 sec latency 12668.309 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 521 0.00 MB/sec execute 1 sec latency 14668.654 ms 1 521 0.00 MB/sec execute 2 sec latency 15668.846 ms oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all 1 521 0.00 MB/sec execute 3 sec latency 16669.102 ms pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 Started lustre-MDT0000 06:02:30 (1765105350) targets are mounted 06:02:30 (1765105350) facet_failover done 1 521 0.00 MB/sec execute 4 sec latency 17669.511 ms 1 521 0.00 MB/sec execute 5 sec latency 18670.836 ms 1 521 0.00 MB/sec execute 6 sec latency 19671.914 ms 1 521 0.00 MB/sec execute 7 sec latency 20672.137 ms 1 542 0.13 MB/sec execute 8 sec latency 21429.728 ms oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 673 0.35 MB/sec execute 9 sec latency 119.215 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 740 0.33 MB/sec execute 10 sec latency 117.837 ms 1 808 0.31 MB/sec execute 11 sec latency 93.283 ms 1 893 0.30 MB/sec execute 12 sec latency 73.636 ms 1 980 0.30 MB/sec execute 13 sec latency 68.897 ms 1 1119 0.35 MB/sec execute 14 sec latency 83.131 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3712 2204800 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 39936 3702784 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 13312 3680256 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 53248 7383040 1% /mnt/lustre 1 1215 0.33 MB/sec execute 15 sec latency 54.182 ms 1 1379 0.50 MB/sec execute 16 sec latency 53.270 ms 1 1481 0.47 MB/sec execute 17 sec latency 55.493 ms 1 1597 0.45 MB/sec execute 18 sec latency 85.453 ms test_26 fail mds1 2 times Failing mds1 on oleg459-server 1 1687 0.43 MB/sec execute 19 sec latency 75.595 ms Stopping /mnt/lustre-mds1 (opts:) on oleg459-server 1 1771 0.41 MB/sec execute 20 sec latency 211.001 ms 1 1771 0.39 MB/sec execute 21 sec latency 1211.177 ms 06:02:47 (1765105367) shut down facet: mds1 facet_host: oleg459-server facet_failover_host: oleg459-server 1 1771 0.37 MB/sec execute 22 sec latency 2211.978 ms 1 1771 0.36 MB/sec execute 23 sec latency 3212.201 ms 1 1771 0.34 MB/sec execute 24 sec latency 4212.549 ms 1 1771 0.33 MB/sec execute 25 sec latency 5212.804 ms 1 1771 0.32 MB/sec execute 26 sec latency 6214.339 ms 1 1771 0.30 MB/sec execute 27 sec latency 7214.691 ms 1 1771 0.29 MB/sec execute 28 sec latency 8215.149 ms 1 1771 0.28 MB/sec execute 29 sec latency 9215.395 ms 1 1771 0.27 MB/sec execute 30 sec latency 10215.749 ms 1 1771 0.27 MB/sec execute 31 sec latency 11216.162 ms Failover mds1 to oleg459-server mount facets: mds1 1 1771 0.26 MB/sec execute 32 sec latency 12216.462 ms 1 1771 0.25 MB/sec execute 33 sec latency 13216.913 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 1771 0.24 MB/sec execute 34 sec latency 14217.841 ms 1 1771 0.23 MB/sec execute 35 sec latency 15218.188 ms 1 1771 0.23 MB/sec execute 36 sec latency 16218.369 ms oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 1 1771 0.22 MB/sec execute 37 sec latency 17219.222 ms Started lustre-MDT0000 06:03:03 (1765105383) targets are mounted 06:03:03 (1765105383) facet_failover done 1 1785 0.22 MB/sec execute 38 sec latency 18064.172 ms 1 1877 0.21 MB/sec execute 39 sec latency 50.083 ms 1 1978 0.21 MB/sec execute 40 sec latency 130.811 ms oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 2041 0.21 MB/sec execute 41 sec latency 66.323 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 2145 0.21 MB/sec execute 42 sec latency 72.655 ms 1 2217 0.20 MB/sec execute 43 sec latency 70.739 ms 1 2346 0.22 MB/sec execute 44 sec latency 164.974 ms 1 2400 0.22 MB/sec execute 45 sec latency 117.989 ms 1 2462 0.23 MB/sec execute 46 sec latency 113.893 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4352 2204160 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 39936 3623936 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 34816 3630080 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 74752 7254016 2% /mnt/lustre 1 2566 0.23 MB/sec execute 47 sec latency 127.781 ms 1 2577 0.23 MB/sec execute 48 sec latency 816.889 ms 1 2764 0.28 MB/sec execute 49 sec latency 69.229 ms 1 2922 0.31 MB/sec execute 50 sec latency 92.928 ms test_26 fail mds1 3 times 1 3000 0.30 MB/sec execute 51 sec latency 136.445 ms Failing mds1 on oleg459-server Stopping /mnt/lustre-mds1 (opts:) on oleg459-server 1 3209 0.31 MB/sec execute 52 sec latency 114.527 ms 1 3273 0.33 MB/sec execute 53 sec latency 529.118 ms 06:03:20 (1765105400) shut down facet: mds1 facet_host: oleg459-server facet_failover_host: oleg459-server 1 3273 0.32 MB/sec execute 54 sec latency 1529.366 ms 1 3273 0.31 MB/sec execute 55 sec latency 2529.859 ms 1 3273 0.31 MB/sec execute 56 sec latency 3530.947 ms 1 3273 0.30 MB/sec execute 57 sec latency 4531.172 ms 1 3273 0.30 MB/sec execute 58 sec latency 5531.414 ms 1 3273 0.29 MB/sec execute 59 sec latency 6531.791 ms 1 3273 0.29 MB/sec execute 60 sec latency 7532.240 ms 1 3273 0.28 MB/sec execute 61 sec latency 8532.479 ms 1 3273 0.28 MB/sec execute 62 sec latency 9533.612 ms 1 3273 0.27 MB/sec execute 63 sec latency 10534.658 ms Failover mds1 to oleg459-server 1 3273 0.27 MB/sec execute 64 sec latency 11534.807 ms mount facets: mds1 1 3273 0.27 MB/sec execute 65 sec latency 12535.361 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 3273 0.26 MB/sec execute 66 sec latency 13535.761 ms 1 3273 0.26 MB/sec execute 67 sec latency 14535.960 ms 1 3273 0.25 MB/sec execute 68 sec latency 15536.833 ms oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all 1 3273 0.25 MB/sec execute 69 sec latency 16537.093 ms pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 Started lustre-MDT0000 06:03:35 (1765105415) targets are mounted 06:03:35 (1765105415) facet_failover done 1 3273 0.25 MB/sec execute 70 sec latency 17537.529 ms 1 3273 0.24 MB/sec execute 71 sec latency 18538.780 ms 1 3273 0.24 MB/sec execute 72 sec latency 19540.463 ms 1 3273 0.24 MB/sec execute 73 sec latency 20541.722 ms 1 3350 0.23 MB/sec execute 74 sec latency 20968.979 ms oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 3534 0.27 MB/sec execute 75 sec latency 47.877 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 3638 0.27 MB/sec execute 76 sec latency 43.461 ms 1 3729 0.27 MB/sec execute 77 sec latency 42.505 ms 1 3820 0.28 MB/sec execute 78 sec latency 73.240 ms 1 3905 0.28 MB/sec execute 79 sec latency 56.247 ms 1 3982 0.27 MB/sec execute 80 sec latency 53.564 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 5120 2203392 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 40960 3652608 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 37888 3602432 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 78848 7255040 2% /mnt/lustre 1 4052 0.27 MB/sec execute 81 sec latency 58.176 ms 1 4126 0.27 MB/sec execute 82 sec latency 60.293 ms 1 4226 0.27 MB/sec execute 83 sec latency 59.111 ms 1 4288 0.27 MB/sec execute 84 sec latency 131.992 ms test_26 fail mds1 4 times 1 4360 0.26 MB/sec execute 85 sec latency 108.973 ms Failing mds1 on oleg459-server Stopping /mnt/lustre-mds1 (opts:) on oleg459-server 1 4430 0.26 MB/sec execute 86 sec latency 144.022 ms 1 4452 0.26 MB/sec execute 87 sec latency 548.550 ms 06:03:54 (1765105434) shut down facet: mds1 facet_host: oleg459-server facet_failover_host: oleg459-server 1 4452 0.26 MB/sec execute 88 sec latency 1548.718 ms 1 4452 0.26 MB/sec execute 89 sec latency 2549.176 ms 1 4452 0.25 MB/sec execute 90 sec latency 3549.388 ms 1 4452 0.25 MB/sec execute 91 sec latency 4550.085 ms 1 4452 0.25 MB/sec execute 92 sec latency 5551.035 ms 1 4452 0.24 MB/sec execute 93 sec latency 6551.472 ms 1 4452 0.24 MB/sec execute 94 sec latency 7551.752 ms 1 4452 0.24 MB/sec execute 95 sec latency 8552.206 ms 1 4452 0.24 MB/sec execute 96 sec latency 9552.431 ms 1 4452 0.23 MB/sec execute 97 sec latency 10553.460 ms 1 4452 0.23 MB/sec execute 98 sec latency 11553.838 ms Failover mds1 to oleg459-server mount facets: mds1 1 4452 0.23 MB/sec execute 99 sec latency 12554.292 ms 1 cleanup 100 sec Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 cleanup 101 sec 1 cleanup 102 sec 1 cleanup 103 sec oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 1 cleanup 104 sec Started lustre-MDT0000 06:04:10 (1765105450) targets are mounted 06:04:10 (1765105450) facet_failover done 1 cleanup 105 sec 1 cleanup 106 sec 1 cleanup 107 sec 1 cleanup 108 sec 1 cleanup 109 sec 1 cleanup 110 sec 1 cleanup 111 sec oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 0 cleanup 111 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 660 121.149 23578.661 Close 491 49.813 21429.684 Rename 28 44.661 72.980 Unlink 130 13.569 44.858 Qpathinfo 598 8.777 59.070 Qfileinfo 119 1.181 10.809 Qfsinfo 117 0.969 8.248 Sfileinfo 62 31.994 82.064 Find 233 3.834 38.068 WriteX 399 5.218 31.229 ReadX 1028 0.922 47.424 LockX 2 5.282 7.575 UnlockX 2 4.743 4.951 Flush 63 64.505 164.959 Throughput 0.229474 MB/sec 1 clients 1 procs max_latency=21429.728 ms stopping dbench on /mnt/lustre at Sun Dec 7 06:04:17 EST 2025 with return code 0 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Sun Dec 7 06:04:20 EST 2025 waiting for dbench pid 48477 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 103 4.78 MB/sec warmup 1 sec latency 102.905 ms 1 181 3.66 MB/sec warmup 2 sec latency 71.526 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 5760 2202752 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 43008 3636224 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 63488 3608576 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 106496 7244800 2% /mnt/lustre 1 264 3.31 MB/sec warmup 3 sec latency 61.371 ms 1 342 3.06 MB/sec warmup 4 sec latency 72.260 ms 1 422 2.97 MB/sec warmup 5 sec latency 78.235 ms 1 513 2.95 MB/sec warmup 6 sec latency 50.072 ms test_26 fail mds1 5 times 1 664 3.02 MB/sec warmup 7 sec latency 73.253 ms Failing mds1 on oleg459-server Stopping /mnt/lustre-mds1 (opts:) on oleg459-server 1 726 2.66 MB/sec warmup 8 sec latency 158.377 ms 1 731 2.37 MB/sec warmup 9 sec latency 907.616 ms 06:04:30 (1765105470) shut down facet: mds1 facet_host: oleg459-server facet_failover_host: oleg459-server 1 731 2.13 MB/sec warmup 10 sec latency 1907.822 ms 1 731 1.94 MB/sec warmup 11 sec latency 2908.219 ms 1 731 1.77 MB/sec warmup 12 sec latency 3908.684 ms 1 731 1.64 MB/sec warmup 13 sec latency 4908.905 ms 1 731 1.52 MB/sec warmup 14 sec latency 5909.098 ms 1 731 1.42 MB/sec warmup 15 sec latency 6909.300 ms 1 731 1.33 MB/sec warmup 16 sec latency 7909.695 ms 1 731 1.25 MB/sec warmup 17 sec latency 8910.811 ms 1 731 1.18 MB/sec warmup 18 sec latency 9911.018 ms 1 731 1.12 MB/sec warmup 19 sec latency 10911.877 ms Failover mds1 to oleg459-server mount facets: mds1 1 731 0.00 MB/sec execute 1 sec latency 12914.763 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 731 0.00 MB/sec execute 2 sec latency 13915.042 ms 1 731 0.00 MB/sec execute 3 sec latency 14915.299 ms 1 731 0.00 MB/sec execute 4 sec latency 15916.456 ms oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all 1 731 0.00 MB/sec execute 5 sec latency 16916.800 ms pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 Started lustre-MDT0000 1 731 0.00 MB/sec execute 6 sec latency 17916.952 ms 06:04:46 (1765105486) targets are mounted 06:04:46 (1765105486) facet_failover done 1 731 0.00 MB/sec execute 7 sec latency 18917.845 ms 1 731 0.00 MB/sec execute 8 sec latency 19918.637 ms 1 731 0.00 MB/sec execute 9 sec latency 20919.078 ms 1 731 0.00 MB/sec execute 10 sec latency 21919.294 ms 1 731 0.00 MB/sec execute 11 sec latency 22919.702 ms 1 767 0.00 MB/sec execute 12 sec latency 23360.355 ms oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 825 0.01 MB/sec execute 13 sec latency 110.167 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 897 0.03 MB/sec execute 14 sec latency 175.028 ms 1 952 0.03 MB/sec execute 15 sec latency 129.169 ms 1 1002 0.05 MB/sec execute 16 sec latency 159.622 ms 1 1084 0.10 MB/sec execute 17 sec latency 223.059 ms 1 1192 0.10 MB/sec execute 18 sec latency 54.860 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 6656 2201856 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 14336 3724288 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 56320 3649536 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 70656 7373824 1% /mnt/lustre 1 1268 0.10 MB/sec execute 19 sec latency 65.185 ms 1 1419 0.24 MB/sec execute 20 sec latency 165.242 ms 1 1499 0.23 MB/sec execute 21 sec latency 154.968 ms 1 1585 0.22 MB/sec execute 22 sec latency 44.671 ms test_26 fail mds1 6 times 1 1660 0.21 MB/sec execute 23 sec latency 155.409 ms Failing mds1 on oleg459-server 1 1753 0.21 MB/sec execute 24 sec latency 75.218 ms Stopping /mnt/lustre-mds1 (opts:) on oleg459-server 1 1824 0.20 MB/sec execute 25 sec latency 310.678 ms 1 1824 0.19 MB/sec execute 26 sec latency 1314.827 ms 06:05:06 (1765105506) shut down facet: mds1 facet_host: oleg459-server facet_failover_host: oleg459-server 1 1824 0.19 MB/sec execute 27 sec latency 2315.051 ms 1 1824 0.18 MB/sec execute 28 sec latency 3316.901 ms 1 1824 0.17 MB/sec execute 29 sec latency 4317.105 ms 1 1824 0.17 MB/sec execute 30 sec latency 5318.753 ms 1 1824 0.16 MB/sec execute 31 sec latency 6319.679 ms 1 1824 0.16 MB/sec execute 32 sec latency 7319.987 ms 1 1824 0.15 MB/sec execute 33 sec latency 8322.360 ms 1 1824 0.15 MB/sec execute 34 sec latency 9322.616 ms 1 1824 0.14 MB/sec execute 35 sec latency 10322.835 ms 1 1824 0.14 MB/sec execute 36 sec latency 11323.088 ms Failover mds1 to oleg459-server mount facets: mds1 1 1824 0.14 MB/sec execute 37 sec latency 12323.437 ms 1 1824 0.13 MB/sec execute 38 sec latency 13323.668 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 1824 0.13 MB/sec execute 39 sec latency 14324.104 ms 1 1824 0.13 MB/sec execute 40 sec latency 15327.513 ms 1 1824 0.12 MB/sec execute 41 sec latency 16327.757 ms oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all 1 1824 0.12 MB/sec execute 42 sec latency 17327.978 ms pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 Started lustre-MDT0000 06:05:23 (1765105523) targets are mounted 06:05:23 (1765105523) facet_failover done 1 1824 0.12 MB/sec execute 43 sec latency 18328.431 ms 1 1824 0.11 MB/sec execute 44 sec latency 19331.988 ms 1 1824 0.11 MB/sec execute 45 sec latency 20332.158 ms 1 1824 0.11 MB/sec execute 46 sec latency 21332.633 ms 1 1824 0.11 MB/sec execute 47 sec latency 22333.649 ms 1 1886 0.11 MB/sec execute 48 sec latency 22362.084 ms oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 1984 0.11 MB/sec execute 49 sec latency 155.465 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 2057 0.11 MB/sec execute 50 sec latency 53.455 ms 1 2166 0.11 MB/sec execute 51 sec latency 43.675 ms 1 2334 0.13 MB/sec execute 52 sec latency 43.484 ms 1 2389 0.13 MB/sec execute 53 sec latency 172.639 ms 1 2473 0.14 MB/sec execute 54 sec latency 117.352 ms 1 2566 0.14 MB/sec execute 55 sec latency 388.886 ms 1 2609 0.14 MB/sec execute 56 sec latency 734.462 ms 1 2897 0.20 MB/sec execute 57 sec latency 173.919 ms 1 2995 0.21 MB/sec execute 58 sec latency 132.477 ms 1 3207 0.22 MB/sec execute 59 sec latency 158.024 ms 1 3361 0.24 MB/sec execute 60 sec latency 48.359 ms 1 3535 0.28 MB/sec execute 61 sec latency 56.250 ms 1 3653 0.28 MB/sec execute 62 sec latency 48.658 ms 1 3760 0.29 MB/sec execute 63 sec latency 51.471 ms 1 3840 0.29 MB/sec execute 64 sec latency 46.086 ms 1 3905 0.29 MB/sec execute 65 sec latency 110.966 ms 1 3983 0.28 MB/sec execute 66 sec latency 59.791 ms 1 4070 0.28 MB/sec execute 67 sec latency 49.348 ms 1 4159 0.28 MB/sec execute 68 sec latency 58.131 ms 1 4241 0.27 MB/sec execute 69 sec latency 152.179 ms 1 4292 0.27 MB/sec execute 70 sec latency 197.090 ms 1 4352 0.27 MB/sec execute 71 sec latency 163.459 ms 1 4417 0.27 MB/sec execute 72 sec latency 229.993 ms 1 4473 0.27 MB/sec execute 73 sec latency 123.774 ms 1 4536 0.26 MB/sec execute 74 sec latency 140.214 ms dbench killed by signal 15 stopping dbench on /mnt/lustre at Sun Dec 7 06:05:55 EST 2025 with return code 0 48477 pts/0 S+ 0:00 dbench -c client.txt 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100 killed dbench main pid 48477 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished PASS 26 (238s) == replay-dual test 28: lock replay should be ordered: waiting after granted ========================================================== 06:06:01 (1765105561) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00586019 s, 699 kB/s fail_loc=0x80000324 fail_loc=0x32a Failing ost1 on oleg459-server Stopping /mnt/lustre-ost1 (opts:) on oleg459-server 06:06:20 (1765105580) shut down facet: ost1 facet_host: oleg459-server facet_failover_host: oleg459-server Failover ost1 to oleg459-server mount facets: ost1 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0105184 s, 389 kB/s oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 Started lustre-OST0000 06:06:38 (1765105598) targets are mounted 06:06:38 (1765105598) facet_failover done oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec PASS 28 (47s) == replay-dual test 29: replay vs update with the same xid ========================================================== 06:06:49 (1765105609) SKIP: replay-dual test_29 needs >= 2 MDTs SKIP 29 (2s) == replay-dual test 30: layout lock replay is not blocked on IO ========================================================== 06:06:50 (1765105610) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.015981 s, 2.6 MB/s 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.0160229 s, 2.6 MB/s fail_loc=0x32e fail_val=4 Failing mds1 on oleg459-server Stopping /mnt/lustre-mds1 (opts:) on oleg459-server 06:06:55 (1765105615) shut down facet: mds1 facet_host: oleg459-server facet_failover_host: oleg459-server Failover mds1 to oleg459-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 Started lustre-MDT0000 06:07:10 (1765105630) targets are mounted 06:07:10 (1765105630) facet_failover done 160+0 records in 160+0 records out 81920 bytes (82 kB, 80 KiB) copied, 22.4227 s, 3.7 kB/s oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 30 (30s) == replay-dual test 31: deadlock on file_remove_privs and occupied mod rpc slots ========================================================== 06:07:20 (1765105640) Failing ost1 on oleg459-server Stopping /mnt/lustre-ost1 (opts:) on oleg459-server 06:07:23 (1765105643) shut down Creating to objid 2465 on ost lustre-OST0000... facet: ost1 facet_host: oleg459-server facet_failover_host: oleg459-server total: 32 open/close in 0.41 seconds: 77.62 ops/second at_max=0 fail_loc=0x80001420 file /mnt/lustre2/d31.replay-dual/mdtdir/f31.replay-dual is ready Failover ost1 to oleg459-server mount facets: ost1 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 Started lustre-OST0000 06:07:40 (1765105660) targets are mounted 06:07:40 (1765105660) facet_failover done oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE FULL state after 0 sec pids: 54510 54511 54516 54517 54518 54519 54520 54521 54522 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3712 2204800 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3766272 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7532544 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 496475 393 496082 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 118219 459 117760 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 118153 393 117760 1% /mnt/lustre[OST:1] filesystem_summary: 235913 393 235520 1% /mnt/lustre at_max=600 PASS 31 (30s) == replay-dual test 32: gap in update llog shouldn't break recovery ========================================================== 06:07:50 (1765105670) SKIP: replay-dual test_32 needs >= 2 MDTs SKIP 32 (1s) == replay-dual test 33: Check for OBD_INCOMPAT_MULTI_RPCS in last_rcvd after abort_recovery ========================================================== 06:07:51 (1765105671) SKIP: replay-dual test_33 ldiskfs only test SKIP 33 (2s) == replay-dual test complete, duration 2606 sec ========== 06:07:53 (1765105673) === replay-dual: start cleanup 06:07:54 (1765105674) === Stopping clients: oleg459-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg459-client.virtnet /mnt/lustre2 opts: === replay-dual: finish cleanup 06:07:58 (1765105678) ===