-----============= acceptance-small: replay-dual ============----- Thu Jun 25 00:17:33 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg101-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/replay-dual.*ex': No such file or directory excepting tests: 14b 21b 21b skipping tests SLOW=no: 21b === replay-dual: start setup 00:17:40 (1782361060) === Starting client oleg101-client.virtnet: -o user_xattr,flock 192.168.201.101@tcp:/lustre /mnt/lustre2 Mount client oleg101-client.virtnet: mount -t lustre -o user_xattr,flock 192.168.201.101@tcp:/lustre /mnt/lustre2 Started clients oleg101-client.virtnet: 192.168.201.101@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) oleg101-client.virtnet: executing check_config_client /mnt/lustre oleg101-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg101-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff924c896b2000.idle_timeout=debug osc.lustre-OST0000-osc-ffff924c98965800.idle_timeout=debug osc.lustre-OST0001-osc-ffff924c896b2000.idle_timeout=debug osc.lustre-OST0001-osc-ffff924c98965800.idle_timeout=debug disable quota as required oleg101-server: oleg101-server.virtnet: executing set_default_debug -1 all === replay-dual: finish setup 00:17:52 (1782361072) === == replay-dual test 0a: expired recovery with lost client ========================================================== 00:17:52 (1782361072) Check file is LU482_FAILED=/tmp/replay-dual.lu482.7pmPS5 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre total: 50 open/close in 0.48 seconds: 104.47 ops/second fail_loc=0x80000514 Failing mds1 on oleg101-server Stopping /mnt/lustre-mds1 (opts:) on oleg101-server 00:17:56 (1782361076) shut down facet: mds1 facet_host: oleg101-server facet_failover_host: oleg101-server Failover mds1 to oleg101-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg101-server: oleg101-server.virtnet: executing set_default_debug -1 all pdsh@oleg101-client: oleg101-server: ssh exited with exit code 1 Started lustre-MDT0000 00:18:10 (1782361090) targets are mounted 00:18:10 (1782361090) facet_failover done oleg101-client: error: invalid path '/mnt/lustre': Input/output error pdsh@oleg101-client: oleg101-client: ssh exited with exit code 5 Starting client: oleg101-client.virtnet: -o user_xattr,flock 192.168.201.101@tcp:/lustre /mnt/lustre2 - unlinked 0 (time 1782361299 ; total 0 ; last 0) unlink(/mnt/lustre/f0a.replay-dual-17) error: No such file or directory total: 17 unlinks in 0 seconds: inf unlinks/second /home/green/git/lustre-release/lustre/tests/replay-dual.sh: line 72: errot: command not found PASS 0a (229s) == replay-dual test 0b: lost client during waiting for next transno ========================================================== 00:21:41 (1782361301) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre Failing mds1 on oleg101-server Stopping /mnt/lustre-mds1 (opts:) on oleg101-server 00:21:45 (1782361305) shut down facet: mds1 facet_host: oleg101-server facet_failover_host: oleg101-server Failover mds1 to oleg101-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg101-server: oleg101-server.virtnet: executing set_default_debug -1 all pdsh@oleg101-client: oleg101-server: ssh exited with exit code 1 Started lustre-MDT0000 00:21:58 (1782361318) targets are mounted 00:21:58 (1782361318) facet_failover done Starting client: oleg101-client.virtnet: -o user_xattr,flock 192.168.201.101@tcp:/lustre /mnt/lustre Starting client: oleg101-client.virtnet: -o user_xattr,flock 192.168.201.101@tcp:/lustre /mnt/lustre2 PASS 0b (91s) == replay-dual test 1: |X| simple create ================= 00:23:12 (1782361392) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre Failing mds1 on oleg101-server Stopping /mnt/lustre-mds1 (opts:) on oleg101-server 00:23:16 (1782361396) shut down facet: mds1 facet_host: oleg101-server facet_failover_host: oleg101-server Failover mds1 to oleg101-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg101-server: oleg101-server.virtnet: executing set_default_debug -1 all pdsh@oleg101-client: oleg101-server: ssh exited with exit code 1 Started lustre-MDT0000 00:23:29 (1782361409) targets are mounted 00:23:29 (1782361409) facet_failover done oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 1 (35s) == replay-dual test 2: |X| mkdir adir ==================== 00:23:47 (1782361427) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre Failing mds1 on oleg101-server Stopping /mnt/lustre-mds1 (opts:) on oleg101-server 00:23:50 (1782361430) shut down facet: mds1 facet_host: oleg101-server facet_failover_host: oleg101-server Failover mds1 to oleg101-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg101-server: oleg101-server.virtnet: executing set_default_debug -1 all pdsh@oleg101-client: oleg101-server: ssh exited with exit code 1 Started lustre-MDT0000 00:24:04 (1782361444) targets are mounted 00:24:04 (1782361444) facet_failover done oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 2 (23s) == replay-dual test 3: |X| mkdir adir, mkdir adir/bdir === 00:24:10 (1782361450) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre Failing mds1 on oleg101-server Stopping /mnt/lustre-mds1 (opts:) on oleg101-server 00:24:14 (1782361454) shut down facet: mds1 facet_host: oleg101-server facet_failover_host: oleg101-server Failover mds1 to oleg101-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg101-server: oleg101-server.virtnet: executing set_default_debug -1 all pdsh@oleg101-client: oleg101-server: ssh exited with exit code 1 Started lustre-MDT0000 00:24:28 (1782361468) targets are mounted 00:24:28 (1782361468) facet_failover done oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 3 (24s) == replay-dual test 4: |X| mkdir adir (-EEXIST), mkdir adir/bdir ========================================================== 00:24:34 (1782361474) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre mkdir: cannot create directory '/mnt/lustre/adir': File exists Failing mds1 on oleg101-server Stopping /mnt/lustre-mds1 (opts:) on oleg101-server 00:24:38 (1782361478) shut down facet: mds1 facet_host: oleg101-server facet_failover_host: oleg101-server Failover mds1 to oleg101-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg101-server: oleg101-server.virtnet: executing set_default_debug -1 all pdsh@oleg101-client: oleg101-server: ssh exited with exit code 1 Started lustre-MDT0000 00:24:51 (1782361491) targets are mounted 00:24:51 (1782361491) facet_failover done oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 4 (25s) == replay-dual test 5: open, unlink |X| close ============ 00:24:59 (1782361499) multiop /mnt/lustre2/a vo_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.7605 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre Failing mds1 on oleg101-server Stopping /mnt/lustre-mds1 (opts:) on oleg101-server 00:25:02 (1782361502) shut down facet: mds1 facet_host: oleg101-server facet_failover_host: oleg101-server Failover mds1 to oleg101-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg101-server: oleg101-server.virtnet: executing set_default_debug -1 all pdsh@oleg101-client: oleg101-server: ssh exited with exit code 1 Started lustre-MDT0000 00:25:16 (1782361516) targets are mounted 00:25:16 (1782361516) facet_failover done oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 5 (24s) == replay-dual test 6: open1, open2, unlink |X| close1 [fail mds1] close2 ========================================================== 00:25:23 (1782361523) multiop /mnt/lustre2/a vo_c TMPPIPE=/tmp/multiop_open_wait_pipe.7605 multiop /mnt/lustre/a vo_c TMPPIPE=/tmp/multiop_open_wait_pipe.7605 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre Failing mds1 on oleg101-server Stopping /mnt/lustre-mds1 (opts:) on oleg101-server 00:25:26 (1782361526) shut down facet: mds1 facet_host: oleg101-server facet_failover_host: oleg101-server Failover mds1 to oleg101-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg101-server: oleg101-server.virtnet: executing set_default_debug -1 all pdsh@oleg101-client: oleg101-server: ssh exited with exit code 1 Started lustre-MDT0000 00:25:40 (1782361540) targets are mounted 00:25:40 (1782361540) facet_failover done oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 6 (24s) == replay-dual test 8: replay of resent request ========== 00:25:47 (1782361547) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre fail_loc=0x119 fail_loc=0 Failing mds1 on oleg101-server Stopping /mnt/lustre-mds1 (opts:) on oleg101-server 00:26:07 (1782361567) shut down facet: mds1 facet_host: oleg101-server facet_failover_host: oleg101-server Failover mds1 to oleg101-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg101-server: oleg101-server.virtnet: executing set_default_debug -1 all pdsh@oleg101-client: oleg101-server: ssh exited with exit code 1 Started lustre-MDT0000 00:26:21 (1782361581) targets are mounted 00:26:21 (1782361581) facet_failover done oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 8 (41s) == replay-dual test 9: resending a replayed create ======= 00:26:28 (1782361588) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre fail_loc=0x80000119 Failing mds1 on oleg101-server Stopping /mnt/lustre-mds1 (opts:) on oleg101-server 00:26:31 (1782361591) shut down facet: mds1 facet_host: oleg101-server facet_failover_host: oleg101-server Failover mds1 to oleg101-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg101-server: oleg101-server.virtnet: executing set_default_debug -1 all pdsh@oleg101-client: oleg101-server: ssh exited with exit code 1 Started lustre-MDT0000 00:26:45 (1782361605) targets are mounted 00:26:45 (1782361605) facet_failover done oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec fail_loc=0 PASS 9 (39s) == replay-dual test 10: resending a replayed unlink ====== 00:27:07 (1782361627) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre fail_loc=0x80000119 Failing mds1 on oleg101-server Stopping /mnt/lustre-mds1 (opts:) on oleg101-server 00:27:11 (1782361631) shut down facet: mds1 facet_host: oleg101-server facet_failover_host: oleg101-server Failover mds1 to oleg101-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg101-server: oleg101-server.virtnet: executing set_default_debug -1 all pdsh@oleg101-client: oleg101-server: ssh exited with exit code 1 Started lustre-MDT0000 00:27:25 (1782361645) targets are mounted 00:27:25 (1782361645) facet_failover done oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec fail_loc=0 PASS 10 (41s) == replay-dual test 11: both clients timeout during replay ========================================================== 00:27:48 (1782361668) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre fail_loc=0x0119 Failing mds1 on oleg101-server Stopping /mnt/lustre-mds1 (opts:) on oleg101-server 00:27:52 (1782361672) shut down facet: mds1 facet_host: oleg101-server facet_failover_host: oleg101-server Failover mds1 to oleg101-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg101-server: oleg101-server.virtnet: executing set_default_debug -1 all pdsh@oleg101-client: oleg101-server: ssh exited with exit code 1 Started lustre-MDT0000 00:28:06 (1782361686) targets are mounted 00:28:06 (1782361686) facet_failover done oleg101-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 16 sec fail_loc=0 PASS 11 (38s) == replay-dual test 12: open resend timeout ============== 00:28:26 (1782361706) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre multiop /mnt/lustre/f12.replay-dual vmo_c TMPPIPE=/tmp/multiop_open_wait_pipe.7605 fail_loc=0x80000302 Failing mds1 on oleg101-server Stopping /mnt/lustre-mds1 (opts:) on oleg101-server 00:28:29 (1782361709) shut down facet: mds1 facet_host: oleg101-server facet_failover_host: oleg101-server Failover mds1 to oleg101-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg101-server: oleg101-server.virtnet: executing set_default_debug -1 all pdsh@oleg101-client: oleg101-server: ssh exited with exit code 1 Started lustre-MDT0000 00:28:43 (1782361723) targets are mounted 00:28:43 (1782361723) facet_failover done fail_loc=0 /mnt/lustre/f12.replay-dual /mnt/lustre/f12.replay-dual has type file OK PASS 12 (34s) == replay-dual test 13: close resend timeout ============= 00:29:00 (1782361740) multiop /mnt/lustre/f13.replay-dual vmo_c TMPPIPE=/tmp/multiop_open_wait_pipe.7605 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre fail_loc=0x80000115 Failing mds1 on oleg101-server Stopping /mnt/lustre-mds1 (opts:) on oleg101-server 00:29:04 (1782361744) shut down facet: mds1 facet_host: oleg101-server facet_failover_host: oleg101-server Failover mds1 to oleg101-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg101-server: oleg101-server.virtnet: executing set_default_debug -1 all pdsh@oleg101-client: oleg101-server: ssh exited with exit code 1 Started lustre-MDT0000 00:29:17 (1782361757) targets are mounted 00:29:17 (1782361757) facet_failover done fail_loc=0 /mnt/lustre/f13.replay-dual /mnt/lustre/f13.replay-dual has type file OK PASS 13 (21s) SKIP: replay-dual test_14b skipping ALWAYS excluded test 14b == replay-dual test 15a: timeout waiting for lost client during replay, 1 client completes ========================================================== 00:29:21 (1782361761) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2048 2206592 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre total: 25 open/close in 0.23 seconds: 106.39 ops/second total: 1 open/close in 0.01 seconds: 103.79 ops/second Failing mds1 on oleg101-server Stopping /mnt/lustre-mds1 (opts:) on oleg101-server 00:29:25 (1782361765) shut down facet: mds1 facet_host: oleg101-server facet_failover_host: oleg101-server Failover mds1 to oleg101-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg101-server: oleg101-server.virtnet: executing set_default_debug -1 all pdsh@oleg101-client: oleg101-server: ssh exited with exit code 1 Started lustre-MDT0000 00:29:38 (1782361778) targets are mounted 00:29:38 (1782361778) facet_failover done oleg101-client: error: invalid path '/mnt/lustre': Input/output error pdsh@oleg101-client: oleg101-client: ssh exited with exit code 5 oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec - unlinked 0 (time 1782361965 ; total 0 ; last 0) unlink(/mnt/lustre/f15a.replay-dual-11) error: No such file or directory total: 11 unlinks in 0 seconds: inf unlinks/second replay-dual test_15a: @@@@@@ FAIL: test_15a failed with 2 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7447:error() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7823:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7886:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7701:run_test() = /home/green/git/lustre-release/lustre/tests/replay-dual.sh:376:main() Dumping lctl log to /tmp/testlogs//replay-dual.test_15a.*.1782361966.log rsync: chown "/tmp/testlogs/.replay-dual.test_15a.debug_log.oleg101-server.1782361966.log.zuEYZq" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.replay-dual.test_15a.dmesg.oleg101-server.1782361966.log.jvbQjr" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3] FAIL 15a (206s) == replay-dual test 15c: remove multiple OST orphans ===== 00:32:49 (1782361969) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3770368 2048 3766272 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7541760 4096 7533568 1% /mnt/lustre umount: /mnt/lustre2: not mounted. Failing mds1 on oleg101-server Stopping /mnt/lustre-mds1 (opts:) on oleg101-server 00:32:53 (1782361973) shut down facet: mds1 facet_host: oleg101-server facet_failover_host: oleg101-server Failover mds1 to oleg101-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg101-server: oleg101-server.virtnet: executing set_default_debug -1 all pdsh@oleg101-client: oleg101-server: ssh exited with exit code 1 Started lustre-MDT0000 00:33:07 (1782361987) targets are mounted 00:33:07 (1782361987) facet_failover done oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Starting client: oleg101-client.virtnet: -o user_xattr,flock 192.168.201.101@tcp:/lustre /mnt/lustre2 PASS 15c (24s) == replay-dual test 16: fail MDS during recovery (3571) == 00:33:13 (1782361993) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre total: 25 open/close in 0.25 seconds: 100.16 ops/second total: 1 open/close in 0.01 seconds: 103.40 ops/second Failing mds1 on oleg101-server Stopping /mnt/lustre-mds1 (opts:) on oleg101-server 00:33:17 (1782361997) shut down facet: mds1 facet_host: oleg101-server facet_failover_host: oleg101-server Failover mds1 to oleg101-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg101-server: oleg101-server.virtnet: executing set_default_debug -1 all pdsh@oleg101-client: oleg101-server: ssh exited with exit code 1 Started lustre-MDT0000 00:33:31 (1782362011) targets are mounted 00:33:31 (1782362011) facet_failover done Failing mds1 on oleg101-server Stopping /mnt/lustre-mds1 (opts:) on oleg101-server 00:33:53 (1782362033) shut down facet: mds1 facet_host: oleg101-server facet_failover_host: oleg101-server Failover mds1 to oleg101-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg101-server: oleg101-server.virtnet: executing set_default_debug -1 all pdsh@oleg101-client: oleg101-server: ssh exited with exit code 1 Started lustre-MDT0000 00:34:07 (1782362047) targets are mounted 00:34:07 (1782362047) facet_failover done oleg101-client: error: invalid path '/mnt/lustre': Input/output error pdsh@oleg101-client: oleg101-client: ssh exited with exit code 5 oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec - unlinked 0 (time 1782362239 ; total 0 ; last 0) unlink(/mnt/lustre/f16.replay-dual-11) error: No such file or directory total: 11 unlinks in 0 seconds: inf unlinks/second replay-dual test_16: @@@@@@ FAIL: test_16 failed with 2 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7447:error() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7823:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7886:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7701:run_test() = /home/green/git/lustre-release/lustre/tests/replay-dual.sh:408:main() Dumping lctl log to /tmp/testlogs//replay-dual.test_16.*.1782362240.log rsync: chown "/tmp/testlogs/.replay-dual.test_16.debug_log.oleg101-server.1782362240.log.x9hnHe" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.replay-dual.test_16.dmesg.oleg101-server.1782362240.log.z4dPNE" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3] FAIL 16 (248s) == replay-dual test 17: fail OST during recovery (3571) == 00:37:24 (1782362244) total: 25 open/close in 0.24 seconds: 103.29 ops/second total: 1 open/close in 0.00 seconds: 28532.68 ops/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre umount: /mnt/lustre2: not mounted. Failing ost1 on oleg101-server Stopping /mnt/lustre-ost1 (opts:) on oleg101-server 00:37:27 (1782362247) shut down facet: ost1 facet_host: oleg101-server facet_failover_host: oleg101-server Failover ost1 to oleg101-server mount facets: ost1 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg101-server: oleg101-server.virtnet: executing set_default_debug -1 all pdsh@oleg101-client: oleg101-server: ssh exited with exit code 1 Started lustre-OST0000 00:37:41 (1782362261) targets are mounted 00:37:41 (1782362261) facet_failover done Failing ost1 on oleg101-server Stopping /mnt/lustre-ost1 (opts:) on oleg101-server 00:38:03 (1782362283) shut down facet: ost1 facet_host: oleg101-server facet_failover_host: oleg101-server Failover ost1 to oleg101-server mount facets: ost1 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg101-server: oleg101-server.virtnet: executing set_default_debug -1 all pdsh@oleg101-client: oleg101-server: ssh exited with exit code 1 Started lustre-OST0000 00:38:18 (1782362298) targets are mounted 00:38:18 (1782362298) facet_failover done oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec - unlinked 0 (time 1782362303 ; total 0 ; last 0) total: 25 unlinks in 0 seconds: inf unlinks/second Starting client: oleg101-client.virtnet: -o user_xattr,flock 192.168.201.101@tcp:/lustre /mnt/lustre2 PASS 17 (61s) == replay-dual test 18: ldlm_handle_enqueue succeeds on evicted export (3822) ========================================================== 00:38:25 (1782362305) debug=+dlmtrace fail_loc=0x8000030b using seed 247478884 running for 500 iterations total: 500 stats in 0 seconds: inf stats/second ldlm.namespaces.MGC192.168.201.101@tcp.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0000-mdc-ffff924c88bea000.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0000-mdc-ffff924c903ed800.early_lock_cancel=0 ldlm.namespaces.lustre-OST0000-osc-ffff924c88bea000.early_lock_cancel=0 ldlm.namespaces.lustre-OST0000-osc-ffff924c903ed800.early_lock_cancel=0 ldlm.namespaces.lustre-OST0001-osc-ffff924c88bea000.early_lock_cancel=0 ldlm.namespaces.lustre-OST0001-osc-ffff924c903ed800.early_lock_cancel=0 fail_loc=0x80000305 Error in opening file "/mnt/lustre2/d18.replay-dual/f18.replay-dual"(flags=O_RDONLY) 2: No such file or directory ldlm.namespaces.MGC192.168.201.101@tcp.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0000-mdc-ffff924c88bea000.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0000-mdc-ffff924c903ed800.early_lock_cancel=1 ldlm.namespaces.lustre-OST0000-osc-ffff924c88bea000.early_lock_cancel=1 ldlm.namespaces.lustre-OST0000-osc-ffff924c903ed800.early_lock_cancel=1 ldlm.namespaces.lustre-OST0001-osc-ffff924c88bea000.early_lock_cancel=1 ldlm.namespaces.lustre-OST0001-osc-ffff924c903ed800.early_lock_cancel=1 fail_loc=0 fail_loc=0 PASS 18 (47s) == replay-dual test 19: resend of open request =========== 00:39:12 (1782362352) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre fail_loc=0x157 - open/close 0 (time 1782362441.92 total 87.21 last 0.00) total: 1 open/close in 87.21 seconds: 0.01 ops/second fail_loc=0 Failing mds1 on oleg101-server Stopping /mnt/lustre-mds1 (opts:) on oleg101-server 00:40:43 (1782362443) shut down facet: mds1 facet_host: oleg101-server facet_failover_host: oleg101-server Failover mds1 to oleg101-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg101-server: oleg101-server.virtnet: executing set_default_debug -1 all pdsh@oleg101-client: oleg101-server: ssh exited with exit code 1 Started lustre-MDT0000 00:40:57 (1782362457) targets are mounted 00:40:57 (1782362457) facet_failover done oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 19 (120s) == replay-dual test 20: recovery time is not increasing == 00:41:12 (1782362472) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 2176 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre Failing mds1 on oleg101-server Stopping /mnt/lustre-mds1 (opts:) on oleg101-server 00:41:15 (1782362475) shut down facet: mds1 facet_host: oleg101-server facet_failover_host: oleg101-server Failover mds1 to oleg101-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg101-server: oleg101-server.virtnet: executing set_default_debug -1 all pdsh@oleg101-client: oleg101-server: ssh exited with exit code 1 Started lustre-MDT0000 00:41:29 (1782362489) targets are mounted 00:41:29 (1782362489) facet_failover done oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Starting client: oleg101-client.virtnet: -o user_xattr,flock 192.168.201.101@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2048 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre Failing mds1 on oleg101-server Stopping /mnt/lustre-mds1 (opts:) on oleg101-server 00:44:04 (1782362644) shut down facet: mds1 facet_host: oleg101-server facet_failover_host: oleg101-server Failover mds1 to oleg101-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg101-server: oleg101-server.virtnet: executing set_default_debug -1 all pdsh@oleg101-client: oleg101-server: ssh exited with exit code 1 Started lustre-MDT0000 00:44:18 (1782362658) targets are mounted 00:44:18 (1782362658) facet_failover done oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Starting client: oleg101-client.virtnet: -o user_xattr,flock 192.168.201.101@tcp:/lustre /mnt/lustre2 PASS 20 (341s) == replay-dual test 21a: commit on sharing =============== 00:46:53 (1782362813) mdt.lustre-MDT0000.commit_on_sharing=1 Replay barrier on lustre-MDT0000 Failing mds1 on oleg101-server Stopping /mnt/lustre-mds1 (opts:) on oleg101-server 00:46:57 (1782362817) shut down facet: mds1 facet_host: oleg101-server facet_failover_host: oleg101-server Failover mds1 to oleg101-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg101-server: oleg101-server.virtnet: executing set_default_debug -1 all pdsh@oleg101-client: oleg101-server: ssh exited with exit code 1 Started lustre-MDT0000 00:47:10 (1782362830) targets are mounted 00:47:10 (1782362830) facet_failover done Starting client: oleg101-client.virtnet: -o user_xattr,flock 192.168.201.101@tcp:/lustre /mnt/lustre2 mdt.lustre-MDT0000.commit_on_sharing=0 PASS 21a (160s) SKIP: replay-dual test_21b skipping SLOW test 21b == replay-dual test 22a: c1 lfs mkdir -i 1 dir1, M1 drop reply & fail, c2 mkdir dir1/dir ========================================================== 00:49:33 (1782362973) SKIP: replay-dual test_22a needs >= 2 MDTs SKIP 22a (1s) == replay-dual test 22b: c1 lfs mkdir -i 1 d1, M1 drop reply & fail M0/M1, c2 mkdir d1/dir ========================================================== 00:49:34 (1782362974) SKIP: replay-dual test_22b needs >= 2 MDTs SKIP 22b (1s) == replay-dual test 22c: c1 lfs mkdir -i 1 d1, M1 drop update & fail M1, c2 mkdir d1/dir ========================================================== 00:49:35 (1782362975) SKIP: replay-dual test_22c needs >= 2 MDTs SKIP 22c (1s) == replay-dual test 22d: c1 lfs mkdir -i 1 d1, M1 drop update & fail M0/M1,c2 mkdir d1/dir ========================================================== 00:49:36 (1782362976) SKIP: replay-dual test_22d needs >= 2 MDTs SKIP 22d (1s) == replay-dual test 23a: c1 rmdir d1, M1 drop reply and fail, client2 mkdir d1 ========================================================== 00:49:37 (1782362977) SKIP: replay-dual test_23a needs >= 2 MDTs SKIP 23a (1s) == replay-dual test 23b: c1 rmdir d1, M1 drop reply and fail M0/M1, c2 mkdir d1 ========================================================== 00:49:38 (1782362978) SKIP: replay-dual test_23b needs >= 2 MDTs SKIP 23b (1s) == replay-dual test 23c: c1 rmdir d1, M0 drop update reply and fail M0, c2 mkdir d1 ========================================================== 00:49:39 (1782362979) SKIP: replay-dual test_23c needs >= 2 MDTs SKIP 23c (1s) == replay-dual test 23d: c1 rmdir d1, M0 drop update reply and fail M0/M1, c2 mkdir d1 ========================================================== 00:49:40 (1782362980) SKIP: replay-dual test_23d needs >= 2 MDTs SKIP 23d (1s) == replay-dual test 24: reconstruct on non-existing object ========================================================== 00:49:41 (1782362981) fail_loc=0x119 fail_loc=0 truncate: cannot truncate '/mnt/lustre/f24.replay-dual' to length 100: No such file or directory PASS 24 (88s) == replay-dual test 25: replay|resend ==================== 00:51:09 (1782363069) 1+0 records in 1+0 records out 512 bytes copied, 0.0041701 s, 123 kB/s fail_loc=0x304 fail_loc=0x80000325 Failing ost1 on oleg101-server Stopping /mnt/lustre-ost1 (opts:) on oleg101-server 00:51:14 (1782363074) shut down facet: ost1 facet_host: oleg101-server facet_failover_host: oleg101-server Failover ost1 to oleg101-server mount facets: ost1 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg101-server: oleg101-server.virtnet: executing set_default_debug -1 all pdsh@oleg101-client: oleg101-server: ssh exited with exit code 1 Started lustre-OST0000 00:51:28 (1782363088) targets are mounted 00:51:28 (1782363088) facet_failover done oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 6933: 43706 Terminated LUSTRE="/home/green/git/lustre-release/lustre" bash -c "multiop /mnt/lustre2/f25.replay-dual Ow512" fail_loc=0 PASS 25 (26s) == replay-dual test 26: dbench and tar with mds failover ========================================================== 00:51:35 (1782363095) Starting client oleg101-client.virtnet: -o user_xattr,flock 192.168.201.101@tcp:/lustre /mnt/lustre Started clients oleg101-client.virtnet: 192.168.201.101@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) Started tar loop with pid 45127 Started dbench loop with 45128 looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Thu Jun 25 00:51:37 EDT 2026 waiting for dbench pid 45161 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs failed to create barrier semaphore 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2176 2206336 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 26624 3719168 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3743744 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 28672 7462912 1% /mnt/lustre 1 111 5.24 MB/sec warmup 1 sec latency 47.427 ms 1 197 3.92 MB/sec warmup 2 sec latency 75.711 ms 1 294 3.61 MB/sec warmup 3 sec latency 48.539 ms 1 402 3.55 MB/sec warmup 4 sec latency 39.813 ms test_26 fail mds1 1 times Failing mds1 on oleg101-server 1 511 3.55 MB/sec warmup 5 sec latency 41.543 ms Stopping /mnt/lustre-mds1 (opts:) on oleg101-server 1 600 3.49 MB/sec warmup 6 sec latency 529.155 ms 00:51:44 (1782363104) shut down facet: mds1 facet_host: oleg101-server facet_failover_host: oleg101-server 1 600 3.00 MB/sec warmup 7 sec latency 1529.316 ms 1 600 2.62 MB/sec warmup 8 sec latency 2529.555 ms 1 600 2.33 MB/sec warmup 9 sec latency 3529.745 ms 1 600 2.10 MB/sec warmup 10 sec latency 4529.924 ms 1 600 1.91 MB/sec warmup 11 sec latency 5530.118 ms 1 600 1.75 MB/sec warmup 12 sec latency 6530.293 ms 1 600 1.61 MB/sec warmup 13 sec latency 7530.547 ms 1 600 1.50 MB/sec warmup 14 sec latency 8530.741 ms 1 600 1.40 MB/sec warmup 15 sec latency 9530.985 ms 1 600 1.31 MB/sec warmup 16 sec latency 10531.230 ms Failover mds1 to oleg101-server mount facets: mds1 1 600 1.23 MB/sec warmup 17 sec latency 11531.423 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 600 1.16 MB/sec warmup 18 sec latency 12531.590 ms 1 600 1.10 MB/sec warmup 19 sec latency 13531.758 ms oleg101-server: oleg101-server.virtnet: executing set_default_debug -1 all pdsh@oleg101-client: oleg101-server: ssh exited with exit code 1 Started lustre-MDT0000 00:51:58 (1782363118) targets are mounted 00:51:58 (1782363118) facet_failover done 1 600 0.00 MB/sec execute 1 sec latency 15532.058 ms 1 600 0.00 MB/sec execute 2 sec latency 16532.186 ms 1 600 0.00 MB/sec execute 3 sec latency 17532.348 ms 1 600 0.00 MB/sec execute 4 sec latency 18532.497 ms 1 600 0.00 MB/sec execute 5 sec latency 19532.653 ms 1 600 0.00 MB/sec execute 6 sec latency 20532.846 ms 1 600 0.00 MB/sec execute 7 sec latency 21533.036 ms 1 600 0.00 MB/sec execute 8 sec latency 22533.215 ms 1 600 0.00 MB/sec execute 9 sec latency 23533.389 ms 1 600 0.00 MB/sec execute 10 sec latency 24533.619 ms 1 600 0.00 MB/sec execute 11 sec latency 25533.784 ms 1 600 0.00 MB/sec execute 12 sec latency 26533.931 ms 1 600 0.00 MB/sec execute 13 sec latency 27534.094 ms 1 600 0.00 MB/sec execute 14 sec latency 28534.248 ms 1 600 0.00 MB/sec execute 15 sec latency 29534.390 ms 1 600 0.00 MB/sec execute 16 sec latency 30534.526 ms 1 600 0.00 MB/sec execute 17 sec latency 31534.664 ms 1 600 0.00 MB/sec execute 18 sec latency 32534.789 ms 1 600 0.00 MB/sec execute 19 sec latency 33534.918 ms 1 600 0.00 MB/sec execute 20 sec latency 34535.071 ms 1 600 0.00 MB/sec execute 21 sec latency 35535.191 ms 1 600 0.00 MB/sec execute 22 sec latency 36535.308 ms 1 600 0.00 MB/sec execute 23 sec latency 37535.498 ms 1 600 0.00 MB/sec execute 24 sec latency 38535.653 ms 1 600 0.00 MB/sec execute 25 sec latency 39535.786 ms 1 600 0.00 MB/sec execute 26 sec latency 40535.931 ms 1 600 0.00 MB/sec execute 27 sec latency 41536.064 ms 1 600 0.00 MB/sec execute 28 sec latency 42537.327 ms 1 600 0.00 MB/sec execute 29 sec latency 43537.494 ms 1 600 0.00 MB/sec execute 30 sec latency 44537.689 ms 1 600 0.00 MB/sec execute 31 sec latency 45537.828 ms 1 600 0.00 MB/sec execute 32 sec latency 46538.047 ms 1 600 0.00 MB/sec execute 33 sec latency 47539.509 ms 1 600 0.00 MB/sec execute 34 sec latency 48539.657 ms 1 600 0.00 MB/sec execute 35 sec latency 49539.797 ms 1 600 0.00 MB/sec execute 36 sec latency 50540.003 ms 1 600 0.00 MB/sec execute 37 sec latency 51540.211 ms 1 600 0.00 MB/sec execute 38 sec latency 52540.351 ms 1 600 0.00 MB/sec execute 39 sec latency 53540.517 ms 1 600 0.00 MB/sec execute 40 sec latency 54540.674 ms 1 600 0.00 MB/sec execute 41 sec latency 55540.812 ms 1 600 0.00 MB/sec execute 42 sec latency 56540.959 ms 1 600 0.00 MB/sec execute 43 sec latency 57541.116 ms 1 600 0.00 MB/sec execute 44 sec latency 58541.251 ms 1 600 0.00 MB/sec execute 45 sec latency 59541.393 ms 1 600 0.00 MB/sec execute 46 sec latency 60541.517 ms 1 600 0.00 MB/sec execute 47 sec latency 61541.639 ms 1 600 0.00 MB/sec execute 48 sec latency 62541.767 ms 1 600 0.00 MB/sec execute 49 sec latency 63541.901 ms 1 600 0.00 MB/sec execute 50 sec latency 64542.041 ms 1 600 0.00 MB/sec execute 51 sec latency 65542.174 ms 1 600 0.00 MB/sec execute 52 sec latency 66542.292 ms 1 600 0.00 MB/sec execute 53 sec latency 67542.434 ms 1 600 0.00 MB/sec execute 54 sec latency 68542.681 ms 1 600 0.00 MB/sec execute 55 sec latency 69542.861 ms 1 600 0.00 MB/sec execute 56 sec latency 70542.994 ms 1 600 0.00 MB/sec execute 57 sec latency 71543.133 ms 1 600 0.00 MB/sec execute 58 sec latency 72543.277 ms 1 600 0.00 MB/sec execute 59 sec latency 73543.404 ms 1 600 0.00 MB/sec execute 60 sec latency 74543.556 ms 1 600 0.00 MB/sec execute 61 sec latency 75543.789 ms 1 600 0.00 MB/sec execute 62 sec latency 76543.975 ms 1 600 0.00 MB/sec execute 63 sec latency 77544.147 ms 1 600 0.00 MB/sec execute 64 sec latency 78544.304 ms 1 600 0.00 MB/sec execute 65 sec latency 79544.513 ms 1 600 0.00 MB/sec execute 66 sec latency 80544.722 ms 1 600 0.00 MB/sec execute 67 sec latency 81544.903 ms 1 600 0.00 MB/sec execute 68 sec latency 82546.160 ms 1 600 0.00 MB/sec execute 69 sec latency 83546.310 ms 1 600 0.00 MB/sec execute 70 sec latency 84546.451 ms 1 600 0.00 MB/sec execute 71 sec latency 85546.641 ms 1 600 0.00 MB/sec execute 72 sec latency 86546.801 ms 1 600 0.00 MB/sec execute 73 sec latency 87546.949 ms 1 600 0.00 MB/sec execute 74 sec latency 88547.071 ms 1 600 0.00 MB/sec execute 75 sec latency 89547.244 ms 1 600 0.00 MB/sec execute 76 sec latency 90547.390 ms 1 600 0.00 MB/sec execute 77 sec latency 91547.560 ms 1 600 0.00 MB/sec execute 78 sec latency 92547.842 ms 1 600 0.00 MB/sec execute 79 sec latency 93548.068 ms 1 600 0.00 MB/sec execute 80 sec latency 94548.333 ms 1 600 0.00 MB/sec execute 81 sec latency 95548.545 ms 1 600 0.00 MB/sec execute 82 sec latency 96548.750 ms 1 600 0.00 MB/sec execute 83 sec latency 97548.919 ms 1 600 0.00 MB/sec execute 84 sec latency 98549.107 ms 1 600 0.00 MB/sec execute 85 sec latency 99549.281 ms 1 600 0.00 MB/sec execute 86 sec latency 100549.508 ms 1 600 0.00 MB/sec execute 87 sec latency 101549.677 ms 1 600 0.00 MB/sec execute 88 sec latency 102549.860 ms 1 600 0.00 MB/sec execute 89 sec latency 103550.052 ms 1 600 0.00 MB/sec execute 90 sec latency 104550.213 ms 1 600 0.00 MB/sec execute 91 sec latency 105550.370 ms 1 600 0.00 MB/sec execute 92 sec latency 106550.540 ms 1 600 0.00 MB/sec execute 93 sec latency 107550.728 ms 1 600 0.00 MB/sec execute 94 sec latency 108550.922 ms 1 600 0.00 MB/sec execute 95 sec latency 109551.089 ms 1 600 0.00 MB/sec execute 96 sec latency 110551.261 ms 1 600 0.00 MB/sec execute 97 sec latency 111551.434 ms 1 600 0.00 MB/sec execute 98 sec latency 112551.621 ms 1 600 0.00 MB/sec execute 99 sec latency 113551.796 ms 1 cleanup 100 sec 1 cleanup 101 sec 1 cleanup 102 sec 1 cleanup 103 sec 1 cleanup 104 sec 1 cleanup 105 sec 1 cleanup 106 sec 1 cleanup 107 sec 1 cleanup 108 sec 1 cleanup 109 sec 1 cleanup 110 sec 1 cleanup 111 sec 1 cleanup 112 sec 1 cleanup 113 sec 1 cleanup 114 sec 1 cleanup 115 sec 1 cleanup 116 sec 1 cleanup 117 sec 1 cleanup 118 sec 1 cleanup 119 sec 1 cleanup 120 sec 1 cleanup 121 sec 1 cleanup 122 sec 1 cleanup 123 sec 1 cleanup 124 sec 1 cleanup 125 sec 1 cleanup 126 sec 1 cleanup 127 sec 1 cleanup 128 sec 1 cleanup 129 sec 1 cleanup 130 sec 1 cleanup 131 sec 1 cleanup 132 sec 1 cleanup 133 sec 1 cleanup 134 sec 1 cleanup 135 sec 1 cleanup 136 sec 1 cleanup 137 sec 1 cleanup 138 sec 1 cleanup 139 sec 1 cleanup 140 sec 1 cleanup 141 sec 1 cleanup 142 sec 1 cleanup 143 sec 1 cleanup 144 sec 1 cleanup 145 sec 1 cleanup 146 sec 1 cleanup 147 sec 1 cleanup 148 sec 1 cleanup 149 sec 1 cleanup 150 sec 1 cleanup 151 sec 1 cleanup 152 sec 1 cleanup 153 sec 1 cleanup 154 sec 1 cleanup 155 sec 1 cleanup 156 sec 1 cleanup 157 sec 1 cleanup 158 sec 1 cleanup 159 sec 1 cleanup 160 sec 1 cleanup 161 sec 1 cleanup 162 sec 1 cleanup 163 sec 1 cleanup 164 sec 1 cleanup 165 sec 1 cleanup 166 sec 1 cleanup 167 sec 1 cleanup 168 sec 1 cleanup 169 sec 1 cleanup 170 sec 1 cleanup 171 sec 1 cleanup 172 sec 1 cleanup 173 sec 1 cleanup 174 sec 1 cleanup 175 sec 1 cleanup 176 sec 1 cleanup 177 sec 1 cleanup 178 sec 1 cleanup 179 sec 1 cleanup 180 sec 1 cleanup 181 sec 1 cleanup 182 sec 1 cleanup 183 sec 1 cleanup 184 sec 1 cleanup 185 sec 1 cleanup 186 sec 1 cleanup 187 sec 1 cleanup 188 sec 1 cleanup 189 sec 1 cleanup 190 sec 1 cleanup 191 sec 1 cleanup 192 sec 1 cleanup 193 sec 1 cleanup 194 sec 1 cleanup 195 sec 1 cleanup 196 sec 1 cleanup 197 sec 1 cleanup 198 sec 1 cleanup 199 sec 1 cleanup 200 sec tar: etc/udev/hwdb.bin: Cannot utime: Input/output error oleg101-client: error: invalid path '/mnt/lustre': Input/output error pdsh@oleg101-client: oleg101-client: ssh exited with exit code 5 tar: etc/udev/hwdb.bin: Cannot change ownership to uid 0, gid 0: Cannot send after transport endpoint shutdown tar: etc/udev/hwdb.bin: Cannot change mode to r--r--r--: Cannot send after transport endpoint shutdown tar: etc/udev/rules.d: Cannot mkdir: Input/output error tar: etc/udev/rules.d/99-lustre-test.rules: Cannot open: Cannot send after transport endpoint shutdown tar: etc/udev/hwdb.d: Cannot mkdir: Input/output error tar: etc/udev: Cannot utime: Cannot send after transport endpoint shutdown 0 cleanup 201 sec Operation Count AvgLat MaxLat ---------------------------------------- Close 1 215185.422 215185.422 Throughput 0 MB/sec 1 clients 1 procs max_latency=113551.796 ms tar: etc/udev: Cannot change ownership to uid 0, gid 0: Cannot send after transport endpoint shutdown tar: etc/udev: Cannot change mode to rwxr-xr-x: Cannot send after transport endpoint shutdown tar: etc/tuned: Cannot mkdir: Cannot send after transport endpoint shutdown tar: etc/tuned/profile_mode: Cannot open: Cannot send after transport endpoint shutdown tar: etc/tuned/post_loaded_profile: Cannot open: Cannot send after transport endpoint shutdown tar: etc/tuned/bootcmdline: Cannot open: Cannot send after transport endpoint shutdown tar: etc/tuned/tuned-main.conf: Cannot open: Cannot send after transport endpoint shutdown tar: etc/tuned/active_profile: Cannot open: Cannot send after transport endpoint shutdown stopping dbench on /mnt/lustre at Thu Jun 25 00:55:18 EDT 2026 with return code 0 tar: etc/tuned/recommend.d: Cannot mkdir: Cannot send after transport endpoint shutdown tar: etc/tmpfiles.d: Cannot mkdir: Cannot send after transport endpoint shutdown tar: etc/terminfo: Cannot mkdir: Cannot send after transport endpoint shutdown tar: etc/systemd: Cannot mkdir: Cannot send after transport endpoint shutdown tar: etc/systemd/user.conf: Cannot open: Cannot send after transport endpoint shutdown tar: etc/systemd/resolved.conf: Cannot open: Cannot send after transport endpoint shutdown tar: etc/systemd/system.conf: Cannot open: Cannot send after transport endpoint shutdown tar: etc/systemd/journald.conf: Cannot open: Cannot send after transport endpoint shutdown tar: etc/systemd/coredump.conf: Cannot open: Cannot send after transport endpoint shutdown tar: etc/systemd/logind.conf: Cannot open: Cannot send after transport endpoint shutdown tar: etc/systemd/pstore.conf: Cannot open: Cannot send after transport endpoint shutdown tar: etc/systemd/user: Cannot mkdir: Cannot send after transport endpoint shutdown tar: etc/systemd/system: Cannot mkdir: Cannot send after transport endpoint shutdown tar: etc/systemd/system/proc-sys-fs-binfmt_misc.mount: Cannot open: Cannot send after transport endpoint shutdown tar: etc/systemd/system/timers.target.wants: Cannot mkdir: Cannot send after transport endpoint shutdown tar: etc/systemd/system/timers.target.wants/unbound-anchor.timer: Cannot open: Cannot send after transport endpoint shutdown tar: etc/systemd/system/sysinit.target.wants: Cannot mkdir: Cannot send after transport endpoint shutdown tar: etc/systemd/system/sysinit.target.wants/selinux-autorelabel-mark.service: Cannot open: Cannot send after transport endpoint shutdown tar: etc/systemd/system/sysinit.target.wants/rngd.service: Cannot open: Cannot send after transport endpoint shutdown tar: etc/systemd/system/sysinit.target.wants/nis-domainname.service: Cannot open: Cannot send after transport endpoint shutdown tar: etc/systemd/system/sysinit.target.wants/loadmodules.service: Cannot open: Cannot send after transport endpoint shutdown tar: etc/systemd/system/sysinit.target.wants/import-state.service: Cannot open: Cannot send after transport endpoint shutdown tar: etc/systemd/system/sockets.target.wants: Cannot mkdir: Cannot send after transport endpoint shutdown tar: etc/systemd/system/sockets.target.wants/rpcbind.socket: Cannot open: Cannot send after transport endpoint shutdown tar: etc/systemd/system/rpc-statd.service.requires: Cannot mkdir: Cannot send after transport endpoint shutdown tar: etc/systemd/system/rpc-statd.service.requires/nfs-convert.service: Cannot open: Cannot send after transport endpoint shutdown tar: etc/systemd/system/rpc-statd-notify.service.requires: Cannot mkdir: Cannot send after transport endpoint shutdown tar: etc/systemd/system/rpc-statd-notify.service.requires/nfs-convert.service: Cannot open: Cannot send after transport endpoint shutdown tar: etc/systemd/system/rpc-gssd.service.requires: Cannot mkdir: Cannot send after transport endpoint shutdown tar: etc/systemd/system/rpc-gssd.service.requires/nfs-convert.service: Cannot open: Cannot send after transport endpoint shutdown tar: etc/systemd/system/remote-fs.target.wants: Cannot mkdir: Cannot send after transport endpoint shutdown tar: etc/systemd/system/remote-fs.target.wants/nfs-client.target: Cannot open: Cannot send after transport endpoint shutdown tar: etc/systemd/system/nfs-server.service.requires: Cannot mkdir: Cannot send after transport endpoint shutdown tar: etc/systemd/system/nfs-server.service.requires/nfs-convert.service: Cannot open: Cannot send after transport endpoint shutdown tar: etc/systemd/system/nfs-mountd.service.requires: Cannot mkdir: Cannot send after transport endpoint shutdown tar: etc/systemd/system/nfs-mountd.service.requires/nfs-convert.service: Cannot open: Cannot send after transport endpoint shutdown tar: etc/systemd/system/nfs-idmapd.service.requires: Cannot mkdir: Cannot send after transport endpoint shutdown tar: etc/systemd/system/nfs-idmapd.service.requires/nfs-convert.service: Cannot open: Cannot send after transport endpoint shutdown tar: etc/systemd/system/nfs-blkmap.service.requires: Cannot mkdir: Cannot send after transport endpoint shutdown tar: etc/systemd/system/nfs-blkmap.service.requires/nfs-convert.service: Cannot open: Cannot send after transport endpoint shutdown tar: etc/systemd/system/network-online.target.wants: Cannot mkdir: Cannot send after transport endpoint shutdown tar: etc/systemd/system/network-online.target.wants/NetworkManager-wait-online.service: Cannot open: Cannot send after transport endpoint shutdown tar: etc/systemd/system/multi-user.target.wants: Cannot mkdir: Cannot send after transport endpoint shutdown tar: etc/systemd/system/multi-user.target.wants/tuned.service: Cannot open: Cannot send after transport endpoint shutdown tar: etc/systemd/system/multi-user.target.wants/sshd.service: Cannot open: Cannot send after transport endpoint shutdown tar: etc/systemd/system/multi-user.target.wants/rsyslog.service: Cannot open: Cannot send after transport endpoint shutdown tar: etc/systemd/system/multi-user.target.wants/rpcbind.service: Cannot open: Cannot send after transport endpoint shutdown tar: etc/systemd/system/multi-user.target.wants/remote-fs.target: Cannot open: Cannot send after transport endpoint shutdown tar: etc/systemd/system/multi-user.target.wants/nfs-client.target: Cannot open: Cannot send after transport endpoint shutdown tar: etc/systemd/system/multi-user.target.wants/kdump.service: Cannot open: Cannot send after transport endpoint shutdown tar: etc/systemd/system/multi-user.target.wants/irqbalance.service: Cannot open: Cannot send after transport endpoint shutdown clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid '/usr/share/dbench/client.txt' -> 'client.txt' mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Thu Jun 25 00:55:20 EDT 2026 waiting for dbench pid 46023 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 71 3.14 MB/sec warmup 1 sec latency 43.639 ms tar: Unexpected EOF in archive tar: Unexpected EOF in archive 1 150 3.14 MB/sec warmup 2 sec latency 62.630 ms tar: etc/xdg/systemd: Cannot stat: No such file or directory tar: etc/xdg: Cannot stat: No such file or directory tar: etc/yum: Cannot stat: No such file or directory tar: Error is not recoverable: exiting now dbench killed by signal 15 stopping dbench on /mnt/lustre at Thu Jun 25 00:55:23 EDT 2026 with return code 0 46023 pts/0 S+ 0:00 dbench -c client.txt 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100 killed dbench main pid 46023 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished PASS 26 (232s) == replay-dual test 28: lock replay should be ordered: waiting after granted ========================================================== 00:55:27 (1782363327) sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00383172 s, 1.1 MB/s fail_loc=0x80000324 fail_loc=0x32a Failing ost1 on oleg101-server Stopping /mnt/lustre-ost1 (opts:) on oleg101-server 00:55:51 (1782363351) shut down facet: ost1 facet_host: oleg101-server facet_failover_host: oleg101-server Failover ost1 to oleg101-server mount facets: ost1 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg101-server: oleg101-server.virtnet: executing set_default_debug -1 all 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00387929 s, 1.1 MB/s pdsh@oleg101-client: oleg101-server: ssh exited with exit code 1 Started lustre-OST0000 00:56:05 (1782363365) targets are mounted 00:56:05 (1782363365) facet_failover done oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec PASS 28 (47s) == replay-dual test 29: replay vs update with the same xid ========================================================== 00:56:14 (1782363374) SKIP: replay-dual test_29 needs >= 2 MDTs SKIP 29 (1s) == replay-dual test 30: layout lock replay is not blocked on IO ========================================================== 00:56:15 (1782363375) 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.0125481 s, 3.3 MB/s 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.0132361 s, 3.1 MB/s fail_loc=0x32e fail_val=4 Failing mds1 on oleg101-server Stopping /mnt/lustre-mds1 (opts:) on oleg101-server 00:56:18 (1782363378) shut down facet: mds1 facet_host: oleg101-server facet_failover_host: oleg101-server Failover mds1 to oleg101-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg101-server: oleg101-server.virtnet: executing set_default_debug -1 all pdsh@oleg101-client: oleg101-server: ssh exited with exit code 1 Started lustre-MDT0000 00:56:32 (1782363392) targets are mounted 00:56:32 (1782363392) facet_failover done 160+0 records in 160+0 records out 81920 bytes (82 kB, 80 KiB) copied, 18.5425 s, 4.4 kB/s oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 30 (24s) == replay-dual test 31: deadlock on file_remove_privs and occupied mod rpc slots ========================================================== 00:56:39 (1782363399) Failing ost1 on oleg101-server Stopping /mnt/lustre-ost1 (opts:) on oleg101-server 00:56:43 (1782363403) shut down facet: ost1 facet_host: oleg101-server facet_failover_host: oleg101-server Creating to objid 929 on ost lustre-OST0000... total: 32 open/close in 0.33 seconds: 97.79 ops/second at_max=0 fail_loc=0x80001420 file /mnt/lustre2/d31.replay-dual/mdtdir/f31.replay-dual is ready Failover ost1 to oleg101-server mount facets: ost1 Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg101-server: oleg101-server.virtnet: executing set_default_debug -1 all pdsh@oleg101-client: oleg101-server: ssh exited with exit code 1 Started lustre-OST0000 00:56:57 (1782363417) targets are mounted 00:56:57 (1782363417) facet_failover done oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL IDLE state after 0 sec pids: 50510 50511 50516 50517 50518 50519 50520 50521 50522 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2432 2206080 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3767296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 4096 7534592 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 527494 396 527098 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 118187 395 117792 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 118186 394 117792 1% /mnt/lustre[OST:1] filesystem_summary: 235980 396 235584 1% /mnt/lustre at_max=600 PASS 31 (26s) == replay-dual test 32: gap in update llog shouldn't break recovery ========================================================== 00:57:06 (1782363426) SKIP: replay-dual test_32 needs >= 2 MDTs SKIP 32 (2s) == replay-dual test 33: Check for OBD_INCOMPAT_MULTI_RPCS in last_rcvd after abort_recovery ========================================================== 00:57:07 (1782363427) SKIP: replay-dual test_33 ldiskfs only test SKIP 33 (1s) == replay-dual test complete, duration 2374 sec ========== 00:57:08 (1782363428) replay-dual: FAIL: test_15a test_15a failed with 2 replay-dual: FAIL: test_16 test_16 failed with 2 === replay-dual: start cleanup 00:57:08 (1782363428) === Stopping clients: oleg101-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg101-client.virtnet /mnt/lustre2 opts: === replay-dual: finish cleanup 00:57:11 (1782363431) ===