-----============= acceptance-small: replay-dual ============----- Thu Jun 25 00:18:05 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg408-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/replay-dual.*ex': No such file or directory excepting tests: 14b 21b skipping tests SLOW=no: 21b === replay-dual: start setup 00:18:12 (1782361092) === Starting client oleg408-client.virtnet: -o user_xattr,flock 192.168.204.108@tcp:/lustre /mnt/lustre2 Mount client oleg408-client.virtnet: mount -t lustre -o user_xattr,flock 192.168.204.108@tcp:/lustre /mnt/lustre2 Started clients oleg408-client.virtnet: 192.168.204.108@tcp:/lustre on /mnt/lustre2 type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) oleg408-client.virtnet: executing check_config_client /mnt/lustre oleg408-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg408-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffffa0f7520a4800.idle_timeout=debug osc.lustre-OST0000-osc-ffffa0f760a5a800.idle_timeout=debug osc.lustre-OST0001-osc-ffffa0f7520a4800.idle_timeout=debug osc.lustre-OST0001-osc-ffffa0f760a5a800.idle_timeout=debug disable quota as required oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all osd-ldiskfs.track_declares_assert=1 === replay-dual: finish setup 00:18:26 (1782361106) === == replay-dual test 0a: expired recovery with lost client ========================================================== 00:18:27 (1782361107) Check file is LU482_FAILED=/tmp/replay-dual.lu482.1De2LX UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1708 1269596 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1532 1269772 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3088 7210952 1% /mnt/lustre total: 50 open/close in 0.50 seconds: 99.70 ops/second fail_loc=0x80000514 Failing mds1 on oleg408-server Stopping /mnt/lustre-mds1 (opts:) on oleg408-server 00:18:33 (1782361113) shut down facet: mds1 facet_host: oleg408-server facet_failover_host: oleg408-server Failover mds1 to oleg408-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Started lustre-MDT0000 00:18:50 (1782361130) targets are mounted 00:18:50 (1782361130) facet_failover done oleg408-client: error: invalid path '/mnt/lustre': Input/output error pdsh@oleg408-client: oleg408-client: ssh exited with exit code 5 Starting client: oleg408-client.virtnet: -o user_xattr,flock 192.168.204.108@tcp:/lustre /mnt/lustre2 - unlinked 0 (time 1782361340 ; total 0 ; last 0) unlink(/mnt/lustre/f0a.replay-dual-17) error: No such file or directory total: 17 unlinks in 0 seconds: inf unlinks/second /home/green/git/lustre-release/lustre/tests/replay-dual.sh: line 72: errot: command not found PASS 0a (235s) == replay-dual test 0b: lost client during waiting for next transno ========================================================== 00:22:22 (1782361342) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1708 1269596 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1564 1269740 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3088 7210952 1% /mnt/lustre Failing mds1 on oleg408-server Stopping /mnt/lustre-mds1 (opts:) on oleg408-server 00:22:27 (1782361347) shut down facet: mds1 facet_host: oleg408-server facet_failover_host: oleg408-server Failover mds1 to oleg408-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Started lustre-MDT0000 00:22:43 (1782361363) targets are mounted 00:22:43 (1782361363) facet_failover done Starting client: oleg408-client.virtnet: -o user_xattr,flock 192.168.204.108@tcp:/lustre /mnt/lustre Starting client: oleg408-client.virtnet: -o user_xattr,flock 192.168.204.108@tcp:/lustre /mnt/lustre2 PASS 0b (100s) == replay-dual test 1: |X| simple create ================= 00:24:02 (1782361442) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1708 1269596 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1564 1269740 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3088 7210952 1% /mnt/lustre Failing mds1 on oleg408-server Stopping /mnt/lustre-mds1 (opts:) on oleg408-server 00:24:06 (1782361446) shut down facet: mds1 facet_host: oleg408-server facet_failover_host: oleg408-server Failover mds1 to oleg408-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Started lustre-MDT0000 00:24:22 (1782361462) targets are mounted 00:24:22 (1782361462) facet_failover done oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 1 (39s) == replay-dual test 2: |X| mkdir adir ==================== 00:24:41 (1782361481) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1708 1269596 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1564 1269740 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3088 7210952 1% /mnt/lustre Failing mds1 on oleg408-server Stopping /mnt/lustre-mds1 (opts:) on oleg408-server 00:24:46 (1782361486) shut down facet: mds1 facet_host: oleg408-server facet_failover_host: oleg408-server Failover mds1 to oleg408-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Started lustre-MDT0000 00:25:01 (1782361501) targets are mounted 00:25:01 (1782361501) facet_failover done oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 2 (29s) == replay-dual test 3: |X| mkdir adir, mkdir adir/bdir === 00:25:10 (1782361510) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1708 1269596 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1564 1269740 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3088 7210952 1% /mnt/lustre Failing mds1 on oleg408-server Stopping /mnt/lustre-mds1 (opts:) on oleg408-server 00:25:15 (1782361515) shut down facet: mds1 facet_host: oleg408-server facet_failover_host: oleg408-server Failover mds1 to oleg408-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Started lustre-MDT0000 00:25:31 (1782361531) targets are mounted 00:25:31 (1782361531) facet_failover done oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 3 (29s) == replay-dual test 4: |X| mkdir adir (-EEXIST), mkdir adir/bdir ========================================================== 00:25:40 (1782361540) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1880 1269424 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1772 1269532 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3088 7210952 1% /mnt/lustre mkdir: cannot create directory '/mnt/lustre/adir': File exists Failing mds1 on oleg408-server Stopping /mnt/lustre-mds1 (opts:) on oleg408-server 00:25:50 (1782361550) shut down facet: mds1 facet_host: oleg408-server facet_failover_host: oleg408-server Failover mds1 to oleg408-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Started lustre-MDT0000 00:26:06 (1782361566) targets are mounted 00:26:06 (1782361566) facet_failover done oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 4 (35s) == replay-dual test 5: open, unlink |X| close ============ 00:26:15 (1782361575) multiop /mnt/lustre2/a vo_tSc TMPPIPE=/tmp/multiop_open_wait_pipe.8196 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1852 1269452 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1772 1269532 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3088 7210952 1% /mnt/lustre Failing mds1 on oleg408-server Stopping /mnt/lustre-mds1 (opts:) on oleg408-server 00:26:20 (1782361580) shut down facet: mds1 facet_host: oleg408-server facet_failover_host: oleg408-server Failover mds1 to oleg408-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Started lustre-MDT0000 00:26:37 (1782361597) targets are mounted 00:26:37 (1782361597) facet_failover done oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 5 (30s) == replay-dual test 6: open1, open2, unlink |X| close1 [fail mds1] close2 ========================================================== 00:26:45 (1782361605) multiop /mnt/lustre2/a vo_c TMPPIPE=/tmp/multiop_open_wait_pipe.8196 multiop /mnt/lustre/a vo_c TMPPIPE=/tmp/multiop_open_wait_pipe.8196 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1852 1269452 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1772 1269532 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3088 7210952 1% /mnt/lustre Failing mds1 on oleg408-server Stopping /mnt/lustre-mds1 (opts:) on oleg408-server 00:26:49 (1782361609) shut down facet: mds1 facet_host: oleg408-server facet_failover_host: oleg408-server Failover mds1 to oleg408-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Started lustre-MDT0000 00:27:05 (1782361625) targets are mounted 00:27:05 (1782361625) facet_failover done oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 6 (28s) == replay-dual test 8: replay of resent request ========== 00:27:13 (1782361633) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1852 1269452 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1772 1269532 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3088 7210952 1% /mnt/lustre fail_loc=0x119 fail_loc=0 Failing mds1 on oleg408-server Stopping /mnt/lustre-mds1 (opts:) on oleg408-server 00:27:33 (1782361653) shut down facet: mds1 facet_host: oleg408-server facet_failover_host: oleg408-server Failover mds1 to oleg408-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Started lustre-MDT0000 00:27:49 (1782361669) targets are mounted 00:27:49 (1782361669) facet_failover done oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 8 (45s) == replay-dual test 9: resending a replayed create ======= 00:27:58 (1782361678) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1852 1269452 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1772 1269532 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3088 7210952 1% /mnt/lustre fail_loc=0x80000119 Failing mds1 on oleg408-server Stopping /mnt/lustre-mds1 (opts:) on oleg408-server 00:28:02 (1782361682) shut down facet: mds1 facet_host: oleg408-server facet_failover_host: oleg408-server Failover mds1 to oleg408-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Started lustre-MDT0000 00:28:18 (1782361698) targets are mounted 00:28:18 (1782361698) facet_failover done oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec fail_loc=0 PASS 9 (43s) == replay-dual test 10: resending a replayed unlink ====== 00:28:41 (1782361721) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1852 1269452 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1772 1269532 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3088 7210952 1% /mnt/lustre fail_loc=0x80000119 Failing mds1 on oleg408-server Stopping /mnt/lustre-mds1 (opts:) on oleg408-server 00:28:45 (1782361725) shut down facet: mds1 facet_host: oleg408-server facet_failover_host: oleg408-server Failover mds1 to oleg408-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Started lustre-MDT0000 00:29:01 (1782361741) targets are mounted 00:29:01 (1782361741) facet_failover done oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec fail_loc=0 PASS 10 (40s) == replay-dual test 11: both clients timeout during replay ========================================================== 00:29:21 (1782361761) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1852 1269452 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1772 1269532 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3088 7210952 1% /mnt/lustre fail_loc=0x0119 Failing mds1 on oleg408-server Stopping /mnt/lustre-mds1 (opts:) on oleg408-server 00:29:26 (1782361766) shut down facet: mds1 facet_host: oleg408-server facet_failover_host: oleg408-server Failover mds1 to oleg408-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Started lustre-MDT0000 00:29:41 (1782361781) targets are mounted 00:29:41 (1782361781) facet_failover done oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 14 sec fail_loc=0 PASS 11 (39s) == replay-dual test 12: open resend timeout ============== 00:30:00 (1782361800) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1852 1269452 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1772 1269532 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3088 7210952 1% /mnt/lustre multiop /mnt/lustre/f12.replay-dual vmo_c TMPPIPE=/tmp/multiop_open_wait_pipe.8196 fail_loc=0x80000302 Failing mds1 on oleg408-server Stopping /mnt/lustre-mds1 (opts:) on oleg408-server 00:30:05 (1782361805) shut down facet: mds1 facet_host: oleg408-server facet_failover_host: oleg408-server Failover mds1 to oleg408-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Started lustre-MDT0000 00:30:20 (1782361820) targets are mounted 00:30:20 (1782361820) facet_failover done fail_loc=0 /mnt/lustre/f12.replay-dual /mnt/lustre/f12.replay-dual has type file OK PASS 12 (25s) == replay-dual test 13: close resend timeout ============= 00:30:25 (1782361825) multiop /mnt/lustre/f13.replay-dual vmo_c TMPPIPE=/tmp/multiop_open_wait_pipe.8196 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1852 1269452 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1772 1269532 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3088 7210952 1% /mnt/lustre fail_loc=0x80000115 Failing mds1 on oleg408-server Stopping /mnt/lustre-mds1 (opts:) on oleg408-server 00:30:30 (1782361830) shut down facet: mds1 facet_host: oleg408-server facet_failover_host: oleg408-server Failover mds1 to oleg408-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Started lustre-MDT0000 00:30:46 (1782361846) targets are mounted 00:30:46 (1782361846) facet_failover done fail_loc=0 /mnt/lustre/f13.replay-dual /mnt/lustre/f13.replay-dual has type file OK PASS 13 (26s) SKIP: replay-dual test_14b skipping ALWAYS excluded test 14b == replay-dual test 15a: timeout waiting for lost client during replay, 1 client completes ========================================================== 00:30:51 (1782361851) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1852 1269452 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1772 1269532 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3088 7210952 1% /mnt/lustre total: 25 open/close in 0.24 seconds: 102.18 ops/second total: 1 open/close in 0.01 seconds: 95.56 ops/second Failing mds1 on oleg408-server Stopping /mnt/lustre-mds1 (opts:) on oleg408-server 00:30:56 (1782361856) shut down facet: mds1 facet_host: oleg408-server facet_failover_host: oleg408-server Failover mds1 to oleg408-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Started lustre-MDT0000 00:31:12 (1782361872) targets are mounted 00:31:12 (1782361872) facet_failover done oleg408-client: error: invalid path '/mnt/lustre': Input/output error pdsh@oleg408-client: oleg408-client: ssh exited with exit code 5 oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec - unlinked 0 (time 1782362059 ; total 0 ; last 0) unlink(/mnt/lustre/f15a.replay-dual-11) error: No such file or directory total: 11 unlinks in 0 seconds: inf unlinks/second replay-dual test_15a: @@@@@@ FAIL: test_15a failed with 2 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7447:error() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7823:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7886:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7701:run_test() = /home/green/git/lustre-release/lustre/tests/replay-dual.sh:376:main() Dumping lctl log to /tmp/testlogs//replay-dual.test_15a.*.1782362060.log rsync: chown "/tmp/testlogs/.replay-dual.test_15a.debug_log.oleg408-server.1782362060.log.6GmeEl" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.replay-dual.test_15a.dmesg.oleg408-server.1782362060.log.iD1X4c" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3] FAIL 15a (211s) == replay-dual test 15c: remove multiple OST orphans ===== 00:34:22 (1782362062) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1852 1269452 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1772 1269532 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3088 7210952 1% /mnt/lustre umount: /mnt/lustre2: not mounted. Failing mds1 on oleg408-server Stopping /mnt/lustre-mds1 (opts:) on oleg408-server 00:34:26 (1782362066) shut down facet: mds1 facet_host: oleg408-server facet_failover_host: oleg408-server Failover mds1 to oleg408-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Started lustre-MDT0000 00:34:42 (1782362082) targets are mounted 00:34:42 (1782362082) facet_failover done oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Starting client: oleg408-client.virtnet: -o user_xattr,flock 192.168.204.108@tcp:/lustre /mnt/lustre2 PASS 15c (28s) == replay-dual test 16: fail MDS during recovery (3571) == 00:34:50 (1782362090) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1852 1269452 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1772 1269532 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3088 7210952 1% /mnt/lustre total: 25 open/close in 0.23 seconds: 106.81 ops/second total: 1 open/close in 0.01 seconds: 96.46 ops/second Failing mds1 on oleg408-server Stopping /mnt/lustre-mds1 (opts:) on oleg408-server 00:34:55 (1782362095) shut down facet: mds1 facet_host: oleg408-server facet_failover_host: oleg408-server Failover mds1 to oleg408-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Started lustre-MDT0000 00:35:11 (1782362111) targets are mounted 00:35:11 (1782362111) facet_failover done Failing mds1 on oleg408-server Stopping /mnt/lustre-mds1 (opts:) on oleg408-server 00:35:33 (1782362133) shut down facet: mds1 facet_host: oleg408-server facet_failover_host: oleg408-server Failover mds1 to oleg408-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Started lustre-MDT0000 00:35:47 (1782362147) targets are mounted 00:35:47 (1782362147) facet_failover done oleg408-client: error: invalid path '/mnt/lustre': Input/output error pdsh@oleg408-client: oleg408-client: ssh exited with exit code 5 oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec - unlinked 0 (time 1782362334 ; total 0 ; last 0) unlink(/mnt/lustre/f16.replay-dual-11) error: No such file or directory total: 11 unlinks in 0 seconds: inf unlinks/second replay-dual test_16: @@@@@@ FAIL: test_16 failed with 2 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7447:error() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7823:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7886:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7701:run_test() = /home/green/git/lustre-release/lustre/tests/replay-dual.sh:408:main() Dumping lctl log to /tmp/testlogs//replay-dual.test_16.*.1782362335.log rsync: chown "/tmp/testlogs/.replay-dual.test_16.debug_log.oleg408-server.1782362335.log.MAjY64" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.replay-dual.test_16.dmesg.oleg408-server.1782362335.log.4wVef9" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3] FAIL 16 (246s) == replay-dual test 17: fail OST during recovery (3571) == 00:38:56 (1782362336) total: 25 open/close in 0.23 seconds: 107.78 ops/second total: 1 open/close in 0.00 seconds: 19239.93 ops/second UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1916 1269388 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1804 1269500 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3088 7210952 1% /mnt/lustre umount: /mnt/lustre2: not mounted. Failing ost1 on oleg408-server Stopping /mnt/lustre-ost1 (opts:) on oleg408-server 00:39:01 (1782362341) shut down facet: ost1 facet_host: oleg408-server facet_failover_host: oleg408-server Failover ost1 to oleg408-server mount facets: ost1 Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Started lustre-OST0000 00:39:17 (1782362357) targets are mounted 00:39:17 (1782362357) facet_failover done Failing ost1 on oleg408-server Stopping /mnt/lustre-ost1 (opts:) on oleg408-server 00:39:39 (1782362379) shut down facet: ost1 facet_host: oleg408-server facet_failover_host: oleg408-server Failover ost1 to oleg408-server mount facets: ost1 Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Started lustre-OST0000 00:39:53 (1782362393) targets are mounted 00:39:53 (1782362393) facet_failover done oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec - unlinked 0 (time 1782362399 ; total 0 ; last 0) total: 25 unlinks in 0 seconds: inf unlinks/second Starting client: oleg408-client.virtnet: -o user_xattr,flock 192.168.204.108@tcp:/lustre /mnt/lustre2 PASS 17 (64s) == replay-dual test 18: ldlm_handle_enqueue succeeds on evicted export (3822) ========================================================== 00:40:01 (1782362401) debug=+dlmtrace fail_loc=0x8000030b using seed 2847579667 running for 500 iterations total: 500 stats in 0 seconds: inf stats/second ldlm.namespaces.MGC192.168.204.108@tcp.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0000-mdc-ffffa0f746b59800.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0000-mdc-ffffa0f7500de000.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0001-mdc-ffffa0f746b59800.early_lock_cancel=0 ldlm.namespaces.lustre-MDT0001-mdc-ffffa0f7500de000.early_lock_cancel=0 ldlm.namespaces.lustre-OST0000-osc-ffffa0f746b59800.early_lock_cancel=0 ldlm.namespaces.lustre-OST0000-osc-ffffa0f7500de000.early_lock_cancel=0 ldlm.namespaces.lustre-OST0001-osc-ffffa0f746b59800.early_lock_cancel=0 ldlm.namespaces.lustre-OST0001-osc-ffffa0f7500de000.early_lock_cancel=0 fail_loc=0x80000305 Error in opening file "/mnt/lustre2/d18.replay-dual/f18.replay-dual"(flags=O_RDONLY) 2: No such file or directory ldlm.namespaces.MGC192.168.204.108@tcp.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0000-mdc-ffffa0f746b59800.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0000-mdc-ffffa0f7500de000.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0001-mdc-ffffa0f746b59800.early_lock_cancel=1 ldlm.namespaces.lustre-MDT0001-mdc-ffffa0f7500de000.early_lock_cancel=1 ldlm.namespaces.lustre-OST0000-osc-ffffa0f746b59800.early_lock_cancel=1 ldlm.namespaces.lustre-OST0000-osc-ffffa0f7500de000.early_lock_cancel=1 ldlm.namespaces.lustre-OST0001-osc-ffffa0f746b59800.early_lock_cancel=1 ldlm.namespaces.lustre-OST0001-osc-ffffa0f7500de000.early_lock_cancel=1 fail_loc=0 fail_loc=0 PASS 18 (46s) == replay-dual test 19: resend of open request =========== 00:40:47 (1782362447) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1920 1269384 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1804 1269500 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1560 3605460 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3104 7210936 1% /mnt/lustre fail_loc=0x157 - open/close 0 (time 1782362536.38 total 85.56 last 0.00) total: 1 open/close in 85.56 seconds: 0.01 ops/second fail_loc=0 Failing mds1 on oleg408-server Stopping /mnt/lustre-mds1 (opts:) on oleg408-server 00:42:18 (1782362538) shut down facet: mds1 facet_host: oleg408-server facet_failover_host: oleg408-server Failover mds1 to oleg408-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Started lustre-MDT0000 00:42:33 (1782362553) targets are mounted 00:42:33 (1782362553) facet_failover done oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 19 (125s) == replay-dual test 20: recovery time is not increasing == 00:42:52 (1782362572) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1920 1269384 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1804 1269500 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1560 3605460 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3104 7210936 1% /mnt/lustre Failing mds1 on oleg408-server Stopping /mnt/lustre-mds1 (opts:) on oleg408-server 00:42:57 (1782362577) shut down facet: mds1 facet_host: oleg408-server facet_failover_host: oleg408-server Failover mds1 to oleg408-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Started lustre-MDT0000 00:43:13 (1782362593) targets are mounted 00:43:13 (1782362593) facet_failover done oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Starting client: oleg408-client.virtnet: -o user_xattr,flock 192.168.204.108@tcp:/lustre /mnt/lustre2 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1920 1269384 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1804 1269500 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1560 3605460 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3104 7210936 1% /mnt/lustre Failing mds1 on oleg408-server Stopping /mnt/lustre-mds1 (opts:) on oleg408-server 00:45:50 (1782362750) shut down facet: mds1 facet_host: oleg408-server facet_failover_host: oleg408-server Failover mds1 to oleg408-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Started lustre-MDT0000 00:46:05 (1782362765) targets are mounted 00:46:05 (1782362765) facet_failover done oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Starting client: oleg408-client.virtnet: -o user_xattr,flock 192.168.204.108@tcp:/lustre /mnt/lustre2 PASS 20 (350s) == replay-dual test 21a: commit on sharing =============== 00:48:42 (1782362922) mdt.lustre-MDT0000.commit_on_sharing=1 mdt.lustre-MDT0001.commit_on_sharing=1 Replay barrier on lustre-MDT0000 Failing mds1 on oleg408-server Stopping /mnt/lustre-mds1 (opts:) on oleg408-server 00:48:47 (1782362927) shut down facet: mds1 facet_host: oleg408-server facet_failover_host: oleg408-server Failover mds1 to oleg408-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Started lustre-MDT0000 00:49:03 (1782362943) targets are mounted 00:49:03 (1782362943) facet_failover done Starting client: oleg408-client.virtnet: -o user_xattr,flock 192.168.204.108@tcp:/lustre /mnt/lustre2 mdt.lustre-MDT0000.commit_on_sharing=0 mdt.lustre-MDT0001.commit_on_sharing=0 PASS 21a (162s) SKIP: replay-dual test_21b skipping SLOW test 21b == replay-dual test 22a: c1 lfs mkdir -i 1 dir1, M1 drop reply & fail, c2 mkdir dir1/dir ========================================================== 00:51:25 (1782363085) fail_loc=0x119 Failing mds2 on oleg408-server Stopping /mnt/lustre-mds2 (opts:) on oleg408-server 00:51:28 (1782363088) shut down facet: mds2 facet_host: oleg408-server facet_failover_host: oleg408-server Failover mds2 to oleg408-server mount facets: mds2 Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Started lustre-MDT0001 00:51:42 (1782363102) targets are mounted 00:51:42 (1782363102) facet_failover done oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1928 1269376 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1808 1269496 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1560 3605460 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3104 7210936 1% /mnt/lustre total: 2 open/close in 0.04 seconds: 53.00 ops/second total: 2 open/close in 0.02 seconds: 119.25 ops/second Failing mds1 on oleg408-server Stopping /mnt/lustre-mds1 (opts:) on oleg408-server 00:51:54 (1782363114) shut down facet: mds1 facet_host: oleg408-server facet_failover_host: oleg408-server Failover mds1 to oleg408-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Started lustre-MDT0000 00:52:09 (1782363129) targets are mounted 00:52:09 (1782363129) facet_failover done oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 22a (53s) == replay-dual test 22b: c1 lfs mkdir -i 1 d1, M1 drop reply & fail M0/M1, c2 mkdir d1/dir ========================================================== 00:52:18 (1782363138) fail_loc=0x119 Failing mds1 on oleg408-server Stopping /mnt/lustre-mds1 (opts:) on oleg408-server Failing mds2 on oleg408-server Stopping /mnt/lustre-mds2 (opts:) on oleg408-server 00:52:23 (1782363143) shut down facet: mds1 facet_host: oleg408-server facet_failover_host: oleg408-server facet: mds2 facet_host: oleg408-server facet_failover_host: oleg408-server Failover mds1 to oleg408-server mount facets: mds1 Failover mds2 to oleg408-server mount facets: mds2 Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Started lustre-MDT0000 Started lustre-MDT0001 00:52:49 (1782363169) targets are mounted 00:52:49 (1782363169) facet_failover done oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1928 1269376 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1808 1269496 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1560 3605460 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3104 7210936 1% /mnt/lustre total: 2 open/close in 0.03 seconds: 62.82 ops/second total: 2 open/close in 0.01 seconds: 136.89 ops/second Failing mds1 on oleg408-server Stopping /mnt/lustre-mds1 (opts:) on oleg408-server 00:53:02 (1782363182) shut down facet: mds1 facet_host: oleg408-server facet_failover_host: oleg408-server Failover mds1 to oleg408-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Started lustre-MDT0000 00:53:17 (1782363197) targets are mounted 00:53:17 (1782363197) facet_failover done oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 22b (68s) == replay-dual test 22c: c1 lfs mkdir -i 1 d1, M1 drop update & fail M1, c2 mkdir d1/dir ========================================================== 00:53:26 (1782363206) fail_loc=0x1701 fail_loc=0 Failing mds1 on oleg408-server Stopping /mnt/lustre-mds1 (opts:) on oleg408-server 00:53:30 (1782363210) shut down facet: mds1 facet_host: oleg408-server facet_failover_host: oleg408-server Failover mds1 to oleg408-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Started lustre-MDT0000 00:53:44 (1782363224) targets are mounted 00:53:44 (1782363224) facet_failover done oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1928 1269376 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1820 1269484 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1560 3605460 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3104 7210936 1% /mnt/lustre total: 2 open/close in 0.03 seconds: 66.74 ops/second total: 2 open/close in 0.02 seconds: 127.52 ops/second Failing mds1 on oleg408-server Stopping /mnt/lustre-mds1 (opts:) on oleg408-server 00:53:55 (1782363235) shut down facet: mds1 facet_host: oleg408-server facet_failover_host: oleg408-server Failover mds1 to oleg408-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Started lustre-MDT0000 00:54:11 (1782363251) targets are mounted 00:54:11 (1782363251) facet_failover done oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 22c (53s) == replay-dual test 22d: c1 lfs mkdir -i 1 d1, M1 drop update & fail M0/M1,c2 mkdir d1/dir ========================================================== 00:54:19 (1782363259) fail_loc=0x1701 fail_loc=0 Failing mds1 on oleg408-server Stopping /mnt/lustre-mds1 (opts:) on oleg408-server Failing mds2 on oleg408-server Stopping /mnt/lustre-mds2 (opts:) on oleg408-server 00:54:32 (1782363272) shut down facet: mds1 facet_host: oleg408-server facet_failover_host: oleg408-server facet: mds2 facet_host: oleg408-server facet_failover_host: oleg408-server Failover mds1 to oleg408-server mount facets: mds1 Failover mds2 to oleg408-server mount facets: mds2 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Started lustre-MDT0001 Started lustre-MDT0000 00:54:53 (1782363293) targets are mounted 00:54:53 (1782363293) facet_failover done oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1892 1269412 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1772 1269532 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1560 3605460 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3104 7210936 1% /mnt/lustre total: 2 open/close in 0.03 seconds: 68.04 ops/second total: 2 open/close in 0.01 seconds: 149.82 ops/second Failing mds1 on oleg408-server Stopping /mnt/lustre-mds1 (opts:) on oleg408-server 00:55:03 (1782363303) shut down facet: mds1 facet_host: oleg408-server facet_failover_host: oleg408-server Failover mds1 to oleg408-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Started lustre-MDT0000 00:55:19 (1782363319) targets are mounted 00:55:19 (1782363319) facet_failover done oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 22d (68s) == replay-dual test 23a: c1 rmdir d1, M1 drop reply and fail, client2 mkdir d1 ========================================================== 00:55:27 (1782363327) fail_loc=0x119 fail_loc=0 Failing mds2 on oleg408-server Stopping /mnt/lustre-mds2 (opts:) on oleg408-server 00:55:37 (1782363337) shut down facet: mds2 facet_host: oleg408-server facet_failover_host: oleg408-server Failover mds2 to oleg408-server mount facets: mds2 Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Started lustre-MDT0001 00:55:51 (1782363351) targets are mounted 00:55:51 (1782363351) facet_failover done oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1924 1269380 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1804 1269500 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1560 3605460 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3104 7210936 1% /mnt/lustre total: 2 open/close in 0.02 seconds: 98.06 ops/second Failing mds1 on oleg408-server Stopping /mnt/lustre-mds1 (opts:) on oleg408-server 00:56:02 (1782363362) shut down facet: mds1 facet_host: oleg408-server facet_failover_host: oleg408-server Failover mds1 to oleg408-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Started lustre-MDT0000 00:56:17 (1782363377) targets are mounted 00:56:17 (1782363377) facet_failover done oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 23a (79s) == replay-dual test 23b: c1 rmdir d1, M1 drop reply and fail M0/M1, c2 mkdir d1 ========================================================== 00:56:46 (1782363406) fail_loc=0x119 fail_loc=0 Failing mds1 on oleg408-server Stopping /mnt/lustre-mds1 (opts:) on oleg408-server Failing mds2 on oleg408-server Stopping /mnt/lustre-mds2 (opts:) on oleg408-server 00:56:57 (1782363417) shut down facet: mds1 facet_host: oleg408-server facet_failover_host: oleg408-server facet: mds2 facet_host: oleg408-server facet_failover_host: oleg408-server Failover mds1 to oleg408-server mount facets: mds1 Failover mds2 to oleg408-server mount facets: mds2 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Started lustre-MDT0001 Started lustre-MDT0000 00:57:17 (1782363437) targets are mounted 00:57:17 (1782363437) facet_failover done oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1924 1269380 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1804 1269500 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1560 3605460 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3104 7210936 1% /mnt/lustre total: 2 open/close in 0.02 seconds: 91.01 ops/second Failing mds1 on oleg408-server Stopping /mnt/lustre-mds1 (opts:) on oleg408-server 00:57:28 (1782363448) shut down facet: mds1 facet_host: oleg408-server facet_failover_host: oleg408-server Failover mds1 to oleg408-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Started lustre-MDT0000 00:57:44 (1782363464) targets are mounted 00:57:44 (1782363464) facet_failover done oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 23b (87s) == replay-dual test 23c: c1 rmdir d1, M0 drop update reply and fail M0, c2 mkdir d1 ========================================================== 00:58:13 (1782363493) fail_loc=0x1701 fail_loc=0 Failing mds1 on oleg408-server Stopping /mnt/lustre-mds1 (opts:) on oleg408-server 00:58:17 (1782363497) shut down facet: mds1 facet_host: oleg408-server facet_failover_host: oleg408-server Failover mds1 to oleg408-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Started lustre-MDT0000 00:58:31 (1782363511) targets are mounted 00:58:31 (1782363511) facet_failover done oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1924 1269380 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1816 1269488 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1560 3605460 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3104 7210936 1% /mnt/lustre total: 2 open/close in 0.02 seconds: 102.75 ops/second Failing mds1 on oleg408-server Stopping /mnt/lustre-mds1 (opts:) on oleg408-server 00:58:42 (1782363522) shut down facet: mds1 facet_host: oleg408-server facet_failover_host: oleg408-server Failover mds1 to oleg408-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Started lustre-MDT0000 00:58:58 (1782363538) targets are mounted 00:58:58 (1782363538) facet_failover done oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 23c (74s) == replay-dual test 23d: c1 rmdir d1, M0 drop update reply and fail M0/M1, c2 mkdir d1 ========================================================== 00:59:27 (1782363567) fail_loc=0x1701 fail_loc=0 Failing mds1 on oleg408-server Stopping /mnt/lustre-mds1 (opts:) on oleg408-server Failing mds2 on oleg408-server Stopping /mnt/lustre-mds2 (opts:) on oleg408-server 00:59:41 (1782363581) shut down facet: mds1 facet_host: oleg408-server facet_failover_host: oleg408-server facet: mds2 facet_host: oleg408-server facet_failover_host: oleg408-server Failover mds1 to oleg408-server mount facets: mds1 Failover mds2 to oleg408-server mount facets: mds2 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Started lustre-MDT0001 Started lustre-MDT0000 01:00:04 (1782363604) targets are mounted 01:00:04 (1782363604) facet_failover done oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 1888 1269416 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1768 1269536 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1560 3605460 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3104 7210936 1% /mnt/lustre total: 2 open/close in 0.02 seconds: 101.48 ops/second Failing mds1 on oleg408-server Stopping /mnt/lustre-mds1 (opts:) on oleg408-server 01:00:25 (1782363625) shut down facet: mds1 facet_host: oleg408-server facet_failover_host: oleg408-server Failover mds1 to oleg408-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Started lustre-MDT0000 01:00:41 (1782363641) targets are mounted 01:00:41 (1782363641) facet_failover done oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec PASS 23d (103s) == replay-dual test 24: reconstruct on non-existing object ========================================================== 01:01:10 (1782363670) fail_loc=0x119 fail_loc=0 truncate: cannot truncate '/mnt/lustre/f24.replay-dual' to length 100: No such file or directory PASS 24 (88s) == replay-dual test 25: replay|resend ==================== 01:02:38 (1782363758) 1+0 records in 1+0 records out 512 bytes copied, 0.0029897 s, 171 kB/s fail_loc=0x304 fail_loc=0x80000325 Failing ost1 on oleg408-server Stopping /mnt/lustre-ost1 (opts:) on oleg408-server 01:02:41 (1782363761) shut down facet: ost1 facet_host: oleg408-server facet_failover_host: oleg408-server Failover ost1 to oleg408-server mount facets: ost1 Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Started lustre-OST0000 01:02:55 (1782363775) targets are mounted 01:02:55 (1782363775) facet_failover done oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 6933: 64036 Terminated LUSTRE="/home/green/git/lustre-release/lustre" bash -c "multiop /mnt/lustre2/f25.replay-dual Ow512" fail_loc=0 PASS 25 (24s) == replay-dual test 26: dbench and tar with mds failover ========================================================== 01:03:02 (1782363782) Starting client oleg408-client.virtnet: -o user_xattr,flock 192.168.204.108@tcp:/lustre /mnt/lustre Started clients oleg408-client.virtnet: 192.168.204.108@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) Started tar loop with pid 65471 Started dbench loop with 65472 striped dir -i0 -c2 -H all_char /mnt/lustre2/d26.replay-dual/run_dbench striped dir -i0 -c2 -H all_char /mnt/lustre/d26.replay-dual/run_tar looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Thu Jun 25 01:03:04 EDT 2026 waiting for dbench pid 65513 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs failed to create barrier semaphore 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2100 1269204 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1964 1269340 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 26140 3579544 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1544 3605476 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 27684 7185020 1% /mnt/lustre 1 128 5.76 MB/sec warmup 1 sec latency 44.133 ms 1 231 4.44 MB/sec warmup 2 sec latency 150.076 ms 1 363 4.33 MB/sec warmup 3 sec latency 49.501 ms 1 477 4.17 MB/sec warmup 4 sec latency 50.943 ms 1 648 4.21 MB/sec warmup 5 sec latency 51.400 ms test_26 fail mds1 1 times Failing mds1 on oleg408-server 1 759 3.56 MB/sec warmup 6 sec latency 40.219 ms Stopping /mnt/lustre-mds1 (opts:) on oleg408-server 1 863 3.08 MB/sec warmup 7 sec latency 65.184 ms 01:03:11 (1782363791) shut down facet: mds1 facet_host: oleg408-server facet_failover_host: oleg408-server 1 902 2.71 MB/sec warmup 8 sec latency 511.735 ms 1 995 2.43 MB/sec warmup 9 sec latency 544.185 ms 1 995 2.19 MB/sec warmup 10 sec latency 1544.394 ms 1 995 1.99 MB/sec warmup 11 sec latency 2544.552 ms 1 995 1.83 MB/sec warmup 12 sec latency 3544.679 ms 1 995 1.68 MB/sec warmup 13 sec latency 4544.846 ms 1 995 1.56 MB/sec warmup 14 sec latency 5545.006 ms 1 995 1.46 MB/sec warmup 15 sec latency 6545.147 ms 1 995 1.37 MB/sec warmup 16 sec latency 7545.300 ms 1 995 1.29 MB/sec warmup 17 sec latency 8545.451 ms Failover mds1 to oleg408-server mount facets: mds1 1 995 1.22 MB/sec warmup 18 sec latency 9545.602 ms 1 995 1.15 MB/sec warmup 19 sec latency 10546.989 ms Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 1 995 0.00 MB/sec execute 1 sec latency 12547.271 ms 1 995 0.00 MB/sec execute 2 sec latency 13547.438 ms oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all 1 995 0.00 MB/sec execute 3 sec latency 14547.558 ms pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Started lustre-MDT0000 01:03:27 (1782363807) targets are mounted 01:03:27 (1782363807) facet_failover done 1 995 0.00 MB/sec execute 4 sec latency 15547.826 ms 1 995 0.00 MB/sec execute 5 sec latency 16547.953 ms 1 995 0.00 MB/sec execute 6 sec latency 17548.092 ms 1 995 0.00 MB/sec execute 7 sec latency 18548.223 ms 1 995 0.00 MB/sec execute 8 sec latency 19548.381 ms 1 995 0.00 MB/sec execute 9 sec latency 20548.604 ms 1 995 0.00 MB/sec execute 10 sec latency 21548.752 ms 1 995 0.00 MB/sec execute 11 sec latency 22548.897 ms 1 995 0.00 MB/sec execute 12 sec latency 23549.032 ms 1 995 0.00 MB/sec execute 13 sec latency 24549.173 ms 1 995 0.00 MB/sec execute 14 sec latency 25549.338 ms 1 995 0.00 MB/sec execute 15 sec latency 26549.587 ms 1 995 0.00 MB/sec execute 16 sec latency 27549.764 ms 1 995 0.00 MB/sec execute 17 sec latency 28549.949 ms 1 995 0.00 MB/sec execute 18 sec latency 29550.125 ms 1 995 0.00 MB/sec execute 19 sec latency 30550.272 ms 1 995 0.00 MB/sec execute 20 sec latency 31550.429 ms 1 995 0.00 MB/sec execute 21 sec latency 32550.575 ms 1 995 0.00 MB/sec execute 22 sec latency 33550.810 ms 1 995 0.00 MB/sec execute 23 sec latency 34550.980 ms 1 995 0.00 MB/sec execute 24 sec latency 35551.148 ms 1 995 0.00 MB/sec execute 25 sec latency 36551.307 ms 1 995 0.00 MB/sec execute 26 sec latency 37551.457 ms 1 995 0.00 MB/sec execute 27 sec latency 38551.622 ms 1 995 0.00 MB/sec execute 28 sec latency 39551.851 ms 1 995 0.00 MB/sec execute 29 sec latency 40552.043 ms 1 995 0.00 MB/sec execute 30 sec latency 41552.200 ms 1 995 0.00 MB/sec execute 31 sec latency 42552.405 ms 1 995 0.00 MB/sec execute 32 sec latency 43552.618 ms 1 995 0.00 MB/sec execute 33 sec latency 44552.840 ms 1 995 0.00 MB/sec execute 34 sec latency 45553.038 ms 1 995 0.00 MB/sec execute 35 sec latency 46553.228 ms 1 995 0.00 MB/sec execute 36 sec latency 47553.478 ms 1 995 0.00 MB/sec execute 37 sec latency 48553.697 ms 1 995 0.00 MB/sec execute 38 sec latency 49553.952 ms 1 995 0.00 MB/sec execute 39 sec latency 50554.218 ms 1 995 0.00 MB/sec execute 40 sec latency 51554.447 ms 1 995 0.00 MB/sec execute 41 sec latency 52554.683 ms 1 995 0.00 MB/sec execute 42 sec latency 53554.918 ms 1 995 0.00 MB/sec execute 43 sec latency 54555.148 ms 1 995 0.00 MB/sec execute 44 sec latency 55555.377 ms 1 995 0.00 MB/sec execute 45 sec latency 56555.689 ms 1 995 0.00 MB/sec execute 46 sec latency 57556.049 ms 1 995 0.00 MB/sec execute 47 sec latency 58556.257 ms 1 995 0.00 MB/sec execute 48 sec latency 59556.481 ms 1 995 0.00 MB/sec execute 49 sec latency 60556.880 ms 1 995 0.00 MB/sec execute 50 sec latency 61557.199 ms 1 995 0.00 MB/sec execute 51 sec latency 62557.546 ms 1 995 0.00 MB/sec execute 52 sec latency 63557.829 ms 1 995 0.00 MB/sec execute 53 sec latency 64558.011 ms 1 995 0.00 MB/sec execute 54 sec latency 65558.216 ms 1 995 0.00 MB/sec execute 55 sec latency 66558.444 ms 1 995 0.00 MB/sec execute 56 sec latency 67558.721 ms 1 995 0.00 MB/sec execute 57 sec latency 68559.012 ms 1 995 0.00 MB/sec execute 58 sec latency 69559.245 ms 1 995 0.00 MB/sec execute 59 sec latency 70559.468 ms 1 995 0.00 MB/sec execute 60 sec latency 71559.682 ms 1 995 0.00 MB/sec execute 61 sec latency 72559.850 ms 1 995 0.00 MB/sec execute 62 sec latency 73560.118 ms 1 995 0.00 MB/sec execute 63 sec latency 74560.352 ms 1 995 0.00 MB/sec execute 64 sec latency 75560.659 ms 1 995 0.00 MB/sec execute 65 sec latency 76560.966 ms 1 995 0.00 MB/sec execute 66 sec latency 77561.224 ms 1 995 0.00 MB/sec execute 67 sec latency 78561.490 ms 1 995 0.00 MB/sec execute 68 sec latency 79561.853 ms 1 995 0.00 MB/sec execute 69 sec latency 80562.036 ms 1 995 0.00 MB/sec execute 70 sec latency 81562.249 ms 1 995 0.00 MB/sec execute 71 sec latency 82562.507 ms 1 995 0.00 MB/sec execute 72 sec latency 83562.742 ms 1 995 0.00 MB/sec execute 73 sec latency 84563.031 ms 1 995 0.00 MB/sec execute 74 sec latency 85563.298 ms 1 995 0.00 MB/sec execute 75 sec latency 86563.528 ms 1 995 0.00 MB/sec execute 76 sec latency 87563.823 ms 1 995 0.00 MB/sec execute 77 sec latency 88564.063 ms 1 995 0.00 MB/sec execute 78 sec latency 89564.330 ms 1 995 0.00 MB/sec execute 79 sec latency 90564.562 ms 1 995 0.00 MB/sec execute 80 sec latency 91564.822 ms 1 995 0.00 MB/sec execute 81 sec latency 92565.034 ms 1 995 0.00 MB/sec execute 82 sec latency 93565.307 ms 1 995 0.00 MB/sec execute 83 sec latency 94565.579 ms 1 995 0.00 MB/sec execute 84 sec latency 95565.900 ms 1 995 0.00 MB/sec execute 85 sec latency 96566.168 ms 1 995 0.00 MB/sec execute 86 sec latency 97566.395 ms 1 995 0.00 MB/sec execute 87 sec latency 98566.623 ms 1 995 0.00 MB/sec execute 88 sec latency 99566.890 ms 1 995 0.00 MB/sec execute 89 sec latency 100567.122 ms 1 995 0.00 MB/sec execute 90 sec latency 101567.330 ms 1 995 0.00 MB/sec execute 91 sec latency 102567.567 ms 1 995 0.00 MB/sec execute 92 sec latency 103567.836 ms 1 995 0.00 MB/sec execute 93 sec latency 104568.191 ms 1 995 0.00 MB/sec execute 94 sec latency 105568.405 ms 1 995 0.00 MB/sec execute 95 sec latency 106568.704 ms 1 995 0.00 MB/sec execute 96 sec latency 107568.927 ms 1 995 0.00 MB/sec execute 97 sec latency 108569.191 ms 1 995 0.00 MB/sec execute 98 sec latency 109569.427 ms 1 995 0.00 MB/sec execute 99 sec latency 110569.860 ms 1 cleanup 100 sec 1 cleanup 101 sec 1 cleanup 102 sec 1 cleanup 103 sec 1 cleanup 104 sec 1 cleanup 105 sec 1 cleanup 106 sec 1 cleanup 107 sec 1 cleanup 108 sec 1 cleanup 109 sec 1 cleanup 110 sec 1 cleanup 111 sec 1 cleanup 112 sec 1 cleanup 113 sec 1 cleanup 114 sec 1 cleanup 115 sec 1 cleanup 116 sec 1 cleanup 117 sec 1 cleanup 118 sec 1 cleanup 119 sec 1 cleanup 120 sec 1 cleanup 121 sec 1 cleanup 122 sec 1 cleanup 123 sec 1 cleanup 124 sec 1 cleanup 125 sec 1 cleanup 126 sec 1 cleanup 127 sec 1 cleanup 128 sec 1 cleanup 129 sec 1 cleanup 130 sec 1 cleanup 131 sec 1 cleanup 132 sec 1 cleanup 133 sec 1 cleanup 134 sec 1 cleanup 135 sec 1 cleanup 136 sec 1 cleanup 137 sec 1 cleanup 138 sec 1 cleanup 139 sec 1 cleanup 140 sec 1 cleanup 141 sec 1 cleanup 142 sec 1 cleanup 143 sec 1 cleanup 144 sec 1 cleanup 145 sec 1 cleanup 146 sec 1 cleanup 147 sec 1 cleanup 148 sec 1 cleanup 149 sec 1 cleanup 150 sec 1 cleanup 151 sec 1 cleanup 152 sec 1 cleanup 153 sec 1 cleanup 154 sec 1 cleanup 155 sec 1 cleanup 156 sec 1 cleanup 157 sec 1 cleanup 158 sec 1 cleanup 159 sec 1 cleanup 160 sec 1 cleanup 161 sec 1 cleanup 162 sec 1 cleanup 163 sec 1 cleanup 164 sec 1 cleanup 165 sec 1 cleanup 166 sec 1 cleanup 167 sec 1 cleanup 168 sec 1 cleanup 169 sec 1 cleanup 170 sec 1 cleanup 171 sec 1 cleanup 172 sec 1 cleanup 173 sec 1 cleanup 174 sec 1 cleanup 175 sec 1 cleanup 176 sec 1 cleanup 177 sec 1 cleanup 178 sec 1 cleanup 179 sec 1 cleanup 180 sec 1 cleanup 181 sec 1 cleanup 182 sec 1 cleanup 183 sec tar: etc/udev/rules.d: Cannot mkdir: Input/output error 1 cleanup 184 sec 1 cleanup 185 sec 1 cleanup 186 sec 1 cleanup 187 sec 1 cleanup 188 sec 1 cleanup 189 sec 1 cleanup 190 sec 1 cleanup 191 sec 1 cleanup 192 sec 1 cleanup 193 sec 1 cleanup 194 sec 1 cleanup 195 sec 1 cleanup 196 sec 1 cleanup 197 sec 1 cleanup 198 sec