== replay-dual test 26: dbench and tar with mds failover ========================================================== 00:51:35 (1782363095) Starting client oleg101-client.virtnet: -o user_xattr,flock 192.168.201.101@tcp:/lustre /mnt/lustre Started clients oleg101-client.virtnet: 192.168.201.101@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) Started tar loop with pid 45127 Started dbench loop with 45128 looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Thu Jun 25 00:51:37 EDT 2026 waiting for dbench pid 45161 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs failed to create barrier semaphore 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2176 2206336 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 26624 3719168 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 2048 3743744 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 28672 7462912 1% /mnt/lustre 1 111 5.24 MB/sec warmup 1 sec latency 47.427 ms 1 197 3.92 MB/sec warmup 2 sec latency 75.711 ms 1 294 3.61 MB/sec warmup 3 sec latency 48.539 ms 1 402 3.55 MB/sec warmup 4 sec latency 39.813 ms test_26 fail mds1 1 times Failing mds1 on oleg101-server 1 511 3.55 MB/sec warmup 5 sec latency 41.543 ms Stopping /mnt/lustre-mds1 (opts:) on oleg101-server 1 600 3.49 MB/sec warmup 6 sec latency 529.155 ms 00:51:44 (1782363104) shut down facet: mds1 facet_host: oleg101-server facet_failover_host: oleg101-server 1 600 3.00 MB/sec warmup 7 sec latency 1529.316 ms 1 600 2.62 MB/sec warmup 8 sec latency 2529.555 ms 1 600 2.33 MB/sec warmup 9 sec latency 3529.745 ms 1 600 2.10 MB/sec warmup 10 sec latency 4529.924 ms 1 600 1.91 MB/sec warmup 11 sec latency 5530.118 ms 1 600 1.75 MB/sec warmup 12 sec latency 6530.293 ms 1 600 1.61 MB/sec warmup 13 sec latency 7530.547 ms 1 600 1.50 MB/sec warmup 14 sec latency 8530.741 ms 1 600 1.40 MB/sec warmup 15 sec latency 9530.985 ms 1 600 1.31 MB/sec warmup 16 sec latency 10531.230 ms Failover mds1 to oleg101-server mount facets: mds1 1 600 1.23 MB/sec warmup 17 sec latency 11531.423 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 600 1.16 MB/sec warmup 18 sec latency 12531.590 ms 1 600 1.10 MB/sec warmup 19 sec latency 13531.758 ms oleg101-server: oleg101-server.virtnet: executing set_default_debug -1 all pdsh@oleg101-client: oleg101-server: ssh exited with exit code 1 Started lustre-MDT0000 00:51:58 (1782363118) targets are mounted 00:51:58 (1782363118) facet_failover done 1 600 0.00 MB/sec execute 1 sec latency 15532.058 ms 1 600 0.00 MB/sec execute 2 sec latency 16532.186 ms 1 600 0.00 MB/sec execute 3 sec latency 17532.348 ms 1 600 0.00 MB/sec execute 4 sec latency 18532.497 ms 1 600 0.00 MB/sec execute 5 sec latency 19532.653 ms 1 600 0.00 MB/sec execute 6 sec latency 20532.846 ms 1 600 0.00 MB/sec execute 7 sec latency 21533.036 ms 1 600 0.00 MB/sec execute 8 sec latency 22533.215 ms 1 600 0.00 MB/sec execute 9 sec latency 23533.389 ms 1 600 0.00 MB/sec execute 10 sec latency 24533.619 ms 1 600 0.00 MB/sec execute 11 sec latency 25533.784 ms 1 600 0.00 MB/sec execute 12 sec latency 26533.931 ms 1 600 0.00 MB/sec execute 13 sec latency 27534.094 ms 1 600 0.00 MB/sec execute 14 sec latency 28534.248 ms 1 600 0.00 MB/sec execute 15 sec latency 29534.390 ms 1 600 0.00 MB/sec execute 16 sec latency 30534.526 ms 1 600 0.00 MB/sec execute 17 sec latency 31534.664 ms 1 600 0.00 MB/sec execute 18 sec latency 32534.789 ms 1 600 0.00 MB/sec execute 19 sec latency 33534.918 ms 1 600 0.00 MB/sec execute 20 sec latency 34535.071 ms 1 600 0.00 MB/sec execute 21 sec latency 35535.191 ms 1 600 0.00 MB/sec execute 22 sec latency 36535.308 ms 1 600 0.00 MB/sec execute 23 sec latency 37535.498 ms 1 600 0.00 MB/sec execute 24 sec latency 38535.653 ms 1 600 0.00 MB/sec execute 25 sec latency 39535.786 ms 1 600 0.00 MB/sec execute 26 sec latency 40535.931 ms 1 600 0.00 MB/sec execute 27 sec latency 41536.064 ms 1 600 0.00 MB/sec execute 28 sec latency 42537.327 ms 1 600 0.00 MB/sec execute 29 sec latency 43537.494 ms 1 600 0.00 MB/sec execute 30 sec latency 44537.689 ms 1 600 0.00 MB/sec execute 31 sec latency 45537.828 ms 1 600 0.00 MB/sec execute 32 sec latency 46538.047 ms 1 600 0.00 MB/sec execute 33 sec latency 47539.509 ms 1 600 0.00 MB/sec execute 34 sec latency 48539.657 ms 1 600 0.00 MB/sec execute 35 sec latency 49539.797 ms 1 600 0.00 MB/sec execute 36 sec latency 50540.003 ms 1 600 0.00 MB/sec execute 37 sec latency 51540.211 ms 1 600 0.00 MB/sec execute 38 sec latency 52540.351 ms 1 600 0.00 MB/sec execute 39 sec latency 53540.517 ms 1 600 0.00 MB/sec execute 40 sec latency 54540.674 ms 1 600 0.00 MB/sec execute 41 sec latency 55540.812 ms 1 600 0.00 MB/sec execute 42 sec latency 56540.959 ms 1 600 0.00 MB/sec execute 43 sec latency 57541.116 ms 1 600 0.00 MB/sec execute 44 sec latency 58541.251 ms 1 600 0.00 MB/sec execute 45 sec latency 59541.393 ms 1 600 0.00 MB/sec execute 46 sec latency 60541.517 ms 1 600 0.00 MB/sec execute 47 sec latency 61541.639 ms 1 600 0.00 MB/sec execute 48 sec latency 62541.767 ms 1 600 0.00 MB/sec execute 49 sec latency 63541.901 ms 1 600 0.00 MB/sec execute 50 sec latency 64542.041 ms 1 600 0.00 MB/sec execute 51 sec latency 65542.174 ms 1 600 0.00 MB/sec execute 52 sec latency 66542.292 ms 1 600 0.00 MB/sec execute 53 sec latency 67542.434 ms 1 600 0.00 MB/sec execute 54 sec latency 68542.681 ms 1 600 0.00 MB/sec execute 55 sec latency 69542.861 ms 1 600 0.00 MB/sec execute 56 sec latency 70542.994 ms 1 600 0.00 MB/sec execute 57 sec latency 71543.133 ms 1 600 0.00 MB/sec execute 58 sec latency 72543.277 ms 1 600 0.00 MB/sec execute 59 sec latency 73543.404 ms 1 600 0.00 MB/sec execute 60 sec latency 74543.556 ms 1 600 0.00 MB/sec execute 61 sec latency 75543.789 ms 1 600 0.00 MB/sec execute 62 sec latency 76543.975 ms 1 600 0.00 MB/sec execute 63 sec latency 77544.147 ms 1 600 0.00 MB/sec execute 64 sec latency 78544.304 ms 1 600 0.00 MB/sec execute 65 sec latency 79544.513 ms 1 600 0.00 MB/sec execute 66 sec latency 80544.722 ms 1 600 0.00 MB/sec execute 67 sec latency 81544.903 ms 1 600 0.00 MB/sec execute 68 sec latency 82546.160 ms 1 600 0.00 MB/sec execute 69 sec latency 83546.310 ms 1 600 0.00 MB/sec execute 70 sec latency 84546.451 ms 1 600 0.00 MB/sec execute 71 sec latency 85546.641 ms 1 600 0.00 MB/sec execute 72 sec latency 86546.801 ms 1 600 0.00 MB/sec execute 73 sec latency 87546.949 ms 1 600 0.00 MB/sec execute 74 sec latency 88547.071 ms 1 600 0.00 MB/sec execute 75 sec latency 89547.244 ms 1 600 0.00 MB/sec execute 76 sec latency 90547.390 ms 1 600 0.00 MB/sec execute 77 sec latency 91547.560 ms 1 600 0.00 MB/sec execute 78 sec latency 92547.842 ms 1 600 0.00 MB/sec execute 79 sec latency 93548.068 ms 1 600 0.00 MB/sec execute 80 sec latency 94548.333 ms 1 600 0.00 MB/sec execute 81 sec latency 95548.545 ms 1 600 0.00 MB/sec execute 82 sec latency 96548.750 ms 1 600 0.00 MB/sec execute 83 sec latency 97548.919 ms 1 600 0.00 MB/sec execute 84 sec latency 98549.107 ms 1 600 0.00 MB/sec execute 85 sec latency 99549.281 ms 1 600 0.00 MB/sec execute 86 sec latency 100549.508 ms 1 600 0.00 MB/sec execute 87 sec latency 101549.677 ms 1 600 0.00 MB/sec execute 88 sec latency 102549.860 ms 1 600 0.00 MB/sec execute 89 sec latency 103550.052 ms 1 600 0.00 MB/sec execute 90 sec latency 104550.213 ms 1 600 0.00 MB/sec execute 91 sec latency 105550.370 ms 1 600 0.00 MB/sec execute 92 sec latency 106550.540 ms 1 600 0.00 MB/sec execute 93 sec latency 107550.728 ms 1 600 0.00 MB/sec execute 94 sec latency 108550.922 ms 1 600 0.00 MB/sec execute 95 sec latency 109551.089 ms 1 600 0.00 MB/sec execute 96 sec latency 110551.261 ms 1 600 0.00 MB/sec execute 97 sec latency 111551.434 ms 1 600 0.00 MB/sec execute 98 sec latency 112551.621 ms 1 600 0.00 MB/sec execute 99 sec latency 113551.796 ms 1 cleanup 100 sec 1 cleanup 101 sec 1 cleanup 102 sec 1 cleanup 103 sec 1 cleanup 104 sec 1 cleanup 105 sec 1 cleanup 106 sec 1 cleanup 107 sec 1 cleanup 108 sec 1 cleanup 109 sec 1 cleanup 110 sec 1 cleanup 111 sec 1 cleanup 112 sec 1 cleanup 113 sec 1 cleanup 114 sec 1 cleanup 115 sec 1 cleanup 116 sec 1 cleanup 117 sec 1 cleanup 118 sec 1 cleanup 119 sec 1 cleanup 120 sec 1 cleanup 121 sec 1 cleanup 122 sec 1 cleanup 123 sec 1 cleanup 124 sec 1 cleanup 125 sec 1 cleanup 126 sec 1 cleanup 127 sec 1 cleanup 128 sec 1 cleanup 129 sec 1 cleanup 130 sec 1 cleanup 131 sec 1 cleanup 132 sec 1 cleanup 133 sec 1 cleanup 134 sec 1 cleanup 135 sec 1 cleanup 136 sec 1 cleanup 137 sec 1 cleanup 138 sec 1 cleanup 139 sec 1 cleanup 140 sec 1 cleanup 141 sec 1 cleanup 142 sec 1 cleanup 143 sec 1 cleanup 144 sec 1 cleanup 145 sec 1 cleanup 146 sec 1 cleanup 147 sec 1 cleanup 148 sec 1 cleanup 149 sec 1 cleanup 150 sec 1 cleanup 151 sec 1 cleanup 152 sec 1 cleanup 153 sec 1 cleanup 154 sec 1 cleanup 155 sec 1 cleanup 156 sec 1 cleanup 157 sec 1 cleanup 158 sec 1 cleanup 159 sec 1 cleanup 160 sec 1 cleanup 161 sec 1 cleanup 162 sec 1 cleanup 163 sec 1 cleanup 164 sec 1 cleanup 165 sec 1 cleanup 166 sec 1 cleanup 167 sec 1 cleanup 168 sec 1 cleanup 169 sec 1 cleanup 170 sec 1 cleanup 171 sec 1 cleanup 172 sec 1 cleanup 173 sec 1 cleanup 174 sec 1 cleanup 175 sec 1 cleanup 176 sec 1 cleanup 177 sec 1 cleanup 178 sec 1 cleanup 179 sec 1 cleanup 180 sec 1 cleanup 181 sec 1 cleanup 182 sec 1 cleanup 183 sec 1 cleanup 184 sec 1 cleanup 185 sec 1 cleanup 186 sec 1 cleanup 187 sec 1 cleanup 188 sec 1 cleanup 189 sec 1 cleanup 190 sec 1 cleanup 191 sec 1 cleanup 192 sec 1 cleanup 193 sec 1 cleanup 194 sec 1 cleanup 195 sec 1 cleanup 196 sec 1 cleanup 197 sec 1 cleanup 198 sec 1 cleanup 199 sec 1 cleanup 200 sec tar: etc/udev/hwdb.bin: Cannot utime: Input/output error oleg101-client: error: invalid path '/mnt/lustre': Input/output error pdsh@oleg101-client: oleg101-client: ssh exited with exit code 5 tar: etc/udev/hwdb.bin: Cannot change ownership to uid 0, gid 0: Cannot send after transport endpoint shutdown tar: etc/udev/hwdb.bin: Cannot change mode to r--r--r--: Cannot send after transport endpoint shutdown tar: etc/udev/rules.d: Cannot mkdir: Input/output error tar: etc/udev/rules.d/99-lustre-test.rules: Cannot open: Cannot send after transport endpoint shutdown tar: etc/udev/hwdb.d: Cannot mkdir: Input/output error tar: etc/udev: Cannot utime: Cannot send after transport endpoint shutdown 0 cleanup 201 sec Operation Count AvgLat MaxLat ---------------------------------------- Close 1 215185.422 215185.422 Throughput 0 MB/sec 1 clients 1 procs max_latency=113551.796 ms tar: etc/udev: Cannot change ownership to uid 0, gid 0: Cannot send after transport endpoint shutdown tar: etc/udev: Cannot change mode to rwxr-xr-x: Cannot send after transport endpoint shutdown tar: etc/tuned: Cannot mkdir: Cannot send after transport endpoint shutdown tar: etc/tuned/profile_mode: Cannot open: Cannot send after transport endpoint shutdown tar: etc/tuned/post_loaded_profile: Cannot open: Cannot send after transport endpoint shutdown tar: etc/tuned/bootcmdline: Cannot open: Cannot send after transport endpoint shutdown tar: etc/tuned/tuned-main.conf: Cannot open: Cannot send after transport endpoint shutdown tar: etc/tuned/active_profile: Cannot open: Cannot send after transport endpoint shutdown stopping dbench on /mnt/lustre at Thu Jun 25 00:55:18 EDT 2026 with return code 0 tar: etc/tuned/recommend.d: Cannot mkdir: Cannot send after transport endpoint shutdown tar: etc/tmpfiles.d: Cannot mkdir: Cannot send after transport endpoint shutdown tar: etc/terminfo: Cannot mkdir: Cannot send after transport endpoint shutdown tar: etc/systemd: Cannot mkdir: Cannot send after transport endpoint shutdown tar: etc/systemd/user.conf: Cannot open: Cannot send after transport endpoint shutdown tar: etc/systemd/resolved.conf: Cannot open: Cannot send after transport endpoint shutdown tar: etc/systemd/system.conf: Cannot open: Cannot send after transport endpoint shutdown tar: etc/systemd/journald.conf: Cannot open: Cannot send after transport endpoint shutdown tar: etc/systemd/coredump.conf: Cannot open: Cannot send after transport endpoint shutdown tar: etc/systemd/logind.conf: Cannot open: Cannot send after transport endpoint shutdown tar: etc/systemd/pstore.conf: Cannot open: Cannot send after transport endpoint shutdown tar: etc/systemd/user: Cannot mkdir: Cannot send after transport endpoint shutdown tar: etc/systemd/system: Cannot mkdir: Cannot send after transport endpoint shutdown tar: etc/systemd/system/proc-sys-fs-binfmt_misc.mount: Cannot open: Cannot send after transport endpoint shutdown tar: etc/systemd/system/timers.target.wants: Cannot mkdir: Cannot send after transport endpoint shutdown tar: etc/systemd/system/timers.target.wants/unbound-anchor.timer: Cannot open: Cannot send after transport endpoint shutdown tar: etc/systemd/system/sysinit.target.wants: Cannot mkdir: Cannot send after transport endpoint shutdown tar: etc/systemd/system/sysinit.target.wants/selinux-autorelabel-mark.service: Cannot open: Cannot send after transport endpoint shutdown tar: etc/systemd/system/sysinit.target.wants/rngd.service: Cannot open: Cannot send after transport endpoint shutdown tar: etc/systemd/system/sysinit.target.wants/nis-domainname.service: Cannot open: Cannot send after transport endpoint shutdown tar: etc/systemd/system/sysinit.target.wants/loadmodules.service: Cannot open: Cannot send after transport endpoint shutdown tar: etc/systemd/system/sysinit.target.wants/import-state.service: Cannot open: Cannot send after transport endpoint shutdown tar: etc/systemd/system/sockets.target.wants: Cannot mkdir: Cannot send after transport endpoint shutdown tar: etc/systemd/system/sockets.target.wants/rpcbind.socket: Cannot open: Cannot send after transport endpoint shutdown tar: etc/systemd/system/rpc-statd.service.requires: Cannot mkdir: Cannot send after transport endpoint shutdown tar: etc/systemd/system/rpc-statd.service.requires/nfs-convert.service: Cannot open: Cannot send after transport endpoint shutdown tar: etc/systemd/system/rpc-statd-notify.service.requires: Cannot mkdir: Cannot send after transport endpoint shutdown tar: etc/systemd/system/rpc-statd-notify.service.requires/nfs-convert.service: Cannot open: Cannot send after transport endpoint shutdown tar: etc/systemd/system/rpc-gssd.service.requires: Cannot mkdir: Cannot send after transport endpoint shutdown tar: etc/systemd/system/rpc-gssd.service.requires/nfs-convert.service: Cannot open: Cannot send after transport endpoint shutdown tar: etc/systemd/system/remote-fs.target.wants: Cannot mkdir: Cannot send after transport endpoint shutdown tar: etc/systemd/system/remote-fs.target.wants/nfs-client.target: Cannot open: Cannot send after transport endpoint shutdown tar: etc/systemd/system/nfs-server.service.requires: Cannot mkdir: Cannot send after transport endpoint shutdown tar: etc/systemd/system/nfs-server.service.requires/nfs-convert.service: Cannot open: Cannot send after transport endpoint shutdown tar: etc/systemd/system/nfs-mountd.service.requires: Cannot mkdir: Cannot send after transport endpoint shutdown tar: etc/systemd/system/nfs-mountd.service.requires/nfs-convert.service: Cannot open: Cannot send after transport endpoint shutdown tar: etc/systemd/system/nfs-idmapd.service.requires: Cannot mkdir: Cannot send after transport endpoint shutdown tar: etc/systemd/system/nfs-idmapd.service.requires/nfs-convert.service: Cannot open: Cannot send after transport endpoint shutdown tar: etc/systemd/system/nfs-blkmap.service.requires: Cannot mkdir: Cannot send after transport endpoint shutdown tar: etc/systemd/system/nfs-blkmap.service.requires/nfs-convert.service: Cannot open: Cannot send after transport endpoint shutdown tar: etc/systemd/system/network-online.target.wants: Cannot mkdir: Cannot send after transport endpoint shutdown tar: etc/systemd/system/network-online.target.wants/NetworkManager-wait-online.service: Cannot open: Cannot send after transport endpoint shutdown tar: etc/systemd/system/multi-user.target.wants: Cannot mkdir: Cannot send after transport endpoint shutdown tar: etc/systemd/system/multi-user.target.wants/tuned.service: Cannot open: Cannot send after transport endpoint shutdown tar: etc/systemd/system/multi-user.target.wants/sshd.service: Cannot open: Cannot send after transport endpoint shutdown tar: etc/systemd/system/multi-user.target.wants/rsyslog.service: Cannot open: Cannot send after transport endpoint shutdown tar: etc/systemd/system/multi-user.target.wants/rpcbind.service: Cannot open: Cannot send after transport endpoint shutdown tar: etc/systemd/system/multi-user.target.wants/remote-fs.target: Cannot open: Cannot send after transport endpoint shutdown tar: etc/systemd/system/multi-user.target.wants/nfs-client.target: Cannot open: Cannot send after transport endpoint shutdown tar: etc/systemd/system/multi-user.target.wants/kdump.service: Cannot open: Cannot send after transport endpoint shutdown tar: etc/systemd/system/multi-user.target.wants/irqbalance.service: Cannot open: Cannot send after transport endpoint shutdown clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt oleg101-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid '/usr/share/dbench/client.txt' -> 'client.txt' mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Thu Jun 25 00:55:20 EDT 2026 waiting for dbench pid 46023 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 71 3.14 MB/sec warmup 1 sec latency 43.639 ms tar: Unexpected EOF in archive tar: Unexpected EOF in archive 1 150 3.14 MB/sec warmup 2 sec latency 62.630 ms tar: etc/xdg/systemd: Cannot stat: No such file or directory tar: etc/xdg: Cannot stat: No such file or directory tar: etc/yum: Cannot stat: No such file or directory tar: Error is not recoverable: exiting now dbench killed by signal 15 stopping dbench on /mnt/lustre at Thu Jun 25 00:55:23 EDT 2026 with return code 0 46023 pts/0 S+ 0:00 dbench -c client.txt 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100 killed dbench main pid 46023 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished