== replay-dual test 26: dbench and tar with mds failover ========================================================== 20:08:46 (1788653326) Starting client oleg229-client.virtnet: -o user_xattr,flock 192.168.202.129@tcp:/lustre /mnt/lustre Started clients oleg229-client.virtnet: 192.168.202.129@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Started tar loop with pid 45262 Started dbench loop with 45263 looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2048 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3762176 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 14336 3730432 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 16384 7492608 1% /mnt/lustre '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Sat Sep 5 20:08:51 EDT 2026 waiting for dbench pid 45317 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs failed to create barrier semaphore 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 27 0.85 MB/sec warmup 1 sec latency 29.423 ms 1 71 1.57 MB/sec warmup 2 sec latency 163.399 ms 1 117 1.77 MB/sec warmup 3 sec latency 209.528 ms 1 147 1.57 MB/sec warmup 4 sec latency 177.619 ms 1 184 1.48 MB/sec warmup 5 sec latency 118.698 ms test_26 fail mds1 1 times 1 247 1.57 MB/sec warmup 6 sec latency 73.400 ms 1 287 1.53 MB/sec warmup 7 sec latency 150.462 ms Failing mds1 on oleg229-server 1 315 1.43 MB/sec warmup 8 sec latency 144.983 ms Stopping /mnt/lustre-mds1 (opts:) on oleg229-server 1 349 1.39 MB/sec warmup 9 sec latency 123.416 ms 1 365 1.30 MB/sec warmup 10 sec latency 698.981 ms 1 365 1.18 MB/sec warmup 11 sec latency 1699.130 ms 1 365 1.09 MB/sec warmup 12 sec latency 2699.697 ms 20:09:04 (1788653344) shut down facet: mds1 facet_host: oleg229-server facet_failover_host: oleg229-server 1 365 1.00 MB/sec warmup 13 sec latency 3699.909 ms 1 365 0.93 MB/sec warmup 14 sec latency 4700.105 ms 1 365 0.87 MB/sec warmup 15 sec latency 5700.308 ms 1 365 0.81 MB/sec warmup 16 sec latency 6700.821 ms 1 365 0.77 MB/sec warmup 17 sec latency 7702.097 ms 1 365 0.72 MB/sec warmup 18 sec latency 8702.314 ms 1 365 0.69 MB/sec warmup 19 sec latency 9704.133 ms 1 365 0.00 MB/sec execute 1 sec latency 11704.886 ms 1 365 0.00 MB/sec execute 2 sec latency 12705.087 ms Failover mds1 to oleg229-server mount facets: mds1 1 365 0.00 MB/sec execute 3 sec latency 13705.748 ms 1 365 0.00 MB/sec execute 4 sec latency 14705.925 ms 1 365 0.00 MB/sec execute 5 sec latency 15706.491 ms 1 365 0.00 MB/sec execute 6 sec latency 16708.569 ms 1 365 0.00 MB/sec execute 7 sec latency 17708.823 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 365 0.00 MB/sec execute 8 sec latency 18711.695 ms 1 365 0.00 MB/sec execute 9 sec latency 19717.323 ms 1 365 0.00 MB/sec execute 10 sec latency 20730.195 ms 1 365 0.00 MB/sec execute 11 sec latency 21730.387 ms 1 365 0.00 MB/sec execute 12 sec latency 22731.519 ms 1 365 0.00 MB/sec execute 13 sec latency 23732.102 ms 1 365 0.00 MB/sec execute 14 sec latency 24732.381 ms 1 365 0.00 MB/sec execute 15 sec latency 25732.585 ms 1 365 0.00 MB/sec execute 16 sec latency 26732.931 ms 1 365 0.00 MB/sec execute 17 sec latency 27733.684 ms 1 365 0.00 MB/sec execute 18 sec latency 28735.054 ms 1 365 0.00 MB/sec execute 19 sec latency 29735.320 ms 1 365 0.00 MB/sec execute 20 sec latency 30735.586 ms 1 365 0.00 MB/sec execute 21 sec latency 31736.497 ms 1 365 0.00 MB/sec execute 22 sec latency 32736.712 ms 1 365 0.00 MB/sec execute 23 sec latency 33736.905 ms oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all 1 365 0.00 MB/sec execute 24 sec latency 34739.060 ms pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 1 365 0.00 MB/sec execute 25 sec latency 35740.635 ms Started lustre-MDT0000 1 365 0.00 MB/sec execute 26 sec latency 36752.914 ms 20:09:38 (1788653378) targets are mounted 20:09:38 (1788653378) facet_failover done 1 365 0.00 MB/sec execute 27 sec latency 37753.107 ms 1 365 0.00 MB/sec execute 28 sec latency 38753.280 ms 1 365 0.00 MB/sec execute 29 sec latency 39753.506 ms 1 365 0.00 MB/sec execute 30 sec latency 40753.699 ms 1 365 0.00 MB/sec execute 31 sec latency 41753.851 ms 1 365 0.00 MB/sec execute 32 sec latency 42756.048 ms 1 382 0.02 MB/sec execute 33 sec latency 43164.144 ms 1 421 0.05 MB/sec execute 34 sec latency 135.047 ms 1 462 0.09 MB/sec execute 35 sec latency 112.303 ms 1 504 0.12 MB/sec execute 36 sec latency 116.990 ms 1 551 0.16 MB/sec execute 37 sec latency 166.819 ms oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 1 616 0.21 MB/sec execute 38 sec latency 90.774 ms 1 664 0.21 MB/sec execute 39 sec latency 124.900 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 686 0.20 MB/sec execute 40 sec latency 256.133 ms 1 710 0.20 MB/sec execute 41 sec latency 141.488 ms 1 733 0.20 MB/sec execute 42 sec latency 241.373 ms 1 765 0.19 MB/sec execute 43 sec latency 188.712 ms 1 794 0.19 MB/sec execute 44 sec latency 171.889 ms 1 821 0.19 MB/sec execute 45 sec latency 128.089 ms 1 873 0.19 MB/sec execute 46 sec latency 219.540 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2304 2206208 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 11264 3695616 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 40960 3696640 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 52224 7392256 1% /mnt/lustre 1 902 0.18 MB/sec execute 47 sec latency 233.405 ms 1 921 0.18 MB/sec execute 48 sec latency 216.253 ms 1 957 0.18 MB/sec execute 49 sec latency 171.102 ms 1 983 0.18 MB/sec execute 50 sec latency 175.036 ms 1 1025 0.19 MB/sec execute 51 sec latency 165.740 ms 1 1074 0.19 MB/sec execute 52 sec latency 84.620 ms test_26 fail mds1 2 times 1 1127 0.19 MB/sec execute 53 sec latency 85.859 ms 1 1159 0.18 MB/sec execute 54 sec latency 154.167 ms Failing mds1 on oleg229-server 1 1192 0.18 MB/sec execute 55 sec latency 101.620 ms Stopping /mnt/lustre-mds1 (opts:) on oleg229-server 1 1231 0.18 MB/sec execute 56 sec latency 126.214 ms 1 1263 0.18 MB/sec execute 57 sec latency 191.899 ms 1 1263 0.17 MB/sec execute 58 sec latency 1192.733 ms 1 1263 0.17 MB/sec execute 59 sec latency 2194.375 ms 20:10:12 (1788653412) shut down 1 1263 0.17 MB/sec execute 60 sec latency 3194.738 ms facet: mds1 facet_host: oleg229-server facet_failover_host: oleg229-server 1 1263 0.16 MB/sec execute 61 sec latency 4195.377 ms 1 1263 0.16 MB/sec execute 62 sec latency 5196.040 ms 1 1263 0.16 MB/sec execute 63 sec latency 6196.465 ms 1 1263 0.16 MB/sec execute 64 sec latency 7197.828 ms 1 1263 0.15 MB/sec execute 65 sec latency 8199.298 ms 1 1263 0.15 MB/sec execute 66 sec latency 9203.631 ms 1 1263 0.15 MB/sec execute 67 sec latency 10204.048 ms 1 1263 0.15 MB/sec execute 68 sec latency 11204.229 ms 1 1263 0.15 MB/sec execute 69 sec latency 12204.828 ms 1 1263 0.14 MB/sec execute 70 sec latency 13205.156 ms Failover mds1 to oleg229-server mount facets: mds1 1 1263 0.14 MB/sec execute 71 sec latency 14213.040 ms 1 1263 0.14 MB/sec execute 72 sec latency 15213.207 ms 1 1263 0.14 MB/sec execute 73 sec latency 16216.039 ms 1 1263 0.14 MB/sec execute 74 sec latency 17216.365 ms 1 1263 0.13 MB/sec execute 75 sec latency 18216.801 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 1263 0.13 MB/sec execute 76 sec latency 19217.144 ms 1 1263 0.13 MB/sec execute 77 sec latency 20218.010 ms 1 1263 0.13 MB/sec execute 78 sec latency 21218.331 ms 1 1263 0.13 MB/sec execute 79 sec latency 22218.638 ms 1 1263 0.13 MB/sec execute 80 sec latency 23221.019 ms 1 1263 0.12 MB/sec execute 81 sec latency 24221.712 ms 1 1263 0.12 MB/sec execute 82 sec latency 25223.327 ms 1 1263 0.12 MB/sec execute 83 sec latency 26223.629 ms 1 1263 0.12 MB/sec execute 84 sec latency 27223.891 ms 1 1263 0.12 MB/sec execute 85 sec latency 28226.570 ms 1 1263 0.12 MB/sec execute 86 sec latency 29226.838 ms 1 1263 0.12 MB/sec execute 87 sec latency 30228.131 ms 1 1263 0.11 MB/sec execute 88 sec latency 31228.336 ms 1 1263 0.11 MB/sec execute 89 sec latency 32228.978 ms 1 1263 0.11 MB/sec execute 90 sec latency 33229.531 ms 1 1263 0.11 MB/sec execute 91 sec latency 34229.738 ms 1 1263 0.11 MB/sec execute 92 sec latency 35230.019 ms oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all 1 1263 0.11 MB/sec execute 93 sec latency 36230.736 ms pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 1 1263 0.11 MB/sec execute 94 sec latency 37230.988 ms Started lustre-MDT0000 1 1263 0.11 MB/sec execute 95 sec latency 38245.082 ms 20:10:47 (1788653447) targets are mounted 20:10:47 (1788653447) facet_failover done 1 1263 0.10 MB/sec execute 96 sec latency 39246.254 ms 1 1263 0.10 MB/sec execute 97 sec latency 40246.442 ms 1 1263 0.10 MB/sec execute 98 sec latency 41247.976 ms 1 1319 0.11 MB/sec execute 99 sec latency 41446.584 ms 1 cleanup 100 sec 1 cleanup 101 sec 1 cleanup 102 sec 1 cleanup 103 sec oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 0 cleanup 104 sec 0 cleanup 104 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 140 380.262 43164.128 Close 125 13.370 37.987 Rename 10 103.545 171.866 Unlink 17 32.274 78.842 Qpathinfo 95 24.692 134.603 Qfileinfo 50 2.091 17.095 Qfsinfo 33 5.528 48.840 Sfileinfo 34 1282.584 41446.567 Find 47 8.329 71.192 WriteX 237 13.771 71.029 ReadX 223 3.238 85.841 Flush 24 129.298 256.119 Throughput 0.112405 MB/sec 1 clients 1 procs max_latency=43164.144 ms stopping dbench on /mnt/lustre at Sat Sep 5 20:10:56 EDT 2026 with return code 0 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Sat Sep 5 20:11:02 EDT 2026 waiting for dbench pid 47327 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 51 2.10 MB/sec warmup 1 sec latency 102.776 ms 1 97 2.19 MB/sec warmup 2 sec latency 139.413 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2560 2205952 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 27648 3724288 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 47104 3703808 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 74752 7428096 1% /mnt/lustre 1 129 1.92 MB/sec warmup 3 sec latency 239.497 ms 1 155 1.63 MB/sec warmup 4 sec latency 172.085 ms 1 184 1.47 MB/sec warmup 5 sec latency 223.816 ms 1 215 1.39 MB/sec warmup 6 sec latency 167.663 ms 1 250 1.34 MB/sec warmup 7 sec latency 155.900 ms 1 297 1.37 MB/sec warmup 8 sec latency 116.237 ms test_26 fail mds1 3 times 1 334 1.33 MB/sec warmup 9 sec latency 126.177 ms 1 357 1.27 MB/sec warmup 10 sec latency 189.946 ms Failing mds1 on oleg229-server 1 398 1.28 MB/sec warmup 11 sec latency 159.538 ms Stopping /mnt/lustre-mds1 (opts:) on oleg229-server 1 437 1.28 MB/sec warmup 12 sec latency 120.476 ms 1 453 1.22 MB/sec warmup 13 sec latency 679.128 ms 1 453 1.13 MB/sec warmup 14 sec latency 1679.347 ms 1 453 1.06 MB/sec warmup 15 sec latency 2679.491 ms 20:11:17 (1788653477) shut down facet: mds1 facet_host: oleg229-server facet_failover_host: oleg229-server 1 453 0.99 MB/sec warmup 16 sec latency 3680.280 ms 1 453 0.93 MB/sec warmup 17 sec latency 4682.090 ms 1 453 0.88 MB/sec warmup 18 sec latency 5683.216 ms 1 453 0.84 MB/sec warmup 19 sec latency 6683.426 ms 1 453 0.00 MB/sec execute 1 sec latency 8683.782 ms 1 453 0.00 MB/sec execute 2 sec latency 9683.965 ms 1 453 0.00 MB/sec execute 3 sec latency 10684.219 ms 1 453 0.00 MB/sec execute 4 sec latency 11686.256 ms 1 453 0.00 MB/sec execute 5 sec latency 12686.689 ms Failover mds1 to oleg229-server mount facets: mds1 1 453 0.00 MB/sec execute 6 sec latency 13686.836 ms 1 453 0.00 MB/sec execute 7 sec latency 14687.054 ms 1 453 0.00 MB/sec execute 8 sec latency 15687.359 ms 1 453 0.00 MB/sec execute 9 sec latency 16687.533 ms 1 453 0.00 MB/sec execute 10 sec latency 17688.478 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 453 0.00 MB/sec execute 11 sec latency 18688.862 ms 1 453 0.00 MB/sec execute 12 sec latency 19689.153 ms 1 453 0.00 MB/sec execute 13 sec latency 20689.341 ms 1 453 0.00 MB/sec execute 14 sec latency 21689.582 ms 1 453 0.00 MB/sec execute 15 sec latency 22695.573 ms 1 453 0.00 MB/sec execute 16 sec latency 23695.887 ms oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all 1 453 0.00 MB/sec execute 17 sec latency 24696.137 ms 1 453 0.00 MB/sec execute 18 sec latency 25696.348 ms pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 1 453 0.00 MB/sec execute 19 sec latency 26696.551 ms Started lustre-MDT0000 20:11:42 (1788653502) targets are mounted 20:11:42 (1788653502) facet_failover done 1 453 0.00 MB/sec execute 20 sec latency 27696.761 ms 1 453 0.00 MB/sec execute 21 sec latency 28703.195 ms 1 453 0.00 MB/sec execute 22 sec latency 29703.309 ms 1 453 0.00 MB/sec execute 23 sec latency 30703.490 ms 1 453 0.00 MB/sec execute 24 sec latency 31704.155 ms 1 453 0.00 MB/sec execute 25 sec latency 32704.377 ms 1 453 0.00 MB/sec execute 26 sec latency 33705.705 ms 1 453 0.00 MB/sec execute 27 sec latency 34706.255 ms 1 453 0.00 MB/sec execute 28 sec latency 35707.589 ms 1 453 0.00 MB/sec execute 29 sec latency 36711.592 ms 1 453 0.00 MB/sec execute 30 sec latency 37712.199 ms 1 453 0.00 MB/sec execute 31 sec latency 38714.694 ms 1 453 0.00 MB/sec execute 32 sec latency 39715.269 ms 1 461 0.01 MB/sec execute 33 sec latency 40466.213 ms 1 499 0.04 MB/sec execute 34 sec latency 181.450 ms 1 536 0.08 MB/sec execute 35 sec latency 134.723 ms 1 609 0.14 MB/sec execute 36 sec latency 182.244 ms 1 655 0.14 MB/sec execute 37 sec latency 168.465 ms 1 676 0.14 MB/sec execute 38 sec latency 295.241 ms oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 1 711 0.14 MB/sec execute 39 sec latency 145.045 ms 1 734 0.14 MB/sec execute 40 sec latency 291.169 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 778 0.13 MB/sec execute 41 sec latency 122.741 ms 1 806 0.13 MB/sec execute 42 sec latency 278.635 ms 1 861 0.13 MB/sec execute 43 sec latency 125.145 ms 1 881 0.13 MB/sec execute 44 sec latency 282.089 ms 1 914 0.13 MB/sec execute 45 sec latency 105.487 ms 1 977 0.13 MB/sec execute 46 sec latency 83.321 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3200 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 39936 3706880 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3715072 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 72704 7421952 1% /mnt/lustre 1 1026 0.15 MB/sec execute 47 sec latency 161.484 ms 1 1077 0.15 MB/sec execute 48 sec latency 84.570 ms 1 1127 0.14 MB/sec execute 49 sec latency 140.287 ms 1 1155 0.14 MB/sec execute 50 sec latency 171.241 ms 1 1192 0.14 MB/sec execute 51 sec latency 116.078 ms 1 1223 0.14 MB/sec execute 52 sec latency 147.561 ms 1 1246 0.14 MB/sec execute 53 sec latency 169.523 ms test_26 fail mds1 4 times 1 1273 0.13 MB/sec execute 54 sec latency 162.747 ms 1 1387 0.18 MB/sec execute 55 sec latency 152.225 ms Failing mds1 on oleg229-server 1 1405 0.18 MB/sec execute 56 sec latency 147.848 ms Stopping /mnt/lustre-mds1 (opts:) on oleg229-server 1 1446 0.18 MB/sec execute 57 sec latency 122.867 ms 1 1481 0.17 MB/sec execute 58 sec latency 158.926 ms 1 1489 0.17 MB/sec execute 59 sec latency 756.337 ms 1 1489 0.17 MB/sec execute 60 sec latency 1771.353 ms 1 1489 0.17 MB/sec execute 61 sec latency 2772.459 ms 20:12:24 (1788653544) shut down facet: mds1 facet_host: oleg229-server facet_failover_host: oleg229-server 1 1489 0.16 MB/sec execute 62 sec latency 3772.726 ms 1 1489 0.16 MB/sec execute 63 sec latency 4772.921 ms 1 1489 0.16 MB/sec execute 64 sec latency 5773.151 ms 1 1489 0.16 MB/sec execute 65 sec latency 6773.788 ms 1 1489 0.15 MB/sec execute 66 sec latency 7775.609 ms 1 1489 0.15 MB/sec execute 67 sec latency 8775.846 ms 1 1489 0.15 MB/sec execute 68 sec latency 9776.708 ms 1 1489 0.15 MB/sec execute 69 sec latency 10777.297 ms 1 1489 0.14 MB/sec execute 70 sec latency 11777.789 ms 1 1489 0.14 MB/sec execute 71 sec latency 12778.304 ms Failover mds1 to oleg229-server 1 1489 0.14 MB/sec execute 72 sec latency 13786.970 ms mount facets: mds1 1 1489 0.14 MB/sec execute 73 sec latency 14799.665 ms 1 1489 0.14 MB/sec execute 74 sec latency 15801.139 ms 1 1489 0.13 MB/sec execute 75 sec latency 16801.380 ms 1 1489 0.13 MB/sec execute 76 sec latency 17801.751 ms 1 1489 0.13 MB/sec execute 77 sec latency 18804.344 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 1489 0.13 MB/sec execute 78 sec latency 19804.643 ms 1 1489 0.13 MB/sec execute 79 sec latency 20804.938 ms 1 1489 0.13 MB/sec execute 80 sec latency 21805.133 ms 1 1489 0.12 MB/sec execute 81 sec latency 22805.873 ms 1 1489 0.12 MB/sec execute 82 sec latency 23808.018 ms 1 1489 0.12 MB/sec execute 83 sec latency 24809.410 ms 1 1489 0.12 MB/sec execute 84 sec latency 25809.721 ms 1 1489 0.12 MB/sec execute 85 sec latency 26811.663 ms 1 1489 0.12 MB/sec execute 86 sec latency 27814.716 ms 1 1489 0.12 MB/sec execute 87 sec latency 28823.870 ms 1 1489 0.12 MB/sec execute 88 sec latency 29824.806 ms 1 1489 0.11 MB/sec execute 89 sec latency 30827.275 ms 1 1489 0.11 MB/sec execute 90 sec latency 31827.522 ms 1 1489 0.11 MB/sec execute 91 sec latency 32828.202 ms oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all 1 1489 0.11 MB/sec execute 92 sec latency 33828.892 ms 1 1489 0.11 MB/sec execute 93 sec latency 34829.891 ms pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 1 1489 0.11 MB/sec execute 94 sec latency 35830.280 ms Started lustre-MDT0000 20:12:57 (1788653577) targets are mounted 20:12:57 (1788653577) facet_failover done 1 1489 0.11 MB/sec execute 95 sec latency 36830.490 ms 1 1489 0.11 MB/sec execute 96 sec latency 37831.715 ms 1 1489 0.10 MB/sec execute 97 sec latency 38834.180 ms 1 1489 0.10 MB/sec execute 98 sec latency 39840.291 ms 1 1490 0.10 MB/sec execute 99 sec latency 40767.853 ms 1 cleanup 100 sec 1 cleanup 101 sec 1 cleanup 102 sec 1 cleanup 103 sec 0 cleanup 104 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 161 571.117 40767.830 Close 137 12.853 41.133 Rename 10 102.743 171.208 Unlink 20 30.688 63.964 Qpathinfo 117 24.576 158.905 Qfileinfo 47 2.396 23.267 Qfsinfo 33 5.881 40.168 Sfileinfo 34 66.810 147.545 Find 52 10.328 92.056 WriteX 200 12.781 99.136 ReadX 235 1.505 81.138 Flush 25 136.101 295.220 Throughput 0.102247 MB/sec 1 clients 1 procs max_latency=40767.853 ms oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 stopping dbench on /mnt/lustre at Sat Sep 5 20:13:06 EDT 2026 with return code 0 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Sat Sep 5 20:13:13 EDT 2026 waiting for dbench pid 49357 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 51 2.08 MB/sec warmup 1 sec latency 78.796 ms 1 109 2.58 MB/sec warmup 2 sec latency 140.951 ms 1 147 2.09 MB/sec warmup 3 sec latency 156.644 ms 1 182 1.82 MB/sec warmup 4 sec latency 190.301 ms tar: Unexpected EOF in archive tar: Unexpected EOF in archive 1 223 1.72 MB/sec warmup 5 sec latency 175.965 ms 1 280 1.74 MB/sec warmup 6 sec latency 104.344 ms 1 323 1.67 MB/sec warmup 7 sec latency 119.486 ms 1 358 1.59 MB/sec warmup 8 sec latency 136.517 ms 1 402 1.57 MB/sec warmup 9 sec latency 127.589 ms 1 446 1.56 MB/sec warmup 10 sec latency 97.407 ms 1 493 1.56 MB/sec warmup 11 sec latency 142.456 ms 1 533 1.54 MB/sec warmup 12 sec latency 115.828 ms 1 636 1.62 MB/sec warmup 13 sec latency 104.845 ms 1 669 1.51 MB/sec warmup 14 sec latency 324.588 ms tar: Error is not recoverable: exiting now dbench killed by signal 15 stopping dbench on /mnt/lustre at Sat Sep 5 20:13:28 EDT 2026 with return code 0 1 699 1.41 MB/sec warmup 15 sec latency 127.765 ms 49357 pts/0 S+ 0:00 dbench -c client.txt 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100 killed dbench main pid 49357 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished