== replay-dual test 26: dbench and tar with mds failover ========================================================== 22:54:03 (1790736843) Starting client oleg405-client.virtnet: -o user_xattr,flock 192.168.204.105@tcp:/lustre /mnt/lustre Started clients oleg405-client.virtnet: 192.168.204.105@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Started tar loop with pid 45161 Started dbench loop with 45162 looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2048 2206464 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 2048 3760128 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 14336 3737600 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 16384 7497728 1% /mnt/lustre '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Tue Sep 29 22:54:10 EDT 2026 waiting for dbench pid 45230 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs failed to create barrier semaphore 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 20 0.39 MB/sec warmup 1 sec latency 141.933 ms 1 71 1.57 MB/sec warmup 2 sec latency 152.782 ms 1 111 1.75 MB/sec warmup 3 sec latency 153.020 ms 1 137 1.51 MB/sec warmup 4 sec latency 274.123 ms 1 165 1.36 MB/sec warmup 5 sec latency 146.137 ms test_26 fail mds1 1 times 1 192 1.26 MB/sec warmup 6 sec latency 224.208 ms 1 216 1.19 MB/sec warmup 7 sec latency 191.718 ms 1 247 1.17 MB/sec warmup 8 sec latency 222.778 ms Failing mds1 on oleg405-server 1 283 1.17 MB/sec warmup 9 sec latency 173.413 ms Stopping /mnt/lustre-mds1 (opts:) on oleg405-server 1 310 1.13 MB/sec warmup 10 sec latency 167.793 ms 1 331 1.09 MB/sec warmup 11 sec latency 347.696 ms 1 331 1.00 MB/sec warmup 12 sec latency 1347.960 ms 1 331 0.92 MB/sec warmup 13 sec latency 2349.820 ms 22:54:24 (1790736864) shut down facet: mds1 facet_host: oleg405-server facet_failover_host: oleg405-server 1 331 0.86 MB/sec warmup 14 sec latency 3351.566 ms 1 331 0.80 MB/sec warmup 15 sec latency 4352.948 ms 1 331 0.75 MB/sec warmup 16 sec latency 5357.257 ms 1 331 0.71 MB/sec warmup 17 sec latency 6357.597 ms 1 331 0.67 MB/sec warmup 18 sec latency 7358.090 ms 1 331 0.63 MB/sec warmup 19 sec latency 8358.316 ms 1 331 0.00 MB/sec execute 1 sec latency 10359.174 ms 1 331 0.00 MB/sec execute 2 sec latency 11359.405 ms 1 331 0.00 MB/sec execute 3 sec latency 12359.666 ms 1 331 0.00 MB/sec execute 4 sec latency 13360.157 ms Failover mds1 to oleg405-server mount facets: mds1 1 331 0.00 MB/sec execute 5 sec latency 14362.328 ms 1 331 0.00 MB/sec execute 6 sec latency 15362.652 ms 1 331 0.00 MB/sec execute 7 sec latency 16362.919 ms 1 331 0.00 MB/sec execute 8 sec latency 17363.286 ms 1 331 0.00 MB/sec execute 9 sec latency 18363.906 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 331 0.00 MB/sec execute 10 sec latency 19364.150 ms 1 331 0.00 MB/sec execute 11 sec latency 20364.477 ms 1 331 0.00 MB/sec execute 12 sec latency 21365.059 ms 1 331 0.00 MB/sec execute 13 sec latency 22365.574 ms 1 331 0.00 MB/sec execute 14 sec latency 23368.229 ms 1 331 0.00 MB/sec execute 15 sec latency 24368.482 ms 1 331 0.00 MB/sec execute 16 sec latency 25368.931 ms 1 331 0.00 MB/sec execute 17 sec latency 26369.193 ms 1 331 0.00 MB/sec execute 18 sec latency 27370.268 ms 1 331 0.00 MB/sec execute 19 sec latency 28370.564 ms 1 331 0.00 MB/sec execute 20 sec latency 29371.060 ms 1 331 0.00 MB/sec execute 21 sec latency 30371.528 ms 1 331 0.00 MB/sec execute 22 sec latency 31373.186 ms 1 331 0.00 MB/sec execute 23 sec latency 32373.747 ms 1 331 0.00 MB/sec execute 24 sec latency 33374.432 ms 1 331 0.00 MB/sec execute 25 sec latency 34374.697 ms oleg405-server: oleg405-server.virtnet: executing set_default_debug -1 all 1 331 0.00 MB/sec execute 26 sec latency 35375.286 ms pdsh@oleg405-client: oleg405-server: ssh exited with exit code 1 1 331 0.00 MB/sec execute 27 sec latency 36375.817 ms Started lustre-MDT0000 22:54:58 (1790736898) targets are mounted 22:54:58 (1790736898) facet_failover done 1 331 0.00 MB/sec execute 28 sec latency 37378.439 ms 1 331 0.00 MB/sec execute 29 sec latency 38379.013 ms 1 331 0.00 MB/sec execute 30 sec latency 39379.287 ms 1 342 0.01 MB/sec execute 31 sec latency 39942.774 ms 1 382 0.05 MB/sec execute 32 sec latency 128.678 ms 1 422 0.09 MB/sec execute 33 sec latency 156.609 ms 1 461 0.12 MB/sec execute 34 sec latency 136.426 ms 1 501 0.16 MB/sec execute 35 sec latency 157.369 ms oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 1 542 0.20 MB/sec execute 36 sec latency 163.444 ms 1 616 0.24 MB/sec execute 37 sec latency 121.989 ms 1 656 0.24 MB/sec execute 38 sec latency 132.689 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 674 0.23 MB/sec execute 39 sec latency 297.705 ms 1 695 0.23 MB/sec execute 40 sec latency 210.960 ms 1 721 0.23 MB/sec execute 41 sec latency 183.698 ms 1 740 0.22 MB/sec execute 42 sec latency 239.974 ms 1 776 0.22 MB/sec execute 43 sec latency 142.695 ms 1 796 0.21 MB/sec execute 44 sec latency 305.106 ms 1 825 0.21 MB/sec execute 45 sec latency 153.644 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2304 2206208 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 14336 3681280 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 35840 3704832 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 50176 7386112 1% /mnt/lustre 1 876 0.21 MB/sec execute 46 sec latency 208.791 ms 1 902 0.21 MB/sec execute 47 sec latency 145.667 ms 1 919 0.20 MB/sec execute 48 sec latency 296.240 ms 1 955 0.20 MB/sec execute 49 sec latency 124.797 ms 1 981 0.20 MB/sec execute 50 sec latency 278.136 ms 1 1040 0.21 MB/sec execute 51 sec latency 133.924 ms 1 1099 0.21 MB/sec execute 52 sec latency 76.216 ms test_26 fail mds1 2 times 1 1138 0.21 MB/sec execute 53 sec latency 76.272 ms 1 1173 0.20 MB/sec execute 54 sec latency 116.749 ms Failing mds1 on oleg405-server 1 1214 0.20 MB/sec execute 55 sec latency 94.563 ms Stopping /mnt/lustre-mds1 (opts:) on oleg405-server 1 1233 0.20 MB/sec execute 56 sec latency 133.229 ms 1 1245 0.19 MB/sec execute 57 sec latency 626.503 ms 1 1245 0.19 MB/sec execute 58 sec latency 1627.516 ms 1 1245 0.19 MB/sec execute 59 sec latency 2628.010 ms 22:55:30 (1790736930) shut down facet: mds1 facet_host: oleg405-server facet_failover_host: oleg405-server 1 1245 0.18 MB/sec execute 60 sec latency 3628.508 ms 1 1245 0.18 MB/sec execute 61 sec latency 4629.030 ms 1 1245 0.18 MB/sec execute 62 sec latency 5629.441 ms 1 1245 0.18 MB/sec execute 63 sec latency 6630.245 ms 1 1245 0.17 MB/sec execute 64 sec latency 7631.166 ms 1 1245 0.17 MB/sec execute 65 sec latency 8633.372 ms 1 1245 0.17 MB/sec execute 66 sec latency 9635.888 ms 1 1245 0.17 MB/sec execute 67 sec latency 10636.279 ms 1 1245 0.16 MB/sec execute 68 sec latency 11638.077 ms 1 1245 0.16 MB/sec execute 69 sec latency 12641.560 ms Failover mds1 to oleg405-server 1 1245 0.16 MB/sec execute 70 sec latency 13641.755 ms mount facets: mds1 1 1245 0.16 MB/sec execute 71 sec latency 14642.729 ms 1 1245 0.15 MB/sec execute 72 sec latency 15643.005 ms 1 1245 0.15 MB/sec execute 73 sec latency 16643.267 ms 1 1245 0.15 MB/sec execute 74 sec latency 17643.627 ms 1 1245 0.15 MB/sec execute 75 sec latency 18644.247 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 1245 0.15 MB/sec execute 76 sec latency 19644.625 ms 1 1245 0.14 MB/sec execute 77 sec latency 20645.038 ms 1 1245 0.14 MB/sec execute 78 sec latency 21646.386 ms 1 1245 0.14 MB/sec execute 79 sec latency 22646.980 ms 1 1245 0.14 MB/sec execute 80 sec latency 23647.420 ms 1 1245 0.14 MB/sec execute 81 sec latency 24650.051 ms 1 1245 0.13 MB/sec execute 82 sec latency 25658.478 ms 1 1245 0.13 MB/sec execute 83 sec latency 26660.030 ms oleg405-server: oleg405-server.virtnet: executing set_default_debug -1 all 1 1245 0.13 MB/sec execute 84 sec latency 27666.481 ms pdsh@oleg405-client: oleg405-server: ssh exited with exit code 1 1 1272 0.13 MB/sec execute 85 sec latency 27679.432 ms Started lustre-MDT0000 22:55:57 (1790736957) targets are mounted 22:55:57 (1790736957) facet_failover done 1 1391 0.16 MB/sec execute 86 sec latency 128.020 ms 1 1426 0.16 MB/sec execute 87 sec latency 116.991 ms 1 1466 0.16 MB/sec execute 88 sec latency 101.968 ms 1 1492 0.16 MB/sec execute 89 sec latency 338.755 ms 1 1524 0.16 MB/sec execute 90 sec latency 131.798 ms 1 1570 0.15 MB/sec execute 91 sec latency 85.272 ms 1 1609 0.15 MB/sec execute 92 sec latency 94.858 ms 1 1637 0.15 MB/sec execute 93 sec latency 305.506 ms oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 1 1684 0.15 MB/sec execute 94 sec latency 124.952 ms 1 1721 0.15 MB/sec execute 95 sec latency 123.963 ms 1 1773 0.15 MB/sec execute 96 sec latency 131.611 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 1834 0.15 MB/sec execute 97 sec latency 110.068 ms 1 1864 0.15 MB/sec execute 98 sec latency 171.376 ms 1 1902 0.15 MB/sec execute 99 sec latency 238.734 ms 1 cleanup 100 sec 1 cleanup 101 sec 1 cleanup 102 sec 1 cleanup 103 sec 0 cleanup 104 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 279 212.058 39942.753 Close 244 14.284 43.659 Rename 11 100.216 151.281 Unlink 26 35.151 65.391 Qpathinfo 223 143.120 27679.400 Qfileinfo 80 2.237 25.493 Qfsinfo 33 3.571 31.656 Sfileinfo 34 67.090 134.306 Find 75 9.729 74.586 WriteX 303 13.140 100.670 ReadX 297 2.276 75.433 Flush 30 163.381 338.722 Throughput 0.145467 MB/sec 1 clients 1 procs max_latency=39942.774 ms stopping dbench on /mnt/lustre at Tue Sep 29 22:56:14 EDT 2026 with return code 0 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 2688 2205824 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 20480 3733504 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 47104 3693568 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 67584 7427072 1% /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Tue Sep 29 22:56:22 EDT 2026 waiting for dbench pid 47273 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients test_26 fail mds1 3 times 1 40 1.44 MB/sec warmup 1 sec latency 138.129 ms 1 98 2.22 MB/sec warmup 2 sec latency 127.507 ms 1 129 1.92 MB/sec warmup 3 sec latency 235.391 ms Failing mds1 on oleg405-server 1 151 1.58 MB/sec warmup 4 sec latency 203.440 ms Stopping /mnt/lustre-mds1 (opts:) on oleg405-server 1 181 1.46 MB/sec warmup 5 sec latency 160.894 ms 1 196 1.30 MB/sec warmup 6 sec latency 523.940 ms 1 196 1.12 MB/sec warmup 7 sec latency 1530.291 ms 1 196 0.98 MB/sec warmup 8 sec latency 2531.641 ms 22:56:30 (1790736990) shut down facet: mds1 facet_host: oleg405-server facet_failover_host: oleg405-server 1 196 0.87 MB/sec warmup 9 sec latency 3532.213 ms 1 196 0.78 MB/sec warmup 10 sec latency 4532.549 ms 1 196 0.71 MB/sec warmup 11 sec latency 5532.800 ms 1 196 0.65 MB/sec warmup 12 sec latency 6533.142 ms 1 196 0.60 MB/sec warmup 13 sec latency 7533.440 ms 1 196 0.56 MB/sec warmup 14 sec latency 8534.477 ms 1 196 0.52 MB/sec warmup 15 sec latency 9535.867 ms 1 196 0.49 MB/sec warmup 16 sec latency 10536.207 ms 1 196 0.46 MB/sec warmup 17 sec latency 11536.514 ms 1 196 0.44 MB/sec warmup 18 sec latency 12537.093 ms Failover mds1 to oleg405-server mount facets: mds1 1 196 0.41 MB/sec warmup 19 sec latency 13538.008 ms 1 196 0.00 MB/sec execute 1 sec latency 15539.526 ms 1 196 0.00 MB/sec execute 2 sec latency 16539.813 ms 1 196 0.00 MB/sec execute 3 sec latency 17541.622 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 196 0.00 MB/sec execute 4 sec latency 18542.110 ms 1 196 0.00 MB/sec execute 5 sec latency 19542.432 ms 1 196 0.00 MB/sec execute 6 sec latency 20542.921 ms 1 196 0.00 MB/sec execute 7 sec latency 21543.462 ms 1 196 0.00 MB/sec execute 8 sec latency 22543.731 ms 1 196 0.00 MB/sec execute 9 sec latency 23544.285 ms 1 196 0.00 MB/sec execute 10 sec latency 24544.772 ms 1 196 0.00 MB/sec execute 11 sec latency 25545.293 ms 1 196 0.00 MB/sec execute 12 sec latency 26545.922 ms 1 196 0.00 MB/sec execute 13 sec latency 27546.383 ms 1 196 0.00 MB/sec execute 14 sec latency 28546.546 ms 1 196 0.00 MB/sec execute 15 sec latency 29546.763 ms 1 196 0.00 MB/sec execute 16 sec latency 30548.525 ms 1 196 0.00 MB/sec execute 17 sec latency 31548.795 ms 1 196 0.00 MB/sec execute 18 sec latency 32549.113 ms oleg405-server: oleg405-server.virtnet: executing set_default_debug -1 all 1 196 0.00 MB/sec execute 19 sec latency 33549.931 ms 1 196 0.00 MB/sec execute 20 sec latency 34550.733 ms pdsh@oleg405-client: oleg405-server: ssh exited with exit code 1 1 196 0.00 MB/sec execute 21 sec latency 35552.864 ms Started lustre-MDT0000 22:57:03 (1790737023) targets are mounted 22:57:03 (1790737023) facet_failover done 1 196 0.00 MB/sec execute 22 sec latency 36553.177 ms 1 196 0.00 MB/sec execute 23 sec latency 37554.507 ms 1 196 0.00 MB/sec execute 24 sec latency 38556.694 ms 1 196 0.00 MB/sec execute 25 sec latency 39558.695 ms 1 196 0.00 MB/sec execute 26 sec latency 40558.972 ms 1 196 0.00 MB/sec execute 27 sec latency 41562.223 ms 1 197 0.00 MB/sec execute 28 sec latency 42309.236 ms 1 224 0.03 MB/sec execute 29 sec latency 414.325 ms 1 263 0.07 MB/sec execute 30 sec latency 112.652 ms 1 297 0.10 MB/sec execute 31 sec latency 137.893 ms 1 331 0.13 MB/sec execute 32 sec latency 149.167 ms 1 358 0.15 MB/sec execute 33 sec latency 193.584 ms oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 1 389 0.18 MB/sec execute 34 sec latency 153.692 ms 1 422 0.20 MB/sec execute 35 sec latency 140.521 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 456 0.22 MB/sec execute 36 sec latency 135.255 ms 1 485 0.25 MB/sec execute 37 sec latency 142.544 ms 1 509 0.26 MB/sec execute 38 sec latency 173.181 ms 1 558 0.29 MB/sec execute 39 sec latency 157.976 ms 1 612 0.33 MB/sec execute 40 sec latency 116.459 ms 1 659 0.32 MB/sec execute 41 sec latency 131.396 ms 1 674 0.32 MB/sec execute 42 sec latency 239.255 ms 1 699 0.31 MB/sec execute 43 sec latency 113.197 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 3072 2205440 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 17408 3696640 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 46080 3687424 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 63488 7384064 1% /mnt/lustre 1 721 0.31 MB/sec execute 44 sec latency 184.812 ms 1 731 0.30 MB/sec execute 45 sec latency 431.936 ms 1 762 0.29 MB/sec execute 46 sec latency 113.641 ms 1 789 0.29 MB/sec execute 47 sec latency 150.858 ms 1 801 0.28 MB/sec execute 48 sec latency 359.354 ms 1 848 0.28 MB/sec execute 49 sec latency 127.575 ms 1 878 0.28 MB/sec execute 50 sec latency 243.697 ms test_26 fail mds1 4 times 1 903 0.27 MB/sec execute 51 sec latency 162.489 ms 1 921 0.27 MB/sec execute 52 sec latency 273.590 ms 1 952 0.26 MB/sec execute 53 sec latency 129.564 ms Failing mds1 on oleg405-server 1 977 0.26 MB/sec execute 54 sec latency 166.094 ms Stopping /mnt/lustre-mds1 (opts:) on oleg405-server 1 1000 0.26 MB/sec execute 55 sec latency 292.820 ms 1 1041 0.27 MB/sec execute 56 sec latency 349.372 ms 1 1041 0.26 MB/sec execute 57 sec latency 1349.750 ms 1 1041 0.26 MB/sec execute 58 sec latency 2353.619 ms 22:57:40 (1790737060) shut down facet: mds1 facet_host: oleg405-server facet_failover_host: oleg405-server 1 1041 0.25 MB/sec execute 59 sec latency 3354.022 ms 1 1041 0.25 MB/sec execute 60 sec latency 4354.514 ms 1 1041 0.25 MB/sec execute 61 sec latency 5354.982 ms 1 1041 0.24 MB/sec execute 62 sec latency 6355.242 ms 1 1041 0.24 MB/sec execute 63 sec latency 7355.985 ms 1 1041 0.23 MB/sec execute 64 sec latency 8356.572 ms 1 1041 0.23 MB/sec execute 65 sec latency 9356.884 ms 1 1041 0.23 MB/sec execute 66 sec latency 10360.268 ms 1 1041 0.22 MB/sec execute 67 sec latency 11360.548 ms 1 1041 0.22 MB/sec execute 68 sec latency 12362.371 ms Failover mds1 to oleg405-server 1 1041 0.22 MB/sec execute 69 sec latency 13362.635 ms mount facets: mds1 1 1041 0.21 MB/sec execute 70 sec latency 14363.292 ms 1 1041 0.21 MB/sec execute 71 sec latency 15363.587 ms 1 1041 0.21 MB/sec execute 72 sec latency 16364.853 ms 1 1041 0.21 MB/sec execute 73 sec latency 17365.393 ms 1 1041 0.20 MB/sec execute 74 sec latency 18367.179 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 1041 0.20 MB/sec execute 75 sec latency 19367.397 ms 1 1041 0.20 MB/sec execute 76 sec latency 20373.586 ms 1 1041 0.20 MB/sec execute 77 sec latency 21378.388 ms 1 1041 0.19 MB/sec execute 78 sec latency 22378.724 ms 1 1041 0.19 MB/sec execute 79 sec latency 23379.905 ms 1 1041 0.19 MB/sec execute 80 sec latency 24380.140 ms 1 1041 0.19 MB/sec execute 81 sec latency 25380.359 ms 1 1041 0.18 MB/sec execute 82 sec latency 26380.887 ms 1 1041 0.18 MB/sec execute 83 sec latency 27384.697 ms 1 1041 0.18 MB/sec execute 84 sec latency 28385.575 ms 1 1041 0.18 MB/sec execute 85 sec latency 29385.799 ms 1 1041 0.17 MB/sec execute 86 sec latency 30387.483 ms 1 1041 0.17 MB/sec execute 87 sec latency 31392.513 ms 1 1041 0.17 MB/sec execute 88 sec latency 32392.901 ms 1 1041 0.17 MB/sec execute 89 sec latency 33396.905 ms 1 1041 0.17 MB/sec execute 90 sec latency 34397.152 ms 1 1041 0.17 MB/sec execute 91 sec latency 35397.510 ms 1 1041 0.16 MB/sec execute 92 sec latency 36397.706 ms oleg405-server: oleg405-server.virtnet: executing set_default_debug -1 all 1 1041 0.16 MB/sec execute 93 sec latency 37398.050 ms 1 1041 0.16 MB/sec execute 94 sec latency 38398.293 ms pdsh@oleg405-client: oleg405-server: ssh exited with exit code 1 1 1041 0.16 MB/sec execute 95 sec latency 39403.477 ms Started lustre-MDT0000 22:58:18 (1790737098) targets are mounted 22:58:18 (1790737098) facet_failover done 1 1041 0.16 MB/sec execute 96 sec latency 40408.472 ms 1 1041 0.15 MB/sec execute 97 sec latency 41408.693 ms 1 1041 0.15 MB/sec execute 98 sec latency 42409.481 ms 1 1041 0.15 MB/sec execute 99 sec latency 43426.022 ms 1 cleanup 100 sec 1 cleanup 101 sec 1 cleanup 102 sec 1 cleanup 103 sec 1 cleanup 104 sec 0 cleanup 105 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 123 454.626 43624.428 Close 115 384.088 42309.202 Rename 4 166.037 184.785 Unlink 11 36.868 67.632 Qpathinfo 40 33.116 128.368 Qfileinfo 64 2.486 11.116 Qfsinfo 23 11.298 79.761 Sfileinfo 18 68.233 111.239 Find 29 10.312 68.341 WriteX 321 17.489 80.531 ReadX 100 5.676 91.912 Flush 24 169.155 431.911 Throughput 0.151755 MB/sec 1 clients 1 procs max_latency=43426.022 ms stopping dbench on /mnt/lustre at Tue Sep 29 22:58:27 EDT 2026 with return code 0 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 removed 'client.txt' /mnt/lustre dbench successfully finished mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Tue Sep 29 22:58:34 EDT 2026 waiting for dbench pid 49236 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 48 1.97 MB/sec warmup 1 sec latency 82.544 ms tar: Unexpected EOF in archive tar: Unexpected EOF in archive 1 113 2.62 MB/sec warmup 2 sec latency 123.955 ms 1 142 2.00 MB/sec warmup 3 sec latency 220.807 ms 1 181 1.83 MB/sec warmup 4 sec latency 140.186 ms 1 213 1.66 MB/sec warmup 5 sec latency 174.021 ms 1 255 1.61 MB/sec warmup 6 sec latency 152.392 ms 1 295 1.56 MB/sec warmup 7 sec latency 148.450 ms 1 337 1.52 MB/sec warmup 8 sec latency 132.138 ms 1 365 1.45 MB/sec warmup 9 sec latency 146.236 ms 1 398 1.41 MB/sec warmup 10 sec latency 150.242 ms 1 436 1.40 MB/sec warmup 11 sec latency 140.283 ms 1 461 1.34 MB/sec warmup 12 sec latency 192.363 ms 1 493 1.32 MB/sec warmup 13 sec latency 266.156 ms tar: Error is not recoverable: exiting now dbench killed by signal 15 stopping dbench on /mnt/lustre at Tue Sep 29 22:58:48 EDT 2026 with return code 0 49236 pts/0 S+ 0:00 dbench -c client.txt 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100 killed dbench main pid 49236 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished