== replay-single test 70b: dbench 2mdts recovery; 1 clients ========================================================== 22:56:53 (1788577013) Starting client oleg334-client.virtnet: -o user_xattr,flock 192.168.203.134@tcp:/lustre /mnt/lustre Started clients oleg334-client.virtnet: 192.168.203.134@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70b.replay-single + MISSING_DBENCH_OK= + PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin::/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests:/sbin:/usr/sbin:/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests/: + DBENCH_LIB= + TESTSUITE=replay-single + TESTNAME=test_70b + MOUNT=/mnt/lustre ++ hostname + DIR=/mnt/lustre/d70b.replay-single/oleg334-client.virtnet + LCTL=/home/green/git/lustre-release/lustre/utils/lctl + rundbench 1 -t 120 dbench: no process found dbench: no process found Started rundbench load pid=135409 ... UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2404 1268900 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2036 1269268 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1600 3605420 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 26152 3576600 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 27752 7182020 1% /mnt/lustre test_70b fail mds1 1 times Failing mds1 on oleg334-server Stopping /mnt/lustre-mds1 (opts:) on oleg334-server 22:57:16 (1788577036) shut down facet: mds1 facet_host: oleg334-server facet_failover_host: oleg334-server Failover mds1 to oleg334-server mount facets: mds1 oleg334-client.virtnet: looking for dbench program oleg334-client.virtnet: /usr/bin/dbench oleg334-client.virtnet: creating output directory /mnt/lustre/d70b.replay-single/oleg334-client.virtnet oleg334-client.virtnet: mkdir: created directory '/mnt/lustre/d70b.replay-single/oleg334-client.virtnet' oleg334-client.virtnet: found dbench client file /usr/share/dbench/client.txt oleg334-client.virtnet: '/usr/share/dbench/client.txt' -> 'client.txt' oleg334-client.virtnet: running 'dbench 1 -t 120' on /mnt/lustre/d70b.replay-single/oleg334-client.virtnet at Fri Sep 4 22:56:57 EDT 2026 oleg334-client.virtnet: waiting for dbench pid 135441 oleg334-client.virtnet: dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 oleg334-client.virtnet: oleg334-client.virtnet: Running for 120 seconds with load 'client.txt' and minimum warmup 24 secs oleg334-client.virtnet: failed to create barrier semaphore oleg334-client.virtnet: 0 of 1 processes prepared for launch 0 sec oleg334-client.virtnet: 1 of 1 processes prepared for launch 0 sec oleg334-client.virtnet: releasing clients oleg334-client.virtnet: 1 111 5.24 MB/sec warmup 1 sec latency 61.724 ms oleg334-client.virtnet: 1 168 3.40 MB/sec warmup 2 sec latency 100.771 ms oleg334-client.virtnet: 1 239 3.05 MB/sec warmup 3 sec latency 99.187 ms oleg334-client.virtnet: 1 315 2.87 MB/sec warmup 4 sec latency 84.466 ms oleg334-client.virtnet: 1 374 2.66 MB/sec warmup 5 sec latency 92.743 ms oleg334-client.virtnet: 1 461 2.69 MB/sec warmup 6 sec latency 73.073 ms oleg334-client.virtnet: 1 546 2.73 MB/sec warmup 7 sec latency 79.692 ms oleg334-client.virtnet: 1 678 2.65 MB/sec warmup 8 sec latency 74.306 ms oleg334-client.virtnet: 1 721 2.36 MB/sec warmup 9 sec latency 469.044 ms oleg334-client.virtnet: 1 721 2.13 MB/sec warmup 10 sec latency 1469.476 ms oleg334-client.virtnet: 1 789 1.95 MB/sec warmup 11 sec latency 1570.096 ms oleg334-client.virtnet: 1 882 1.80 MB/sec warmup 12 sec latency 64.088 ms oleg334-client.virtnet: 1 923 1.67 MB/sec warmup 13 sec latency 144.601 ms oleg334-client.virtnet: 1 986 1.56 MB/sec warmup 14 sec latency 73.590 ms oleg334-client.virtnet: 1 1103 1.53 MB/sec warmup 15 sec latency 64.083 ms oleg334-client.virtnet: 1 1128 1.44 MB/sec warmup 16 sec latency 829.583 ms oleg334-client.virtnet: 1 1128 1.35 MB/sec warmup 17 sec latency 1829.764 ms oleg334-client.virtnet: 1 1128 1.28 MB/sec warmup 18 sec latency 2830.502 ms oleg334-client.virtnet: 1 1128 1.21 MB/sec warmup 19 sec latency 3830.717 ms oleg334-client.virtnet: 1 1128 1.15 MB/sec warmup 20 sec latency 4830.978 ms oleg334-client.virtnet: 1 1128 1.09 MB/sec warmup 21 sec latency 5832.003 ms oleg334-client.virtnet: 1 1128 1.04 MB/sec warmup 22 sec latency 6832.398 ms oleg334-client.virtnet: 1 1128 1.00 MB/sec warmup 23 sec latency 7832.799 ms oleg334-client.virtnet: 1 1128 0.00 MB/sec execute 1 sec latency 9833.410 ms oleg334-client.virtnet: 1 1128 0.00 MB/sec execute 2 sec latency 10834.797 ms oleg334-client.virtnet: 1 1128 0.00 MB/sec execute 3 sec latency 11835.385 ms oleg334-client.virtnet: 1 1128 0.00 MB/sec execute 4 sec latency 12839.487 ms oleg334-client.virtnet: 1 1128 0.00 MB/sec execute 5 sec latency 13840.085 ms oleg334-client.virtnet: 1 1128 0.00 MB/sec execute 6 sec latency 14841.238 ms oleg334-client.virtnet: 1 1128 0.00 MB/sec execute 7 sec latency 15841.974 ms oleg334-client.virtnet: 1 1128 0.00 MB/sec execute 8 sec latency 16844.727 ms oleg334-client.virtnet: 1 1128 0.00 MB/sec execute 9 sec latency 17846.119 ms oleg334-client.virtnet: 1 1128 0.00 MB/sec execute 10 sec Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg334-server: oleg334-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg334-client: oleg334-server: ssh exited with exit code 1 Started lustre-MDT0000 22:57:50 (1788577070) targets are mounted 22:57:50 (1788577070) facet_failover done oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2352 1268952 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1996 1269308 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 13252 3592964 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 36452 3569984 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49704 7162948 1% /mnt/lustre test_70b fail mds2 2 times latency 18847.134 ms oleg334-client.virtnet: 1 1128 0.00 MB/sec execute 11 sec latency 19847.507 ms oleg334-client.virtnet: 1 1128 0.00 MB/sec execute 12 sec latency 20853.183 ms oleg334-client.virtnet: 1 1128 0.00 MB/sec execute 13 sec latency 21853.544 ms oleg334-client.virtnet: 1 1128 0.00 MB/sec execute 14 sec latency 22855.772 ms oleg334-client.virtnet: 1 1128 0.00 MB/sec execute 15 sec latency 23858.911 ms oleg334-client.virtnet: 1 1128 0.00 MB/sec execute 16 sec latency 24859.427 ms oleg334-client.virtnet: 1 1128 0.00 MB/sec execute 17 sec latency 25859.862 ms oleg334-client.virtnet: 1 1128 0.00 MB/sec execute 18 sec latency 26860.101 ms oleg334-client.virtnet: 1 1128 0.00 MB/sec execute 19 sec latency 27860.429 ms oleg334-client.virtnet: 1 1128 0.00 MB/sec execute 20 sec latency 28860.648 ms oleg334-client.virtnet: 1 1128 0.00 MB/sec execute 21 sec latency 29861.198 ms oleg334-client.virtnet: 1 1128 0.00 MB/sec execute 22 sec latency 30867.461 ms oleg334-client.virtnet: 1 1128 0.00 MB/sec execute 23 sec latency 31867.765 ms oleg334-client.virtnet: 1 1128 0.00 MB/sec execute 24 sec latency 32868.765 ms oleg334-client.virtnet: 1 1128 0.00 MB/sec execute 25 sec latency 33873.432 ms oleg334-client.virtnet: 1 1128 0.00 MB/sec execute 26 sec latency 34877.251 ms oleg334-client.virtnet: 1 1148 0.00 MB/sec execute 27 sec latency 35508.427 ms oleg334-client.virtnet: 1 1223 0.00 MB/sec execute 28 sec latency 77.862 ms oleg334-client.virtnet: 1 1274 0.01 MB/sec execute 29 sec latency 81.527 ms oleg334-client.virtnet: 1 1431 0.10 MB/sec execute 30 sec latency 65.511 ms oleg334-client.virtnet: 1 1497 0.10 MB/sec execute 31 sec latency 112.579 ms oleg334-client.virtnet: 1 1553 0.10 MB/sec execute 32 sec latency 100.543 ms oleg334-client.virtnet: 1 1646 0.10 MB/sec execute 33 sec latency 78.883 ms oleg334-client.virtnet: 1 1686 0.09 MB/sec execute 34 sec latency 112.756 ms oleg334-client.virtnet: 1 1785 0.09 MB/sec execute 35 sec latency 69.224 ms oleg334-client.virtnet: 1 1868 0.09 MB/sec execute 36 sec latency 82.274 ms oleg334-client.virtnet: 1 1969 0.09 MB/sec execute 37 sec latency 75.027 ms oleg334-client.virtnet: 1 2006 0.09 MB/sec execute 38 sec latency 104.524 ms oleg334-client.virtnet: 1 2097 0.09 MB/sec execute 39 sec latency 101.487 ms oleg334-client.virtnet: 1 2164 0.09 MB/sec execute 40 sec latency 102.606 ms oleg334-client.virtnet: 1 2333 0.12 MB/sec execute 41 sec latency 60.260 ms oleg334-client.virtnet: 1 2383 0.12 MB/sec execute 42 sec latency 108.440 ms oleg334-client.virtnet: 1 2466 0.13 MB/sec execute 43 sec latency 82.840 ms oleg334-client.virtnet: 1 2577 0.14 MB/sec execute 44 sec latency 98.326 ms oleg334-client.virtnet: 1 2764 0.20 MB/sec execute 45 sec latency 112.447 ms oleg334-client.virtnet: 1 2904 0.21 MB/sec execute 46 sec latency 71.092 ms oleg334-client.virtnet: 1 2984 0.22 MB/sec execute 47 sec latency 91.862 ms oleg334-client.virtnet: 1 3177 0.23 MB/sec execute 48 sec latency 84.706 ms oleg334-client.virtnet: 1 3342 0.25 MB/sec execute 49 sec latency 81.158 ms oleg334-client.virtnet: 1 3432 0.27 MB/sec execute 50 sec latency 83.801 ms oleg334-client.virtnet: 1 3589 0.30 MB/sec execute 51 sec latency 66.141 ms oleg334-client.virtnet: 1 3663 0.30 MB/sec execute 52 sec latency 94.651 ms oleg334-client.virtnet: 1 3728 0.30 MB/sec execute 53 sec latency 90.279 ms oleg334-client.virtnet: 1 3820 0.31 MB/sec execute 54 sec latency 95.893 ms oleg334-cliFailing mds2 on oleg334-server Stopping /mnt/lustre-mds2 (opts:) on oleg334-server 22:58:23 (1788577103) shut down facet: mds2 facet_host: oleg334-server facet_failover_host: oleg334-server Failover mds2 to oleg334-server mount facets: mds2 Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg334-server: oleg334-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg334-client: oleg334-server: ssh exited with exit code 1 Started lustre-MDT0001 22:58:50 (1788577130) targets are mounted 22:58:50 (1788577130) facet_failover done oleg334-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec ent.virtnet: 1 3874 0.30 MB/sec execute 55 sec latency 89.978 ms oleg334-client.virtnet: 1 3927 0.30 MB/sec execute 56 sec latency 94.990 ms oleg334-client.virtnet: 1 3990 0.30 MB/sec execute 57 sec latency 87.265 ms oleg334-client.virtnet: 1 4044 0.29 MB/sec execute 58 sec latency 121.800 ms oleg334-client.virtnet: 1 4089 0.29 MB/sec execute 59 sec latency 155.537 ms oleg334-client.virtnet: 1 4184 0.28 MB/sec execute 60 sec latency 50.775 ms oleg334-client.virtnet: 1 4248 0.28 MB/sec execute 61 sec latency 137.741 ms oleg334-client.virtnet: 1 4306 0.28 MB/sec execute 62 sec latency 86.621 ms oleg334-client.virtnet: 1 4386 0.28 MB/sec execute 63 sec latency 61.118 ms oleg334-client.virtnet: 1 4462 0.28 MB/sec execute 64 sec latency 94.320 ms oleg334-client.virtnet: 1 4546 0.28 MB/sec execute 65 sec latency 68.021 ms oleg334-client.virtnet: 1 4654 0.29 MB/sec execute 66 sec latency 72.095 ms oleg334-client.virtnet: 1 4759 0.29 MB/sec execute 67 sec latency 59.724 ms oleg334-client.virtnet: 1 4833 0.28 MB/sec execute 68 sec latency 80.770 ms oleg334-client.virtnet: 1 4985 0.32 MB/sec execute 69 sec latency 74.492 ms oleg334-client.virtnet: 1 5063 0.32 MB/sec execute 70 sec latency 57.648 ms oleg334-client.virtnet: 1 5147 0.31 MB/sec execute 71 sec latency 71.382 ms oleg334-client.virtnet: 1 5233 0.31 MB/sec execute 72 sec latency 78.789 ms oleg334-client.virtnet: 1 5286 0.31 MB/sec execute 73 sec latency 74.534 ms oleg334-client.virtnet: 1 5352 0.30 MB/sec execute 74 sec latency 78.371 ms oleg334-client.virtnet: 1 5408 0.30 MB/sec execute 75 sec latency 81.188 ms oleg334-client.virtnet: 1 5538 0.30 MB/sec execute 76 sec latency 53.133 ms oleg334-client.virtnet: 1 5581 0.29 MB/sec execute 77 sec latency 96.150 ms oleg334-client.virtnet: 1 5674 0.29 MB/sec execute 78 sec latency 78.029 ms oleg334-client.virtnet: 1 5746 0.29 MB/sec execute 79 sec latency 66.288 ms oleg334-client.virtnet: 1 5852 0.30 MB/sec execute 80 sec latency 55.873 ms oleg334-client.virtnet: 1 5938 0.30 MB/sec execute 81 sec latency 76.774 ms oleg334-client.virtnet: 1 5994 0.30 MB/sec execute 82 sec latency 83.992 ms oleg334-client.virtnet: 1 6117 0.30 MB/sec execute 83 sec latency 79.757 ms oleg334-client.virtnet: 1 6191 0.30 MB/sec execute 84 sec latency 68.973 ms oleg334-client.virtnet: 1 6459 0.34 MB/sec execute 85 sec latency 71.653 ms oleg334-client.virtnet: 1 6567 0.34 MB/sec execute 86 sec latency 80.775 ms oleg334-client.virtnet: 1 6884 0.36 MB/sec execute 87 sec latency 58.311 ms oleg334-client.virtnet: 1 7085 0.39 MB/sec execute 88 sec latency 60.879 ms oleg334-client.virtnet: 1 7196 0.39 MB/sec execute 89 sec latency 63.268 ms oleg334-client.virtnet: 1 7312 0.40 MB/sec execute 90 sec latency 68.656 ms oleg334-client.virtnet: 1 7376 0.39 MB/sec execute 91 sec latency 84.588 ms oleg334-client.virtnet: 1 7449 0.39 MB/sec execute 92 sec latency 84.683 ms oleg334-client.virtnet: 1 7519 0.39 MB/sec execute 93 sec latency 81.610 ms oleg334-client.virtnet: 1 7590 0.38 MB/sec execute 94 sec latency 53.883 ms oleg334-client.virtnet: 1 7657 0.38 MB/sec execute 95 sec latency 95.162 ms oleg334-client.virtnet: 1 7768 0.38 MB/sec execute 96 sec latency 64.007 ms oleg334-client.virtnet: 1 7824 0.38 MB/sec execute 97 sec latency 87.334 ms oleg334-client.virtnet: 1 7882 0.37 MB/sec execute 98 sec latency 98.952 ms oleg334-client.virtnet: 1 7957 0.37 MB/sec execute 99 sec latency 67.266 ms oleg334-client.virtnet: 1 8021 0.37 MB/sec execute 100 sec latency 87.993 ms oleg334-client.virtnet: 1 8092 0.37 MB/sec execute 101 sec latency 76.899 ms oleg334-client.virtnet: 1 8235 0.37 MB/sec execute 102 sec latency 59.756 ms oleg334-client.virtnet: 1 8323 0.37 MB/sec execute 103 sec latency 50.619 ms oleg334-client.virtnet: 1 8493 0.40 MB/sec execute 104 sec latency 73.285 ms oleg334-client.virtnet: 1 8560 0.39 MB/sec execute 105 sec latency 70.023 ms oleg334-client.virtnet: 1 8670 0.39 MB/sec execute 106 sec latency 63.893 ms oleg334-client.virtnet: 1 8751 0.39 MB/sec execute 107 sec latency 54.437 ms oleg334-client.virtnet: 1 8825 0.38 MB/sec execute 108 sec latency 76.681 ms oleg334-client.virtnet: 1 8899 0.38 MB/sec execute 109 sec latency 97.980 ms oleg334-client.virtnet: 1 8997 0.38 MB/sec execute 110 sec latency 57.993 ms oleg334-client.virtnet: 1 9113 0.38 MB/sec execute 111 sec latency 63.853 ms oleg334-client.virtnet: 1 9209 0.37 MB/sec execute 112 sec latency 66.677 ms oleg334-client.virtnet: 1 9298 0.37 MB/sec execute 113 sec latency 59.211 ms oleg334-client.virtnet: 1 9457 0.38 MB/sec execute 114 sec latency 51.301 ms oleg334-client.virtnet: 1 9538 0.38 MB/sec execute 115 sec latency 73.200 ms oleg334-client.virtnet: 1 9661 0.38 MB/sec execute 116 sec latency 54.984 ms oleg334-client.virtnet: 1 9864 0.40 MB/sec execute 117 sec latency 58.839 ms oleg334-client.virtnet: 1 10024 0.41 MB/sec execute 118 sec latency 56.313 ms oleg334-client.virtnet: 1 10279 0.41 MB/sec execute 119 sec latency 62.334 ms oleg334-client.virtnet: 1 cleanup 120 sec oleg334-client.virtnet: 1 cleanup 121 sec oleg334-client.virtnet: 0 cleanup 122 sec oleg334-client.virtnet: oleg334-client.virtnet: Operation Count AvgLat MaxLat oleg334-client.virtnet: ---------------------------------------- oleg334-client.virtnet: NTCreateX 1558 32.504 155.515 oleg334-client.virtnet: Close 1172 7.691 43.868 oleg334-client.virtnet: Rename 69 45.566 112.414 oleg334-client.virtnet: Unlink 286 12.473 33.323 oleg334-client.virtnet: Qpathinfo 1479 32.293 35508.409 oleg334-client.virtnet: Qfileinfo 271 0.788 11.849 oleg334-client.virtnet: Qfsinfo 250 0.868 15.822 oleg334-client.virtnet: Sfileinfo 128 31.765 81.087 oleg334-client.virtnet: Find 579 1.989 50.744 oleg334-client.virtnet: WriteX 798 6.012 61.014 oleg334-client.virtnet: ReadX 2617 0.053 21.041 oleg334-client.virtnet: LockX 6 10.503 36.891 oleg334-client.virtnet: UnlockX 6 8.768 10.778 oleg334-client.virtnet: Flush 105 37.425 108.426 oleg334-client.virtnet: oleg334-client.virtnet: Throughput 0.414283 MB/sec 1 clients 1 procs max_latency=35508.427 ms oleg334-client.virtnet: stopping dbench on /mnt/lustre/d70b.replay-single/oleg334-client.virtnet at Fri Sep 4 22:59:23 EDT 2026 with return code 0 oleg334-client.virtnet: clean dbench files on /mnt/lustre/d70b.replay-single/oleg334-client.virtnet oleg334-client.virtnet: /mnt/lustre/d70b.replay-single/oleg334-client.virtnet /mnt/lustre/d70b.replay-single/oleg334-client.virtnet oleg334-client.virtnet: removed directory 'clients/client0/~dmtmp/PWRPNT' oleg334-client.virtnet: removed directory 'clients/client0/~dmtmp/WORD' oleg334-client.virtnet: removed directory 'clients/client0/~dmtmp/ACCESS' oleg334-client.virtnet: removed directory 'clients/client0/~dmtmp/WORDPRO' oleg334-client.virtnet: removed directory 'clients/client0/~dmtmp/EXCEL' oleg334-client.virtnet: removed directory 'clients/client0/~dmtmp/PARADOX' oleg334-client.virtnet: removed directory 'clients/client0/~dmtmp/SEED' oleg334-client.virtnet: removed directory 'clients/client0/~dmtmp/PM' oleg334-client.virtnet: removed directory 'clients/client0/~dmtmp/COREL' oleg334-client.virtnet: removed directory 'clients/client0/~dmtmp' oleg334-client.virtnet: removed directory 'clients/client0' oleg334-client.virtnet: removed directory 'clients' oleg334-client.virtnet: removed 'client.txt' oleg334-client.virtnet: /mnt/lustre/d70b.replay-single/oleg334-client.virtnet oleg334-client.virtnet: dbench successfully finished