== replay-single test 70b: dbench 1mdts recovery; 1 clients ========================================================== 11:30:04 (1780327804) Starting client oleg238-client.virtnet: -o user_xattr,flock 192.168.202.138@tcp:/lustre /mnt/lustre Started clients oleg238-client.virtnet: 192.168.202.138@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) + MISSING_DBENCH_OK= + PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin::/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests:/sbin:/usr/sbin:/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests/: + DBENCH_LIB= + TESTSUITE=replay-single + TESTNAME=test_70b + MOUNT=/mnt/lustre ++ hostname + DIR=/mnt/lustre/d70b.replay-single/oleg238-client.virtnet + LCTL=/home/green/git/lustre-release/lustre/utils/lctl + rundbench 1 -t 120 dbench: no process found Started rundbench load pid=126426 ... UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210304 2432 2205824 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 4096 3744768 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 26624 3727360 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 30720 7472128 1% /mnt/lustre test_70b fail mds1 1 times Failing mds1 on oleg238-server Stopping /mnt/lustre-mds1 (opts:) on oleg238-server 11:30:21 (1780327821) shut down facet: mds1 facet_host: oleg238-server facet_failover_host: oleg238-server Failover mds1 to oleg238-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg238-client.virtnet: looking for dbench program oleg238-client.virtnet: /usr/bin/dbench oleg238-client.virtnet: creating output directory /mnt/lustre/d70b.replay-single/oleg238-client.virtnet oleg238-client.virtnet: mkdir: created directory '/mnt/lustre/d70b.replay-single' oleg238-client.virtnet: mkdir: created directory '/mnt/lustre/d70b.replay-single/oleg238-client.virtnet' oleg238-client.virtnet: found dbench client file /usr/share/dbench/client.txt oleg238-client.virtnet: '/usr/share/dbench/client.txt' -> 'client.txt' oleg238-client.virtnet: running 'dbench 1 -t 120' on /mnt/lustre/d70b.replay-single/oleg238-client.virtnet at Mon Jun 1 11:30:06 EDT 2026 oleg238-client.virtnet: waiting for dbench pid 126451 oleg238-client.virtnet: dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 oleg238-client.virtnet: oleg238-client.virtnet: Running for 120 seconds with load 'client.txt' and minimum warmup 24 secs oleg238-client.virtnet: failed to create barrier semaphore oleg238-client.virtnet: 0 of 1 processes prepared for launch 0 sec oleg238-client.virtnet: 1 of 1 processes prepared for launch 0 sec oleg238-client.virtnet: releasing clients oleg238-client.virtnet: 1 139 6.02 MB/sec warmup 1 sec latency 94.839 ms oleg238-client.virtnet: 1 287 5.35 MB/sec warmup 2 sec latency 58.492 ms oleg238-client.virtnet: 1 466 5.42 MB/sec warmup 3 sec latency 36.172 ms oleg238-client.virtnet: 1 668 5.28 MB/sec warmup 4 sec latency 110.593 ms oleg238-client.virtnet: 1 778 4.27 MB/sec warmup 5 sec latency 48.468 ms oleg238-client.virtnet: 1 904 3.61 MB/sec warmup 6 sec latency 59.526 ms oleg238-client.virtnet: 1 1067 3.27 MB/sec warmup 7 sec latency 117.830 ms oleg238-client.virtnet: 1 1082 2.86 MB/sec warmup 8 sec latency 937.732 ms oleg238-client.virtnet: 1 1082 2.55 MB/sec warmup 9 sec latency 1938.373 ms oleg238-client.virtnet: 1 1082 2.29 MB/sec warmup 10 sec latency 2938.568 ms oleg238-client.virtnet: 1 1082 2.08 MB/sec warmup 11 sec latency 3938.776 ms oleg238-client.virtnet: 1 1082 1.91 MB/sec warmup 12 sec latency 4938.986 ms oleg238-client.virtnet: 1 1082 1.76 MB/sec warmup 13 sec latency 5939.615 ms oleg238-client.virtnet: 1 1082 1.64 MB/sec warmup 14 sec latency 6939.763 ms oleg238-client.virtnet: 1 1082 1.53 MB/sec warmup 15 sec latency 7939.956 ms oleg238-client.virtnet: 1 1082 1.43 MB/sec warmup 16 sec latency 8940.153 ms oleg238-client.virtnet: 1 1082 1.35 MB/sec warmup 17 sec latency 9940.447 ms oleg238-client.virtnet: 1 1082 1.27 MB/sec warmup 18 sec latency 10941.716 ms oleg238-client.virtnet: 1 1082 1.21 MB/sec warmup 19 sec latency 11941.920 ms oleg238-client.virtnet: 1 1082 1.15 MB/sec warmup 20 sec latency 12942.514 ms oleg238-client.virtnet: 1 1082 1.09 MB/sec warmup 21 sec latency 13942.807 ms oleg238-client.virtnet: 1 1082 1.04 MB/sec warmup 22 sec latency 14943.259 ms oleg238-client.virtnet: 1 1082 1.00 MB/sec warmup 23 sec latency 15944.535 ms oleg238-client.virtnet: 1 1082 0.00 MB/sec execute 1 sec latency 17945.456 ms oleg238-client.virtnet: 1 1082 0.00 MB/sec execute 2 sec latency 18945.642 ms oleg238-client.virtnet: 1 1082 0.00 MB/sec execute 3 sec latency 19946.055 ms oleg238-client.virtnet: 1 1082 0.00 MB/sec execute 4 sec latency 20946.311 ms oleg238-client.virtnet: 1 1082 0.00 MB/sec execute 5 sec latency 21946.721 ms oleg238-client.virtnet: 1 1082 0.00 MB/sec execute 6 sec latency 22951.062 ms oleg238-client.virtnet: 1 1082 0.00 MB/sec execute 7 sec latency 23951.244 ms oleg238-client.virtnet: 1 1082 0.00 MB/sec execute 8 sec latency 25107.555 ms oleg238-client.virtnet: 1 1082 0.00 MB/sec execute oleg238-server: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1 Started lustre-MDT0000 11:30:58 (1780327858) targets are mounted 11:30:58 (1780327858) facet_failover done oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210304 2560 2205696 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 12288 3743744 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 39936 3723264 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 52224 7467008 1% /mnt/lustre 9 sec latency 26107.818 ms oleg238-client.virtnet: 1 1082 0.00 MB/sec execute 10 sec latency 27108.081 ms oleg238-client.virtnet: 1 1082 0.00 MB/sec execute 11 sec latency 28108.473 ms oleg238-client.virtnet: 1 1082 0.00 MB/sec execute 12 sec latency 29109.172 ms oleg238-client.virtnet: 1 1082 0.00 MB/sec execute 13 sec latency 30109.360 ms oleg238-client.virtnet: 1 1082 0.00 MB/sec execute 14 sec latency 31109.561 ms oleg238-client.virtnet: 1 1082 0.00 MB/sec execute 15 sec latency 32110.198 ms oleg238-client.virtnet: 1 1082 0.00 MB/sec execute 16 sec latency 33110.404 ms oleg238-client.virtnet: 1 1082 0.00 MB/sec execute 17 sec latency 34113.935 ms oleg238-client.virtnet: 1 1082 0.00 MB/sec execute 18 sec latency 35115.270 ms oleg238-client.virtnet: 1 1082 0.00 MB/sec execute 19 sec latency 36115.667 ms oleg238-client.virtnet: 1 1082 0.00 MB/sec execute 20 sec latency 37117.589 ms oleg238-client.virtnet: 1 1082 0.00 MB/sec execute 21 sec latency 38119.249 ms oleg238-client.virtnet: 1 1082 0.00 MB/sec execute 22 sec latency 39404.317 ms oleg238-client.virtnet: 1 1082 0.00 MB/sec execute 23 sec latency 40407.365 ms oleg238-client.virtnet: 1 1082 0.00 MB/sec execute 24 sec latency 41407.584 ms oleg238-client.virtnet: 1 1082 0.00 MB/sec execute 26 sec latency 42511.493 ms oleg238-client.virtnet: 1 1082 0.00 MB/sec execute 27 sec latency 43511.704 ms oleg238-client.virtnet: 1 1082 0.00 MB/sec execute 28 sec latency 44512.030 ms oleg238-client.virtnet: 1 1082 0.00 MB/sec execute 29 sec latency 45512.258 ms oleg238-client.virtnet: 1 1082 0.00 MB/sec execute 30 sec latency 46517.973 ms oleg238-client.virtnet: 1 1082 0.00 MB/sec execute 31 sec latency 47518.217 ms oleg238-client.virtnet: 1 1082 0.00 MB/sec execute 32 sec latency 48552.190 ms oleg238-client.virtnet: 1 1082 0.00 MB/sec execute 33 sec latency 49552.393 ms oleg238-client.virtnet: 1 1082 0.00 MB/sec execute 34 sec latency 50554.093 ms oleg238-client.virtnet: 1 1082 0.00 MB/sec execute 35 sec latency 51554.486 ms oleg238-client.virtnet: 1 1082 0.00 MB/sec execute 36 sec latency 52555.482 ms oleg238-client.virtnet: 1 1117 0.00 MB/sec execute 37 sec latency 52689.985 ms oleg238-client.virtnet: 1 1145 0.00 MB/sec execute 38 sec latency 195.901 ms oleg238-client.virtnet: 1 1177 0.00 MB/sec execute 39 sec latency 366.802 ms oleg238-client.virtnet: 1 1224 0.00 MB/sec execute 40 sec latency 88.695 ms oleg238-client.virtnet: 1 1268 0.01 MB/sec execute 41 sec latency 168.798 ms oleg238-client.virtnet: 1 1410 0.07 MB/sec execute 42 sec latency 84.596 ms oleg238-client.virtnet: 1 1496 0.07 MB/sec execute 43 sec latency 104.856 ms oleg238-client.virtnet: 1 1538 0.07 MB/sec execute 44 sec latency 127.099 ms oleg238-client.virtnet: 1 1588 0.07 MB/sec execute 45 sec latency 96.596 ms oleg238-client.virtnet: 1 1632 0.07 MB/sec execute 46 sec latency 165.378 ms oleg238-client.virtnet: 1 1678 0.07 MB/sec execute 47 sec latency 324.496 ms oleg238-client.virtnet: 1 1736 0.07 MB/sec execute 48 sec latency 98.395 ms oleg238-client.virtnet: 1 1801 0.07 MB/sec execute 49 sec latency 98.503 ms oleg238-client.virtnet: 1 1861 0.07 MB/sec execute 50 sec latency 133.806 ms oleg238-client.virtnet: 1 1905 0.07 MB/sec execute 51 sec latency 302.086 ms oleg238-client.virtnet: 1 1982 0.07 MB/sec execute 52 sec latency 88.494 ms oleg238-client.virtnet: 1 2062 0.07 MB/sec execute 53 sec latency 103.767 ms oleg238-client.virtnet: 1 2138 0.07 MB/sec execute test_70b fail mds1 2 times Failing mds1 on oleg238-server Stopping /mnt/lustre-mds1 (opts:) on oleg238-server 11:31:36 (1780327896) shut down facet: mds1 facet_host: oleg238-server facet_failover_host: oleg238-server Failover mds1 to oleg238-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 54 sec latency 133.446 ms oleg238-client.virtnet: 1 2191 0.07 MB/sec execute 55 sec latency 115.067 ms oleg238-client.virtnet: 1 2294 0.09 MB/sec execute 56 sec latency 83.838 ms oleg238-client.virtnet: 1 2358 0.09 MB/sec execute 57 sec latency 303.241 ms oleg238-client.virtnet: 1 2416 0.09 MB/sec execute 58 sec latency 180.780 ms oleg238-client.virtnet: 1 2453 0.09 MB/sec execute 59 sec latency 279.661 ms oleg238-client.virtnet: 1 2503 0.10 MB/sec execute 60 sec latency 255.653 ms oleg238-client.virtnet: 1 2571 0.10 MB/sec execute 61 sec latency 175.225 ms oleg238-client.virtnet: 1 2697 0.14 MB/sec execute 62 sec latency 207.594 ms oleg238-client.virtnet: 1 2879 0.15 MB/sec execute 63 sec latency 229.366 ms oleg238-client.virtnet: 1 2879 0.15 MB/sec execute 64 sec latency 1053.017 ms oleg238-client.virtnet: 1 2879 0.14 MB/sec execute 65 sec latency 2053.212 ms oleg238-client.virtnet: 1 2879 0.14 MB/sec execute 66 sec latency 3053.607 ms oleg238-client.virtnet: 1 2879 0.14 MB/sec execute 67 sec latency 4054.005 ms oleg238-client.virtnet: 1 2879 0.14 MB/sec execute 68 sec latency 5056.395 ms oleg238-client.virtnet: 1 2879 0.13 MB/sec execute 69 sec latency 6058.992 ms oleg238-client.virtnet: 1 2879 0.13 MB/sec execute 70 sec latency 7059.254 ms oleg238-client.virtnet: 1 2879 0.13 MB/sec execute 71 sec latency 8059.752 ms oleg238-client.virtnet: 1 2879 0.13 MB/sec execute 72 sec latency 9064.716 ms oleg238-client.virtnet: 1 2879 0.13 MB/sec execute 73 sec latency 10065.538 ms oleg238-client.virtnet: 1 2879 0.13 MB/sec execute 74 sec latency 11066.876 ms oleg238-client.virtnet: 1 2879 0.12 MB/sec execute 75 sec latency 12067.100 ms oleg238-client.virtnet: 1 2879 0.12 MB/sec execute 76 sec latency 13067.318 ms oleg238-client.virtnet: 1 2879 0.12 MB/sec execute 77 sec latency 14068.379 ms oleg238-client.virtnet: 1 2879 0.12 MB/sec execute 78 sec latency 15068.569 ms oleg238-client.virtnet: 1 2879 0.12 MB/sec execute 79 sec latency 16068.778 ms oleg238-client.virtnet: 1 2879 0.12 MB/sec execute 80 sec latency 17070.556 ms oleg238-client.virtnet: 1 2879 0.11 MB/sec execute 81 sec latency 18072.991 ms oleg238-client.virtnet: 1 2879 0.11 MB/sec execute 82 sec latency 19073.320 ms oleg238-client.virtnet: 1 2879 0.11 MB/sec execute 83 sec latency 20076.213 ms oleg238-client.virtnet: 1 2879 0.11 MB/sec execute 84 sec latency 21076.514 ms oleg238-client.virtnet: 1 2879 0.11 MB/sec execute 85 sec latency 22078.398 ms oleg238-client.virtnet: 1 2879 0.11 MB/sec execute 86 sec latency 23078.630 ms oleg238-client.virtnet: 1 2879 0.11 MB/sec execute 87 sec latency 24079.077 ms oleg238-client.virtnet: 1 2879 0.11 MB/sec execute 88 sec latency 25079.323 ms oleg238-client.virtnet: 1 2879 0.10 MB/sec execute 89 sec latency 26079.527 ms oleg238-client.virtnet: 1 2879 0.10 MB/sec execute 90 sec latency 27080.052 ms oleg238-client.virtnet: 1 2879 0.10 MB/sec execute 91 sec latency 28080.436 ms oleg238-client.virtnet: 1 2879 0.10 MB/sec execute 92 sec latency 29080.691 ms oleg238-client.virtnet: 1 2879 0.10 MB/sec execute 93 sec latency 30080.942 ms oleg238-client.virtnet: 1 2879 0.10 MB/sec execute 94 sec latency 31082.425 ms oleg238-client.virtnet: 1 2879 0.10 MB/sec execute 95 sec latency 32082.673 ms oleg238-client.virtnet: 1 2879 0.10 MB/sec execute 96 sec latency 33082.895 ms oleg238-client.virtnet: 1 2879 0.10 MB/sec execute 97 sec latency 34085.354 ms oleg238-client.virtnet: 1 2879 0.09 MB/secoleg238-server: oleg238-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1 Started lustre-MDT0000 11:32:12 (1780327932) targets are mounted 11:32:12 (1780327932) facet_failover done oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec execute 98 sec latency 35085.519 ms oleg238-client.virtnet: 1 2879 0.09 MB/sec execute 99 sec latency 36087.818 ms oleg238-client.virtnet: 1 2879 0.09 MB/sec execute 100 sec latency 37088.717 ms oleg238-client.virtnet: 1 2879 0.09 MB/sec execute 101 sec latency 38088.913 ms oleg238-client.virtnet: 1 2879 0.09 MB/sec execute 102 sec latency 39089.958 ms oleg238-client.virtnet: 1 2880 0.09 MB/sec execute 103 sec latency 40075.318 ms oleg238-client.virtnet: 1 2929 0.10 MB/sec execute 104 sec latency 277.432 ms oleg238-client.virtnet: 1 3008 0.10 MB/sec execute 105 sec latency 169.184 ms oleg238-client.virtnet: 1 3210 0.11 MB/sec execute 106 sec latency 149.067 ms oleg238-client.virtnet: 1 3354 0.12 MB/sec execute 107 sec latency 89.014 ms oleg238-client.virtnet: 1 3504 0.14 MB/sec execute 108 sec latency 103.798 ms oleg238-client.virtnet: 1 3584 0.14 MB/sec execute 109 sec latency 94.180 ms oleg238-client.virtnet: 1 3685 0.14 MB/sec execute 110 sec latency 90.358 ms oleg238-client.virtnet: 1 3769 0.15 MB/sec execute 111 sec latency 92.226 ms oleg238-client.virtnet: 1 3836 0.15 MB/sec execute 112 sec latency 96.328 ms oleg238-client.virtnet: 1 3949 0.15 MB/sec execute 113 sec latency 40.770 ms oleg238-client.virtnet: 1 4059 0.15 MB/sec execute 114 sec latency 67.716 ms oleg238-client.virtnet: 1 4226 0.15 MB/sec execute 115 sec latency 47.895 ms oleg238-client.virtnet: 1 4260 0.15 MB/sec execute 116 sec latency 302.949 ms oleg238-client.virtnet: 1 4330 0.15 MB/sec execute 117 sec latency 138.682 ms oleg238-client.virtnet: 1 4418 0.15 MB/sec execute 118 sec latency 166.312 ms oleg238-client.virtnet: 1 4486 0.15 MB/sec execute 119 sec latency 163.376 ms oleg238-client.virtnet: 1 4552 0.15 MB/sec execute 120 sec latency 236.941 ms oleg238-client.virtnet: 1 cleanup 121 sec oleg238-client.virtnet: 1 cleanup 122 sec oleg238-client.virtnet: 0 cleanup 122 sec oleg238-client.virtnet: oleg238-client.virtnet: Operation Count AvgLat MaxLat oleg238-client.virtnet: ---------------------------------------- oleg238-client.virtnet: NTCreateX 622 117.777 52689.972 oleg238-client.virtnet: Close 458 8.839 55.288 oleg238-client.virtnet: Rename 28 69.466 366.780 oleg238-client.virtnet: Unlink 125 332.824 40075.293 oleg238-client.virtnet: Qpathinfo 592 11.085 306.688 oleg238-client.virtnet: Qfileinfo 98 0.784 9.131 oleg238-client.virtnet: Qfsinfo 105 1.654 27.879 oleg238-client.virtnet: Sfileinfo 52 38.456 175.208 oleg238-client.virtnet: Find 218 1.871 45.030 oleg238-client.virtnet: WriteX 305 6.170 66.642 oleg238-client.virtnet: ReadX 972 0.109 14.429 oleg238-client.virtnet: LockX 2 11.162 16.401 oleg238-client.virtnet: UnlockX 2 7.116 8.599 oleg238-client.virtnet: Flush 43 125.651 324.477 oleg238-client.virtnet: oleg238-client.virtnet: Throughput 0.151088 MB/sec 1 clients 1 procs max_latency=52689.985 ms oleg238-client.virtnet: stopping dbench on /mnt/lustre/d70b.replay-single/oleg238-client.virtnet at Mon Jun 1 11:32:32 EDT 2026 with return code 0 oleg238-client.virtnet: clean dbench files on /mnt/lustre/d70b.replay-single/oleg238-client.virtnet oleg238-client.virtnet: /mnt/lustre/d70b.replay-single/oleg238-client.virtnet /mnt/lustre/d70b.replay-single/oleg238-client.virtnet oleg238-client.virtnet: removed directory 'clients/client0/~dmtmp/PWRPNT' oleg238-client.virtnet: removed directory 'clients/client0/~dmtmp/COREL' oleg238-client.virtnet: removed directory 'clients/client0/~dmtmp/SEED' oleg238-client.virtnet: removed directory 'clients/client0/~dmtmp/WORD' oleg238-client.virtnet: removed directory 'clients/client0/~dmtmp/PARADOX' oleg238-client.virtnet: removed directory 'clients/client0/~dmtmp/ACCESS' oleg238-client.virtnet: removed directory 'clients/client0/~dmtmp/EXCEL' oleg238-client.virtnet: removed directory 'clients/client0/~dmtmp/WORDPRO' oleg238-client.virtnet: removed directory 'clients/client0/~dmtmp/PM' oleg238-client.virtnet: removed directory 'clients/client0/~dmtmp' oleg238-client.virtnet: removed directory 'clients/client0' oleg238-client.virtnet: removed directory 'clients' oleg238-client.virtnet: removed 'client.txt' oleg238-client.virtnet: /mnt/lustre/d70b.replay-single/oleg238-client.virtnet oleg238-client.virtnet: dbench successfully finished