== replay-single test 70b: dbench 2mdts recovery; 1 clients ========================================================== 23:07:15 (1781233635) Starting client oleg432-client.virtnet: -o user_xattr,flock 192.168.204.132@tcp:/lustre /mnt/lustre Started clients oleg432-client.virtnet: 192.168.204.132@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) striped dir -i0 -c2 -H all_char /mnt/lustre/d70b.replay-single + MISSING_DBENCH_OK= + PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin::/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests:/sbin:/usr/sbin:/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests/: + DBENCH_LIB= + TESTSUITE=replay-single + TESTNAME=test_70b + MOUNT=/mnt/lustre ++ hostname + DIR=/mnt/lustre/d70b.replay-single/oleg432-client.virtnet + LCTL=/home/green/git/lustre-release/lustre/utils/lctl + rundbench 1 -t 120 dbench: no process found dbench: no process found dbench: no process found Started rundbench load pid=133987 ... UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2400 1268904 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2036 1269268 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 3640 3598276 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 29236 3572676 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 32876 7170952 1% /mnt/lustre test_70b fail mds1 1 times Failing mds1 on oleg432-server Stopping /mnt/lustre-mds1 (opts:) on oleg432-server 23:07:48 (1781233668) shut down facet: mds1 facet_host: oleg432-server facet_failover_host: oleg432-server oleg432-client.virtnet: looking for dbench program oleg432-client.virtnet: /usr/bin/dbench oleg432-client.virtnet: creating output directory /mnt/lustre/d70b.replay-single/oleg432-client.virtnet oleg432-client.virtnet: mkdir: created directory '/mnt/lustre/d70b.replay-single/oleg432-client.virtnet' oleg432-client.virtnet: found dbench client file /usr/share/dbench/client.txt oleg432-client.virtnet: '/usr/share/dbench/client.txt' -> 'client.txt' oleg432-client.virtnet: running 'dbench 1 -t 120' on /mnt/lustre/d70b.replay-single/oleg432-client.virtnet at Thu Jun 11 23:07:20 EDT 2026 oleg432-client.virtnet: waiting for dbench pid 134025 oleg432-client.virtnet: dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 oleg432-client.virtnet: oleg432-client.virtnet: Running for 120 seconds with load 'client.txt' and minimum warmup 24 secs oleg432-client.virtnet: failed to create barrier semaphore oleg432-client.virtnet: 0 of 1 processes prepared for launch 0 sec oleg432-client.virtnet: 1 of 1 processes prepared for launch 0 sec oleg432-client.virtnet: releasing clients oleg432-client.virtnet: 1 111 5.24 MB/sec warmup 1 sec latency 90.161 ms oleg432-client.virtnet: 1 158 3.27 MB/sec warmup 2 sec latency 129.945 ms oleg432-client.virtnet: 1 196 2.61 MB/sec warmup 3 sec latency 233.086 ms oleg432-client.virtnet: 1 238 2.28 MB/sec warmup 4 sec latency 150.586 ms oleg432-client.virtnet: 1 306 2.24 MB/sec warmup 5 sec latency 94.032 ms oleg432-client.virtnet: 1 356 2.13 MB/sec warmup 6 sec latency 135.631 ms oleg432-client.virtnet: 1 397 2.01 MB/sec warmup 7 sec latency 169.011 ms oleg432-client.virtnet: 1 449 1.96 MB/sec warmup 8 sec latency 150.550 ms oleg432-client.virtnet: 1 500 1.94 MB/sec warmup 9 sec latency 154.932 ms oleg432-client.virtnet: 1 526 1.81 MB/sec warmup 10 sec latency 225.339 ms oleg432-client.virtnet: 1 616 1.90 MB/sec warmup 11 sec latency 145.618 ms oleg432-client.virtnet: 1 674 1.76 MB/sec warmup 12 sec latency 93.427 ms oleg432-client.virtnet: 1 702 1.63 MB/sec warmup 13 sec latency 222.023 ms oleg432-client.virtnet: 1 721 1.52 MB/sec warmup 14 sec latency 544.241 ms oleg432-client.virtnet: 1 721 1.42 MB/sec warmup 15 sec latency 1544.513 ms oleg432-client.virtnet: 1 740 1.33 MB/sec warmup 16 sec latency 1955.316 ms oleg432-client.virtnet: 1 789 1.26 MB/sec warmup 17 sec latency 132.953 ms oleg432-client.virtnet: 1 848 1.20 MB/sec warmup 18 sec latency 83.563 ms oleg432-client.virtnet: 1 885 1.14 MB/sec warmup 19 sec latency 183.190 ms oleg432-client.virtnet: 1 904 1.08 MB/sec warmup 20 sec latency 411.402 ms oleg432-client.virtnet: 1 949 1.04 MB/sec warmup 21 sec latency 153.859 ms oleg432-client.virtnet: 1 981 0.99 MB/sec warmup 22 sec latency 143.113 ms oleg432-client.virtnet: 1 1050 0.99 MB/sec warmup 23 sec latency 100.970 ms oleg432-client.virtnet: 1 1126 0.02 MB/sec execute 1 sec latency 787.580 ms oleg432-client.virtnet: 1 1126 0.01 MB/sec execute 2 sec latency 1787.957 ms oleg432-client.virtnet: 1 1126 0.01 MB/sec execute 3 sec latency 2788.326 ms oleg432-client.virtnet: 1 1126 0.00 MB/sec execute 4 sec latency 3788.739 ms oleg432-client.virtnet: 1 1126 0.00 MB/sec execute 5 sec latency 4789.750 ms oleg432-client.virtnet: 1 1126 0.00 MB/sec execute 6 sec latency 5790.564 ms oleg432-client.virtnet: 1 1126 0.00 MB/sec execute 7 sec latency 6791.055 ms oleg432-client.virtnet: 1 1126 0.00 MB/sec execute 8 sec latency 7791.252 ms oleg432-client.virtnet: 1 1126 0.00 MB/sec execute 9 sec latency 8791.521 ms oleg432-client.virtnet: 1 1126 0.00 MB/sec execute 10 sec latency 9Failover mds1 to oleg432-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg432-server: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg432-client: oleg432-server: ssh exited with exit code 1 Started lustre-MDT0000 23:08:24 (1781233704) targets are mounted 23:08:25 (1781233705) facet_failover done oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2348 1268956 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 1996 1269308 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12112 3593916 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 37436 3569556 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49548 7163472 1% /mnt/lustre 792.338 ms oleg432-client.virtnet: 1 1126 0.00 MB/sec execute 11 sec latency 10793.002 ms oleg432-client.virtnet: 1 1126 0.00 MB/sec execute 12 sec latency 11794.089 ms oleg432-client.virtnet: 1 1126 0.00 MB/sec execute 13 sec latency 12794.698 ms oleg432-client.virtnet: 1 1126 0.00 MB/sec execute 14 sec latency 13795.060 ms oleg432-client.virtnet: 1 1126 0.00 MB/sec execute 15 sec latency 14795.263 ms oleg432-client.virtnet: 1 1126 0.00 MB/sec execute 16 sec latency 15795.704 ms oleg432-client.virtnet: 1 1126 0.00 MB/sec execute 17 sec latency 16796.008 ms oleg432-client.virtnet: 1 1126 0.00 MB/sec execute 18 sec latency 17797.490 ms oleg432-client.virtnet: 1 1126 0.00 MB/sec execute 19 sec latency 18798.208 ms oleg432-client.virtnet: 1 1126 0.00 MB/sec execute 20 sec latency 19800.795 ms oleg432-client.virtnet: 1 1126 0.00 MB/sec execute 21 sec latency 20801.975 ms oleg432-client.virtnet: 1 1126 0.00 MB/sec execute 22 sec latency 21802.913 ms oleg432-client.virtnet: 1 1126 0.00 MB/sec execute 23 sec latency 22803.224 ms oleg432-client.virtnet: 1 1126 0.00 MB/sec execute 24 sec latency 23803.853 ms oleg432-client.virtnet: 1 1126 0.00 MB/sec execute 25 sec latency 24804.959 ms oleg432-client.virtnet: 1 1126 0.00 MB/sec execute 26 sec latency 25805.233 ms oleg432-client.virtnet: 1 1126 0.00 MB/sec execute 27 sec latency 26812.968 ms oleg432-client.virtnet: 1 1126 0.00 MB/sec execute 28 sec latency 27814.647 ms oleg432-client.virtnet: 1 1126 0.00 MB/sec execute 29 sec latency 28814.937 ms oleg432-client.virtnet: 1 1126 0.00 MB/sec execute 30 sec latency 29815.438 ms oleg432-client.virtnet: 1 1126 0.00 MB/sec execute 31 sec latency 30816.235 ms oleg432-client.virtnet: 1 1126 0.00 MB/sec execute 32 sec latency 31819.136 ms oleg432-client.virtnet: 1 1126 0.00 MB/sec execute 33 sec latency 32820.962 ms oleg432-client.virtnet: 1 1126 0.00 MB/sec execute 34 sec latency 33821.212 ms oleg432-client.virtnet: 1 1126 0.00 MB/sec execute 35 sec latency 34821.479 ms oleg432-client.virtnet: 1 1126 0.00 MB/sec execute 36 sec latency 35821.977 ms oleg432-client.virtnet: 1 1126 0.00 MB/sec execute 37 sec latency 36822.853 ms oleg432-client.virtnet: 1 1126 0.00 MB/sec execute 38 sec latency 37823.203 ms oleg432-client.virtnet: 1 1147 0.00 MB/sec execute 39 sec latency 38433.621 ms oleg432-client.virtnet: 1 1211 0.00 MB/sec execute 40 sec latency 79.411 ms oleg432-client.virtnet: 1 1268 0.00 MB/sec execute 41 sec latency 71.770 ms oleg432-client.virtnet: 1 1412 0.07 MB/sec execute 42 sec latency 72.201 ms oleg432-client.virtnet: 1 1474 0.07 MB/sec execute 43 sec latency 98.115 ms oleg432-client.virtnet: 1 1537 0.07 MB/sec execute 44 sec latency 80.176 ms oleg432-client.virtnet: 1 1619 0.07 MB/sec execute 45 sec latency 52.394 ms oleg432-client.virtnet: 1 1686 0.07 MB/sec execute 46 sec latency 73.088 ms oleg432-client.virtnet: 1 1762 0.07 MB/sec execute 47 sec latency 98.707 ms oleg432-client.virtnet: 1 1840 0.07 MB/sec execute 48 sec latency 100.769 ms oleg432-client.virtnet: 1 1923 0.07 MB/sec execute 49 sec latency 96.637 ms oleg432-client.virtnet: 1 2022 0.07 MB/sec execute 50 sec latency 71.921 ms oleg432-client.virtnet: 1 2126 0.07 MB/sec execute 51 sec latency 71.537 ms oleg432-client.virtnet: 1 2212 0.07 MB/sec execute 52 sec latency 53.683 ms oleg432-client.virtnet: 1 2345 0.09 MB/sec execute 53 sec latency 67.071 ms oleg432-client.virtnet: 1 2388 0.09 MB/sec execute 54 sec latency 79.test_70b fail mds2 2 times Failing mds2 on oleg432-server Stopping /mnt/lustre-mds2 (opts:) on oleg432-server 23:08:54 (1781233734) shut down facet: mds2 facet_host: oleg432-server facet_failover_host: oleg432-server Failover mds2 to oleg432-server mount facets: mds2 Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg432-server: oleg432-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg432-client: oleg432-server: ssh exited with exit code 1 613 ms oleg432-client.virtnet: 1 2442 0.09 MB/sec execute 55 sec latency 97.895 ms oleg432-client.virtnet: 1 2508 0.10 MB/sec execute 56 sec latency 82.842 ms oleg432-client.virtnet: 1 2604 0.11 MB/sec execute 57 sec latency 95.402 ms oleg432-client.virtnet: 1 2776 0.15 MB/sec execute 58 sec latency 80.308 ms oleg432-client.virtnet: 1 2934 0.18 MB/sec execute 59 sec latency 83.098 ms oleg432-client.virtnet: 1 3026 0.18 MB/sec execute 60 sec latency 76.438 ms oleg432-client.virtnet: 1 3277 0.20 MB/sec execute 61 sec latency 129.960 ms oleg432-client.virtnet: 1 3498 0.25 MB/sec execute 62 sec latency 73.065 ms oleg432-client.virtnet: 1 3590 0.25 MB/sec execute 63 sec latency 62.444 ms oleg432-client.virtnet: 1 3663 0.25 MB/sec execute 64 sec latency 109.235 ms oleg432-client.virtnet: 1 3716 0.24 MB/sec execute 65 sec latency 77.945 ms oleg432-client.virtnet: 1 3792 0.25 MB/sec execute 66 sec latency 91.405 ms oleg432-client.virtnet: 1 3852 0.25 MB/sec execute 67 sec latency 98.476 ms oleg432-client.virtnet: 1 3923 0.25 MB/sec execute 68 sec latency 76.509 ms oleg432-client.virtnet: 1 3975 0.25 MB/sec execute 69 sec latency 91.356 ms oleg432-client.virtnet: 1 4043 0.24 MB/sec execute 70 sec latency 97.777 ms oleg432-client.virtnet: 1 4117 0.24 MB/sec execute 71 sec latency 111.053 ms oleg432-client.virtnet: 1 4249 0.24 MB/sec execute 72 sec latency 43.636 ms oleg432-client.virtnet: 1 4311 0.24 MB/sec execute 73 sec latency 81.779 ms oleg432-client.virtnet: 1 4370 0.24 MB/sec execute 74 sec latency 116.947 ms oleg432-client.virtnet: 1 4458 0.24 MB/sec execute 75 sec latency 95.856 ms oleg432-client.virtnet: 1 4536 0.24 MB/sec execute 76 sec latency 88.523 ms oleg432-client.virtnet: 1 4644 0.25 MB/sec execute 77 sec latency 62.268 ms oleg432-client.virtnet: 1 4746 0.25 MB/sec execute 78 sec latency 55.221 ms oleg432-client.virtnet: 1 4829 0.24 MB/sec execute 79 sec latency 67.302 ms oleg432-client.virtnet: 1 4978 0.28 MB/sec execute 80 sec latency 66.088 ms oleg432-client.virtnet: 1 5036 0.27 MB/sec execute 81 sec latency 83.097 ms oleg432-client.virtnet: 1 5103 0.27 MB/sec execute 82 sec latency 93.982 ms oleg432-client.virtnet: 1 5143 0.27 MB/sec execute 83 sec latency 101.662 ms oleg432-client.virtnet: 1 5197 0.27 MB/sec execute 84 sec latency 116.725 ms oleg432-client.virtnet: 1 5254 0.26 MB/sec execute 85 sec latency 83.311 ms oleg432-client.virtnet: 1 5288 0.26 MB/sec execute 86 sec latency 106.598 ms oleg432-client.virtnet: 1 5348 0.26 MB/sec execute 87 sec latency 92.473 ms oleg432-client.virtnet: 1 5395 0.26 MB/sec execute 88 sec latency 108.248 ms oleg432-client.virtnet: 1 5496 0.25 MB/sec execute 89 sec latency 66.028 ms oleg432-client.virtnet: 1 5554 0.25 MB/sec execute 90 sec latency 101.677 ms oleg432-client.virtnet: 1 5622 0.25 MB/sec execute 91 sec latency 86.549 ms oleg432-client.virtnet: 1 5707 0.25 MB/sec execute 92 sec latency 72.133 ms oleg432-client.virtnet: 1 5761 0.25 MB/sec execute 93 sec latency 95.045 ms oleg432-client.virtnet: 1 5859 0.25 MB/sec execute 94 sec latency 94.294 ms oleg432-client.virtnet: 1 5927 0.25 MB/sec execute 95 sec latency 93.309 ms oleg432-client.virtnet: 1 5988 0.25 MB/sec execute 96 sec latency 82.905 ms oleg432-client.virtnet: 1 6057 0.26 MB/sec execute 97 sec latency 77.505 ms oleg432-client.virtnet: 1 6160 0.26 MB/sec execute 98 sec latency 78.378 ms oleg432-client.virtnet: 1 6329 0.28 MB/sec execute 99 sec latStarted lustre-MDT0001 23:09:24 (1781233764) targets are mounted 23:09:24 (1781233764) facet_failover done oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec ency 106.137 ms oleg432-client.virtnet: 1 6484 0.30 MB/sec execute 100 sec latency 113.710 ms oleg432-client.virtnet: 1 6585 0.29 MB/sec execute 101 sec latency 83.218 ms oleg432-client.virtnet: 1 6829 0.31 MB/sec execute 102 sec latency 103.419 ms oleg432-client.virtnet: 1 6932 0.31 MB/sec execute 103 sec latency 71.747 ms oleg432-client.virtnet: 1 7105 0.33 MB/sec execute 104 sec latency 97.542 ms oleg432-client.virtnet: 1 7191 0.33 MB/sec execute 105 sec latency 54.306 ms oleg432-client.virtnet: 1 7261 0.33 MB/sec execute 106 sec latency 76.554 ms oleg432-client.virtnet: 1 7326 0.34 MB/sec execute 107 sec latency 114.805 ms oleg432-client.virtnet: 1 7376 0.33 MB/sec execute 108 sec latency 177.061 ms oleg432-client.virtnet: 1 7441 0.33 MB/sec execute 109 sec latency 72.146 ms oleg432-client.virtnet: 1 7546 0.33 MB/sec execute 110 sec latency 70.901 ms oleg432-client.virtnet: 1 7603 0.33 MB/sec execute 111 sec latency 105.869 ms oleg432-client.virtnet: 1 7697 0.32 MB/sec execute 112 sec latency 64.990 ms oleg432-client.virtnet: 1 7780 0.32 MB/sec execute 113 sec latency 80.953 ms oleg432-client.virtnet: 1 7831 0.32 MB/sec execute 114 sec latency 85.424 ms oleg432-client.virtnet: 1 7892 0.32 MB/sec execute 115 sec latency 91.240 ms oleg432-client.virtnet: 1 7994 0.32 MB/sec execute 116 sec latency 73.664 ms oleg432-client.virtnet: 1 8060 0.32 MB/sec execute 117 sec latency 75.711 ms oleg432-client.virtnet: 1 8139 0.32 MB/sec execute 118 sec latency 79.220 ms oleg432-client.virtnet: 1 8258 0.32 MB/sec execute 119 sec latency 79.239 ms oleg432-client.virtnet: 1 cleanup 120 sec oleg432-client.virtnet: 1 cleanup 121 sec oleg432-client.virtnet: 0 cleanup 122 sec oleg432-client.virtnet: oleg432-client.virtnet: Operation Count AvgLat MaxLat oleg432-client.virtnet: ---------------------------------------- oleg432-client.virtnet: NTCreateX 1239 34.430 177.041 oleg432-client.virtnet: Close 912 9.265 35.493 oleg432-client.virtnet: Rename 54 52.339 129.882 oleg432-client.virtnet: Unlink 249 12.303 41.103 oleg432-client.virtnet: Qpathinfo 1152 9.713 106.103 oleg432-client.virtnet: Qfileinfo 197 0.879 7.506 oleg432-client.virtnet: Qfsinfo 207 1.060 31.683 oleg432-client.virtnet: Sfileinfo 107 395.823 38433.597 oleg432-client.virtnet: Find 435 2.314 54.919 oleg432-client.virtnet: WriteX 614 6.780 62.292 oleg432-client.virtnet: ReadX 1935 0.072 16.340 oleg432-client.virtnet: LockX 4 14.198 19.134 oleg432-client.virtnet: UnlockX 4 12.674 28.377 oleg432-client.virtnet: Flush 86 40.665 116.924 oleg432-client.virtnet: oleg432-client.virtnet: Throughput 0.321207 MB/sec 1 clients 1 procs max_latency=38433.621 ms oleg432-client.virtnet: stopping dbench on /mnt/lustre/d70b.replay-single/oleg432-client.virtnet at Thu Jun 11 23:09:46 EDT 2026 with return code 0 oleg432-client.virtnet: clean dbench files on /mnt/lustre/d70b.replay-single/oleg432-client.virtnet oleg432-client.virtnet: /mnt/lustre/d70b.replay-single/oleg432-client.virtnet /mnt/lustre/d70b.replay-single/oleg432-client.virtnet oleg432-client.virtnet: removed directory 'clients/client0/~dmtmp/SEED' oleg432-client.virtnet: removed directory 'clients/client0/~dmtmp/PM' oleg432-client.virtnet: removed directory 'clients/client0/~dmtmp/PWRPNT' oleg432-client.virtnet: removed directory 'clients/client0/~dmtmp/WORDPRO' oleg432-client.virtnet: removed directory 'clients/client0/~dmtmp/ACCESS' oleg432-client.virtnet: removed directory 'clients/client0/~dmtmp/PARADOX' oleg432-client.virtnet: removed directory 'clients/client0/~dmtmp/COREL' oleg432-client.virtnet: removed directory 'clients/client0/~dmtmp/EXCEL' oleg432-client.virtnet: removed directory 'clients/client0/~dmtmp/WORD' oleg432-client.virtnet: removed directory 'clients/client0/~dmtmp' oleg432-client.virtnet: removed directory 'clients/client0' oleg432-client.virtnet: removed directory 'clients' oleg432-client.virtnet: removed 'client.txt' oleg432-client.virtnet: /mnt/lustre/d70b.replay-single/oleg432-client.virtnet oleg432-client.virtnet: dbench successfully finished