== replay-single test 70b: dbench 2mdts recovery; 1 clients ========================================================== 06:42:05 (1744886525) Starting client oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre Started clients oleg318-client.virtnet: 192.168.203.118@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d70b.replay-single + MISSING_DBENCH_OK= + PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin::/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests:/sbin:/usr/sbin:/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests/: + DBENCH_LIB= + TESTSUITE=replay-single + TESTNAME=test_70b + MOUNT=/mnt/lustre ++ hostname + DIR=/mnt/lustre/d70b.replay-single/oleg318-client.virtnet + LCTL=/home/green/git/lustre-release/lustre/utils/lctl + rundbench 1 -t 300 dbench: no process found Started rundbench load pid=13504 ... UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2524 1285164 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2116 1285572 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1748 3593604 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 26144 3556156 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 27892 7149760 1% /mnt/lustre test_70b fail mds1 1 times Failing mds1 on oleg318-server Stopping /mnt/lustre-mds1 (opts:) on oleg318-server 06:42:12 (1744886532) shut down facet: mds1 facet_host: oleg318-server facet_failover_host: oleg318-server Failover mds1 to oleg318-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg318-server: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-MDT0000 06:42:26 (1744886546) targets are mounted 06:42:26 (1744886546) facet_failover done oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2480 1285208 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2076 1285612 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1588 3594340 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 26564 3568548 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 28152 7162888 1% /mnt/lustre test_70b fail mds2 2 times Failing mds2 on oleg318-server Stopping /mnt/lustre-mds2 (opts:) on oleg318-server 06:42:38 (1744886558) shut down facet: mds2 facet_host: oleg318-server facet_failover_host: oleg318-server oleg318-client.virtnet: looking for dbench program oleg318-client.virtnet: /usr/bin/dbench oleg318-client.virtnet: creating output directory /mnt/lustre/d70b.replay-single/oleg318-client.virtnet oleg318-client.virtnet: mkdir: created directory '/mnt/lustre/d70b.replay-single/oleg318-client.virtnet' oleg318-client.virtnet: found dbench client file /usr/share/dbench/client.txt oleg318-client.virtnet: '/usr/share/dbench/client.txt' -> 'client.txt' oleg318-client.virtnet: running 'dbench 1 -t 300' on /mnt/lustre/d70b.replay-single/oleg318-client.virtnet at Thu Apr 17 06:42:06 EDT 2025 oleg318-client.virtnet: waiting for dbench pid 13529 oleg318-client.virtnet: dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 oleg318-client.virtnet: oleg318-client.virtnet: Running for 300 seconds with load 'client.txt' and minimum warmup 60 secs oleg318-client.virtnet: failed to create barrier semaphore oleg318-client.virtnet: 0 of 1 processes prepared for launch 0 sec oleg318-client.virtnet: 1 of 1 processes prepared for launch 0 sec oleg318-client.virtnet: releasing clients oleg318-client.virtnet: 1 669 21.17 MB/sec warmup 1 sec latency 11.126 ms oleg318-client.virtnet: 1 1403 13.01 MB/sec warmup 2 sec latency 9.337 ms oleg318-client.virtnet: 1 2082 8.88 MB/sec warmup 3 sec latency 8.708 ms oleg318-client.virtnet: 1 2896 8.21 MB/sec warmup 4 sec latency 186.710 ms oleg318-client.virtnet: 1 3880 7.95 MB/sec warmup 5 sec latency 8.226 ms oleg318-client.virtnet: 1 4211 6.70 MB/sec warmup 6 sec latency 497.455 ms oleg318-client.virtnet: 1 4211 5.74 MB/sec warmup 7 sec latency 1497.608 ms oleg318-client.virtnet: 1 4211 5.02 MB/sec warmup 8 sec latency 2497.790 ms oleg318-client.virtnet: 1 4211 4.46 MB/sec warmup 9 sec latency 3497.955 ms oleg318-client.virtnet: 1 4211 4.02 MB/sec warmup 10 sec latency 4498.103 ms oleg318-client.virtnet: 1 4211 3.65 MB/sec warmup 11 sec latency 5498.242 ms oleg318-client.virtnet: 1 4211 3.35 MB/sec warmup 12 sec latency 6498.344 ms oleg318-client.virtnet: 1 4211 3.09 MB/sec warmup 13 sec latency 7498.614 ms oleg318-client.virtnet: 1 4211 2.87 MB/sec warmup 14 sec latency 8498.788 ms oleg318-client.virtnet: 1 4211 2.68 MB/sec warmup 15 sec latency 9498.959 ms oleg318-client.virtnet: 1 4211 2.51 MB/sec warmup 16 sec latency 10499.174 ms oleg318-client.virtnet: 1 4211 2.36 MB/sec warmup 17 sec latency 11499.356 ms oleg318-client.virtnet: 1 4211 2.23 MB/sec warmup 18 sec latency 12499.486 ms oleg318-client.virtnet: 1 4211 2.11 MB/sec warmup 19 sec latency 13499.618 ms oleg318-client.virtnet: 1 4211 2.01 MB/sec warmup 20 sec latency 14499.773 ms oleg318-client.virtnet: 1 4211 1.91 MB/sec warmup 21 sec latency 15499.927 ms oleg318-client.virtnet: 1 4211 1.83 MB/sec warmup 22 sec latency 16500.072 ms oleg318-client.virtnet: 1 4211 1.75 MB/sec warmup 23 sec latency 17500.229 ms oleg318-client.virtnet: 1 4279 1.68 MB/sec warmup 24 sec latency 18368.280 ms oleg318-client.virtnet: 1 4944 1.80 MB/sec warmup 25 sec latency 13.581 ms oleg318-client.virtnet: 1 5565 1.75 MB/sec warmup 26 sec latency 12.509 ms oleg318-client.virtnet: 1 6477 1.95 MB/sec warmup 27 sec latency 8.317 ms oleg318-client.virtnet: 1 7412 2.10 MB/sec warmup 28 sec latency 8.881 ms oleg318-client.virtnet: 1 8002 2.07 MB/sec warmup 29 sec latency 10.902 ms oleg318-client.virtnet: 1 8724 2.15 MB/sec warmup 30 sec latency 8.266 ms oleg318-client.virtnet: 1 9409 2.13 MB/sec warmup 31 sec latency 8.948 ms oleg318-client.virtnet: 1 10486 2.30 MB/sec warmup 32 sec latency 8.610 ms oleg318-client.virtnet: 1 11136 2.37 MB/sec warmup 33 sec latency 8.462 ms olFailover mds2 to oleg318-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg318-server: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-MDT0001 06:42:51 (1744886571) targets are mounted 06:42:51 (1744886571) facet_failover done oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2476 1285212 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12052 3593976 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 37584 3567796 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49636 7161772 1% /mnt/lustre test_70b fail mds1 3 times Failing mds1 on oleg318-server Stopping /mnt/lustre-mds1 (opts:) on oleg318-server 06:43:03 (1744886583) shut down facet: mds1 facet_host: oleg318-server facet_failover_host: oleg318-server Failover mds1 to oleg318-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg318-server: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-MDT0000 06:43:16 (1744886596) targets are mounted 06:43:16 (1744886596) facet_failover done oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2440 1285248 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12000 3593864 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 37512 3569004 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49512 7162868 1% /mnt/lustre eg318-client.virtnet: 1 11794 2.36 MB/sec warmup 34 sec latency 9.737 ms oleg318-client.virtnet: 1 12446 2.39 MB/sec warmup 35 sec latency 12.545 ms oleg318-client.virtnet: 1 13155 2.39 MB/sec warmup 36 sec latency 8.168 ms oleg318-client.virtnet: 1 14337 2.60 MB/sec warmup 37 sec latency 8.509 ms oleg318-client.virtnet: 1 14930 2.57 MB/sec warmup 38 sec latency 9.568 ms oleg318-client.virtnet: 1 15649 2.63 MB/sec warmup 39 sec latency 8.581 ms oleg318-client.virtnet: 1 16306 2.58 MB/sec warmup 40 sec latency 8.909 ms oleg318-client.virtnet: 1 17204 2.68 MB/sec warmup 41 sec latency 8.569 ms oleg318-client.virtnet: 1 18090 2.77 MB/sec warmup 42 sec latency 8.773 ms oleg318-client.virtnet: 1 18667 2.73 MB/sec warmup 43 sec latency 11.141 ms oleg318-client.virtnet: 1 19362 2.77 MB/sec warmup 44 sec latency 8.997 ms oleg318-client.virtnet: 1 19991 2.72 MB/sec warmup 45 sec latency 9.779 ms oleg318-client.virtnet: 1 20723 2.81 MB/sec warmup 46 sec latency 249.082 ms oleg318-client.virtnet: 1 20723 2.75 MB/sec warmup 47 sec latency 1249.226 ms oleg318-client.virtnet: 1 20723 2.69 MB/sec warmup 48 sec latency 2249.394 ms oleg318-client.virtnet: 1 20723 2.64 MB/sec warmup 49 sec latency 3249.569 ms oleg318-client.virtnet: 1 21617 2.71 MB/sec warmup 50 sec latency 3288.086 ms oleg318-client.virtnet: 1 22212 2.67 MB/sec warmup 51 sec latency 9.904 ms oleg318-client.virtnet: 1 22908 2.71 MB/sec warmup 52 sec latency 8.871 ms oleg318-client.virtnet: 1 23599 2.68 MB/sec warmup 53 sec latency 9.705 ms oleg318-client.virtnet: 1 24092 2.72 MB/sec warmup 54 sec latency 434.368 ms oleg318-client.virtnet: 1 25113 2.81 MB/sec warmup 55 sec latency 442.315 ms oleg318-client.virtnet: 1 25674 2.77 MB/sec warmup 56 sec latency 9.287 ms oleg318-client.virtnet: 1 25702 2.73 MB/sec warmup 57 sec latency 947.082 ms oleg318-client.virtnet: 1 25702 2.68 MB/sec warmup 58 sec latency 1947.288 ms oleg318-client.virtnet: 1 25702 2.63 MB/sec warmup 59 sec latency 2947.463 ms oleg318-client.virtnet: 1 25702 0.00 MB/sec execute 1 sec latency 4947.693 ms oleg318-client.virtnet: 1 25702 0.00 MB/sec execute 2 sec latency 5947.833 ms oleg318-client.virtnet: 1 25702 0.00 MB/sec execute 3 sec latency 6948.022 ms oleg318-client.virtnet: 1 25702 0.00 MB/sec execute 4 sec latency 7948.194 ms oleg318-client.virtnet: 1 25702 0.00 MB/sec execute 5 sec latency 8948.333 ms oleg318-client.virtnet: 1 25702 0.00 MB/sec execute 6 sec latency 9948.440 ms oleg318-client.virtnet: 1 25702 0.00 MB/sec execute 7 sec latency 10948.590 ms oleg318-client.virtnet: 1 25702 0.00 MB/sec execute 8 sec latency 11948.696 ms oleg318-client.virtnet: 1 25702 0.00 MB/sec execute 9 sec latency 12948.790 ms oleg318-client.virtnet: 1 25702 0.00 MB/sec execute 10 sec latency 13948.919 ms oleg318-client.virtnet: 1 25702 0.00 MB/sec execute 11 sec latency 14949.119 ms oleg318-client.virtnet: 1 25702 0.00 MB/sec execute 12 sec latency 15949.282 ms oleg318-client.virtnet: 1 25702 0.00 MB/sec execute 13 sec latency 16949.453 ms oleg318-client.virtnet: 1 25702 0.00 MB/sec execute 14 sec latency 17949.564 ms oleg318-client.virtnet: 1 26127 0.11 MB/sec execute 15 sec latency 18242.707 ms oleg318-client.virtnet: 1 26790 0.31 MB/sec execute 16 sec latency 10.516 ms oleg318-client.virtnet: 1 27633 0.62 MB/sec execute 17 sec latency 8.912 ms oleg318-client.virtnet: 1 28660 1.02 MB/sec execute 18 sec latency 9.167 ms oleg318-client.virtnet: 1 29214 1.00 MB/sec execute 19 sec latency test_70b fail mds2 4 times Failing mds2 on oleg318-server Stopping /mnt/lustre-mds2 (opts:) on oleg318-server 06:43:28 (1744886608) shut down facet: mds2 facet_host: oleg318-server facet_failover_host: oleg318-server Failover mds2 to oleg318-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg318-server: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-MDT0001 06:43:42 (1744886622) targets are mounted 06:43:42 (1744886622) facet_failover done oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2476 1285212 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12100 3593316 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 37548 3568944 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49648 7162260 1% /mnt/lustre test_70b fail mds1 5 times Failing mds1 on oleg318-server Stopping /mnt/lustre-mds1 (opts:) on oleg318-server 06:43:53 (1744886633) shut down facet: mds1 facet_host: oleg318-server facet_failover_host: oleg318-server Failover mds1 to oleg318-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg318-server: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-MDT0000 06:44:07 (1744886647) targets are mounted 06:44:07 (1744886647) facet_failover done 9.750 ms oleg318-client.virtnet: 1 29917 1.18 MB/sec execute 20 sec latency 8.927 ms oleg318-client.virtnet: 1 30577 1.16 MB/sec execute 21 sec latency 9.553 ms oleg318-client.virtnet: 1 31592 1.44 MB/sec execute 22 sec latency 8.793 ms oleg318-client.virtnet: 1 32362 1.64 MB/sec execute 23 sec latency 8.493 ms oleg318-client.virtnet: 1 32954 1.61 MB/sec execute 24 sec latency 9.246 ms oleg318-client.virtnet: 1 33670 1.72 MB/sec execute 25 sec latency 9.932 ms oleg318-client.virtnet: 1 34387 1.73 MB/sec execute 26 sec latency 8.276 ms oleg318-client.virtnet: 1 35540 2.04 MB/sec execute 27 sec latency 8.402 ms oleg318-client.virtnet: 1 36103 2.03 MB/sec execute 28 sec latency 8.346 ms oleg318-client.virtnet: 1 36738 2.03 MB/sec execute 29 sec latency 11.495 ms oleg318-client.virtnet: 1 37402 2.08 MB/sec execute 30 sec latency 9.715 ms oleg318-client.virtnet: 1 38265 2.19 MB/sec execute 31 sec latency 8.967 ms oleg318-client.virtnet: 1 39276 2.36 MB/sec execute 32 sec latency 12.177 ms oleg318-client.virtnet: 1 39840 2.31 MB/sec execute 33 sec latency 8.673 ms oleg318-client.virtnet: 1 40533 2.38 MB/sec execute 34 sec latency 8.962 ms oleg318-client.virtnet: 1 41200 2.33 MB/sec execute 35 sec latency 9.087 ms oleg318-client.virtnet: 1 41544 2.33 MB/sec execute 36 sec latency 489.146 ms oleg318-client.virtnet: 1 41544 2.26 MB/sec execute 37 sec latency 1489.273 ms oleg318-client.virtnet: 1 41544 2.20 MB/sec execute 38 sec latency 2489.422 ms oleg318-client.virtnet: 1 41544 2.15 MB/sec execute 39 sec latency 3489.596 ms oleg318-client.virtnet: 1 42263 2.23 MB/sec execute 40 sec latency 3923.629 ms oleg318-client.virtnet: 1 42996 2.32 MB/sec execute 41 sec latency 8.975 ms oleg318-client.virtnet: 1 43586 2.29 MB/sec execute 42 sec latency 9.455 ms oleg318-client.virtnet: 1 44281 2.33 MB/sec execute 43 sec latency 9.795 ms oleg318-client.virtnet: 1 44975 2.32 MB/sec execute 44 sec latency 18.211 ms oleg318-client.virtnet: 1 45562 2.39 MB/sec execute 45 sec latency 439.777 ms oleg318-client.virtnet: 1 46452 2.48 MB/sec execute 46 sec latency 8.158 ms oleg318-client.virtnet: 1 46715 2.43 MB/sec execute 47 sec latency 503.474 ms oleg318-client.virtnet: 1 46715 2.38 MB/sec execute 48 sec latency 1503.621 ms oleg318-client.virtnet: 1 46715 2.33 MB/sec execute 49 sec latency 2503.781 ms oleg318-client.virtnet: 1 46715 2.28 MB/sec execute 50 sec latency 3503.936 ms oleg318-client.virtnet: 1 46715 2.24 MB/sec execute 51 sec latency 4504.089 ms oleg318-client.virtnet: 1 46715 2.20 MB/sec execute 52 sec latency 5504.248 ms oleg318-client.virtnet: 1 46715 2.15 MB/sec execute 53 sec latency 6504.430 ms oleg318-client.virtnet: 1 46715 2.11 MB/sec execute 54 sec latency 7504.587 ms oleg318-client.virtnet: 1 46715 2.08 MB/sec execute 55 sec latency 8504.744 ms oleg318-client.virtnet: 1 46715 2.04 MB/sec execute 56 sec latency 9504.944 ms oleg318-client.virtnet: 1 46715 2.00 MB/sec execute 57 sec latency 10505.124 ms oleg318-client.virtnet: 1 46715 1.97 MB/sec execute 58 sec latency 11505.240 ms oleg318-client.virtnet: 1 46715 1.94 MB/sec execute 59 sec latency 12505.427 ms oleg318-client.virtnet: 1 46715 1.90 MB/sec execute 60 sec latency 13505.572 ms oleg318-client.virtnet: 1 46715 1.87 MB/sec execute 61 sec latency 14505.754 ms oleg318-client.virtnet: 1 46715 1.84 MB/sec execute 62 sec latency 15506.046 ms oleg318-client.virtnet: 1 46715 1.81 MB/sec execute 63 sec latency 16506.305 ms oleg318-client.virtnet: 1 46715 1.78 MB/oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2444 1285244 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12168 3591828 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 37548 3568912 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49716 7160740 1% /mnt/lustre test_70b fail mds2 6 times Failing mds2 on oleg318-server Stopping /mnt/lustre-mds2 (opts:) on oleg318-server 06:44:19 (1744886659) shut down facet: mds2 facet_host: oleg318-server facet_failover_host: oleg318-server Failover mds2 to oleg318-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg318-server: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-MDT0001 06:44:33 (1744886673) targets are mounted 06:44:33 (1744886673) facet_failover done oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2468 1285220 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12076 3594056 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 37412 3569096 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49488 7163152 1% /mnt/lustre test_70b fail mds1 7 times Failing mds1 on oleg318-server Stopping /mnt/lustre-mds1 (opts:) on oleg318-server 06:44:45 (1744886685) shut down facet: mds1 facet_host: oleg318-server facet_failover_host: oleg318-server Failover mds1 to oleg318-server mount facets: mds1 sec execute 64 sec latency 17506.611 ms oleg318-client.virtnet: 1 46715 1.76 MB/sec execute 65 sec latency 18506.757 ms oleg318-client.virtnet: 1 47136 1.74 MB/sec execute 66 sec latency 18727.102 ms oleg318-client.virtnet: 1 47796 1.78 MB/sec execute 67 sec latency 9.179 ms oleg318-client.virtnet: 1 48494 1.78 MB/sec execute 68 sec latency 10.461 ms oleg318-client.virtnet: 1 49551 1.87 MB/sec execute 69 sec latency 9.536 ms oleg318-client.virtnet: 1 50153 1.90 MB/sec execute 70 sec latency 9.262 ms oleg318-client.virtnet: 1 50796 1.91 MB/sec execute 71 sec latency 9.967 ms oleg318-client.virtnet: 1 51481 1.93 MB/sec execute 72 sec latency 9.329 ms oleg318-client.virtnet: 1 52170 1.93 MB/sec execute 73 sec latency 9.331 ms oleg318-client.virtnet: 1 53052 2.00 MB/sec execute 74 sec latency 15.892 ms oleg318-client.virtnet: 1 53572 2.03 MB/sec execute 75 sec latency 12.654 ms oleg318-client.virtnet: 1 53958 2.01 MB/sec execute 76 sec latency 23.224 ms oleg318-client.virtnet: 1 54273 2.00 MB/sec execute 77 sec latency 18.109 ms oleg318-client.virtnet: 1 54766 2.02 MB/sec execute 78 sec latency 17.917 ms oleg318-client.virtnet: 1 55308 2.00 MB/sec execute 79 sec latency 10.194 ms oleg318-client.virtnet: 1 55845 2.01 MB/sec execute 80 sec latency 16.938 ms oleg318-client.virtnet: 1 56757 2.10 MB/sec execute 81 sec latency 17.447 ms oleg318-client.virtnet: 1 57159 2.09 MB/sec execute 82 sec latency 11.469 ms oleg318-client.virtnet: 1 57665 2.08 MB/sec execute 83 sec latency 11.499 ms oleg318-client.virtnet: 1 58046 2.07 MB/sec execute 84 sec latency 17.144 ms oleg318-client.virtnet: 1 58760 2.08 MB/sec execute 85 sec latency 9.881 ms oleg318-client.virtnet: 1 59553 2.12 MB/sec execute 86 sec latency 10.101 ms oleg318-client.virtnet: 1 60403 2.17 MB/sec execute 87 sec latency 289.822 ms oleg318-client.virtnet: 1 60403 2.15 MB/sec execute 88 sec latency 1289.908 ms oleg318-client.virtnet: 1 60403 2.13 MB/sec execute 89 sec latency 2290.052 ms oleg318-client.virtnet: 1 60403 2.10 MB/sec execute 90 sec latency 3290.199 ms oleg318-client.virtnet: 1 60574 2.09 MB/sec execute 91 sec latency 3997.222 ms oleg318-client.virtnet: 1 61119 2.08 MB/sec execute 92 sec latency 12.855 ms oleg318-client.virtnet: 1 61806 2.10 MB/sec execute 93 sec latency 10.759 ms oleg318-client.virtnet: 1 62401 2.09 MB/sec execute 94 sec latency 11.182 ms oleg318-client.virtnet: 1 63308 2.14 MB/sec execute 95 sec latency 8.681 ms oleg318-client.virtnet: 1 64200 2.18 MB/sec execute 96 sec latency 9.091 ms oleg318-client.virtnet: 1 64758 2.17 MB/sec execute 97 sec latency 9.808 ms oleg318-client.virtnet: 1 65327 2.19 MB/sec execute 98 sec latency 199.283 ms oleg318-client.virtnet: 1 65327 2.17 MB/sec execute 99 sec latency 1199.502 ms oleg318-client.virtnet: 1 65327 2.15 MB/sec execute 100 sec latency 2199.700 ms oleg318-client.virtnet: 1 65327 2.13 MB/sec execute 101 sec latency 3199.894 ms oleg318-client.virtnet: 1 65327 2.11 MB/sec execute 102 sec latency 4200.056 ms oleg318-client.virtnet: 1 65327 2.09 MB/sec execute 103 sec latency 5200.283 ms oleg318-client.virtnet: 1 65327 2.07 MB/sec execute 104 sec latency 6200.449 ms oleg318-client.virtnet: 1 65327 2.05 MB/sec execute 105 sec latency 7200.598 ms oleg318-client.virtnet: 1 65327 2.03 MB/sec execute 106 sec latency 8200.767 ms oleg318-client.virtnet: 1 65327 2.01 MB/sec execute 107 sec latency 9200.913 ms oleg318-client.virtnet: 1 65327 1.99 MB/sec execute 108 sec latency 10201.074 ms oleg318-client.virtnetStarting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg318-server: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-MDT0000 06:44:58 (1744886698) targets are mounted 06:44:58 (1744886698) facet_failover done oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2444 1285244 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12216 3593664 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 37424 3566832 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49640 7160496 1% /mnt/lustre test_70b fail mds2 8 times Failing mds2 on oleg318-server Stopping /mnt/lustre-mds2 (opts:) on oleg318-server 06:45:10 (1744886710) shut down facet: mds2 facet_host: oleg318-server facet_failover_host: oleg318-server Failover mds2 to oleg318-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg318-server: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-MDT0001 06:45:24 (1744886724) targets are mounted 06:45:24 (1744886724) facet_failover done oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2476 1285212 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12052 3593976 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 37596 3567784 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49648 7161760 1% /mnt/lustre test_70b fail mds1 9 times Failing mds1 on oleg318-server Stopping /mnt/lustre-mds1 (opts:) on oleg318-server 06:45:35 (1744886735) shut down facet: mds1 facet_host: oleg318-server facet_failover_host: oleg318-server : 1 65327 1.97 MB/sec execute 109 sec latency 11201.182 ms oleg318-client.virtnet: 1 65327 1.95 MB/sec execute 110 sec latency 12201.308 ms oleg318-client.virtnet: 1 65327 1.94 MB/sec execute 111 sec latency 13201.454 ms oleg318-client.virtnet: 1 65327 1.92 MB/sec execute 112 sec latency 14201.618 ms oleg318-client.virtnet: 1 65327 1.90 MB/sec execute 113 sec latency 15201.789 ms oleg318-client.virtnet: 1 65327 1.88 MB/sec execute 114 sec latency 16201.950 ms oleg318-client.virtnet: 1 65327 1.87 MB/sec execute 115 sec latency 17202.125 ms oleg318-client.virtnet: 1 65327 1.85 MB/sec execute 116 sec latency 18202.253 ms oleg318-client.virtnet: 1 65761 1.84 MB/sec execute 117 sec latency 18408.689 ms oleg318-client.virtnet: 1 66478 1.85 MB/sec execute 118 sec latency 8.929 ms oleg318-client.virtnet: 1 67621 1.92 MB/sec execute 119 sec latency 10.118 ms oleg318-client.virtnet: 1 68168 1.91 MB/sec execute 120 sec latency 9.687 ms oleg318-client.virtnet: 1 68855 1.93 MB/sec execute 121 sec latency 9.513 ms oleg318-client.virtnet: 1 69465 1.92 MB/sec execute 122 sec latency 9.858 ms oleg318-client.virtnet: 1 70351 1.96 MB/sec execute 123 sec latency 10.082 ms oleg318-client.virtnet: 1 71245 2.00 MB/sec execute 124 sec latency 8.976 ms oleg318-client.virtnet: 1 71836 1.99 MB/sec execute 125 sec latency 12.423 ms oleg318-client.virtnet: 1 72533 2.01 MB/sec execute 126 sec latency 8.991 ms oleg318-client.virtnet: 1 73096 2.00 MB/sec execute 127 sec latency 9.917 ms oleg318-client.virtnet: 1 73961 2.04 MB/sec execute 128 sec latency 10.961 ms oleg318-client.virtnet: 1 74856 2.07 MB/sec execute 129 sec latency 8.912 ms oleg318-client.virtnet: 1 75454 2.06 MB/sec execute 130 sec latency 10.235 ms oleg318-client.virtnet: 1 76125 2.08 MB/sec execute 131 sec latency 9.736 ms oleg318-client.virtnet: 1 76790 2.07 MB/sec execute 132 sec latency 10.204 ms oleg318-client.virtnet: 1 77888 2.12 MB/sec execute 133 sec latency 8.680 ms oleg318-client.virtnet: 1 78547 2.14 MB/sec execute 134 sec latency 8.517 ms oleg318-client.virtnet: 1 79181 2.14 MB/sec execute 135 sec latency 9.431 ms oleg318-client.virtnet: 1 79869 2.14 MB/sec execute 136 sec latency 9.712 ms oleg318-client.virtnet: 1 80566 2.15 MB/sec execute 137 sec latency 8.928 ms oleg318-client.virtnet: 1 80803 2.15 MB/sec execute 138 sec latency 805.687 ms oleg318-client.virtnet: 1 80803 2.14 MB/sec execute 139 sec latency 1805.858 ms oleg318-client.virtnet: 1 80803 2.12 MB/sec execute 140 sec latency 2806.003 ms oleg318-client.virtnet: 1 80803 2.11 MB/sec execute 141 sec latency 3806.159 ms oleg318-client.virtnet: 1 81610 2.14 MB/sec execute 142 sec latency 4176.868 ms oleg318-client.virtnet: 1 82153 2.14 MB/sec execute 143 sec latency 8.381 ms oleg318-client.virtnet: 1 82816 2.14 MB/sec execute 144 sec latency 8.629 ms oleg318-client.virtnet: 1 83498 2.14 MB/sec execute 145 sec latency 9.593 ms oleg318-client.virtnet: 1 84237 2.15 MB/sec execute 146 sec latency 9.201 ms oleg318-client.virtnet: 1 85186 2.20 MB/sec execute 147 sec latency 296.690 ms oleg318-client.virtnet: 1 85649 2.19 MB/sec execute 148 sec latency 14.999 ms oleg318-client.virtnet: 1 85957 2.18 MB/sec execute 149 sec latency 530.843 ms oleg318-client.virtnet: 1 85957 2.17 MB/sec execute 150 sec latency 1530.971 ms oleg318-client.virtnet: 1 85957 2.15 MB/sec execute 151 sec latency 2531.123 ms oleg318-client.virtnet: 1 85957 2.14 MB/sec execute 152 sec latency 3531.313 ms oleg318-client.virtnet: 1 85957 2.12 MB/sec execute 153 sec latency 4531.49Failover mds1 to oleg318-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg318-server: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-MDT0000 06:45:49 (1744886749) targets are mounted 06:45:49 (1744886749) facet_failover done oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2436 1285252 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 11932 3594316 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 37580 3568776 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49512 7163092 1% /mnt/lustre test_70b fail mds2 10 times Failing mds2 on oleg318-server Stopping /mnt/lustre-mds2 (opts:) on oleg318-server 06:46:00 (1744886760) shut down facet: mds2 facet_host: oleg318-server facet_failover_host: oleg318-server Failover mds2 to oleg318-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg318-server: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-MDT0001 06:46:14 (1744886774) targets are mounted 06:46:14 (1744886774) facet_failover done oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2476 1285212 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12060 3593848 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 37696 3566192 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49756 7160040 1% /mnt/lustre 3 ms oleg318-client.virtnet: 1 85957 2.11 MB/sec execute 154 sec latency 5531.632 ms oleg318-client.virtnet: 1 85957 2.10 MB/sec execute 155 sec latency 6531.797 ms oleg318-client.virtnet: 1 85957 2.08 MB/sec execute 156 sec latency 7531.939 ms oleg318-client.virtnet: 1 85957 2.07 MB/sec execute 157 sec latency 8532.118 ms oleg318-client.virtnet: 1 85957 2.06 MB/sec execute 158 sec latency 9532.316 ms oleg318-client.virtnet: 1 85957 2.04 MB/sec execute 159 sec latency 10532.518 ms oleg318-client.virtnet: 1 85957 2.03 MB/sec execute 160 sec latency 11532.630 ms oleg318-client.virtnet: 1 85957 2.02 MB/sec execute 161 sec latency 12532.757 ms oleg318-client.virtnet: 1 85957 2.01 MB/sec execute 162 sec latency 13532.900 ms oleg318-client.virtnet: 1 85957 1.99 MB/sec execute 163 sec latency 14533.046 ms oleg318-client.virtnet: 1 85957 1.98 MB/sec execute 164 sec latency 15533.202 ms oleg318-client.virtnet: 1 85957 1.97 MB/sec execute 165 sec latency 16533.367 ms oleg318-client.virtnet: 1 85957 1.96 MB/sec execute 166 sec latency 17533.533 ms oleg318-client.virtnet: 1 86091 1.95 MB/sec execute 167 sec latency 18295.496 ms oleg318-client.virtnet: 1 86769 1.96 MB/sec execute 168 sec latency 10.659 ms oleg318-client.virtnet: 1 87469 1.96 MB/sec execute 169 sec latency 9.597 ms oleg318-client.virtnet: 1 88551 1.99 MB/sec execute 170 sec latency 8.967 ms oleg318-client.virtnet: 1 89194 2.01 MB/sec execute 171 sec latency 8.241 ms oleg318-client.virtnet: 1 89826 2.01 MB/sec execute 172 sec latency 12.896 ms oleg318-client.virtnet: 1 90428 2.02 MB/sec execute 173 sec latency 11.842 ms oleg318-client.virtnet: 1 91128 2.02 MB/sec execute 174 sec latency 10.788 ms oleg318-client.virtnet: 1 91997 2.05 MB/sec execute 175 sec latency 13.731 ms oleg318-client.virtnet: 1 92678 2.06 MB/sec execute 176 sec latency 8.542 ms oleg318-client.virtnet: 1 93203 2.05 MB/sec execute 177 sec latency 15.556 ms oleg318-client.virtnet: 1 93843 2.07 MB/sec execute 178 sec latency 10.750 ms oleg318-client.virtnet: 1 94482 2.06 MB/sec execute 179 sec latency 11.147 ms oleg318-client.virtnet: 1 95596 2.10 MB/sec execute 180 sec latency 10.549 ms oleg318-client.virtnet: 1 96257 2.11 MB/sec execute 181 sec latency 8.305 ms oleg318-client.virtnet: 1 96805 2.10 MB/sec execute 182 sec latency 10.999 ms oleg318-client.virtnet: 1 97499 2.12 MB/sec execute 183 sec latency 9.063 ms oleg318-client.virtnet: 1 98207 2.11 MB/sec execute 184 sec latency 8.807 ms oleg318-client.virtnet: 1 99344 2.16 MB/sec execute 185 sec latency 8.808 ms oleg318-client.virtnet: 1 99800 2.16 MB/sec execute 186 sec latency 11.692 ms oleg318-client.virtnet: 1 100426 2.15 MB/sec execute 187 sec latency 10.745 ms oleg318-client.virtnet: 1 100593 2.14 MB/sec execute 188 sec latency 731.838 ms oleg318-client.virtnet: 1 100593 2.13 MB/sec execute 189 sec latency 1731.956 ms oleg318-client.virtnet: 1 100593 2.12 MB/sec execute 190 sec latency 2732.128 ms oleg318-client.virtnet: 1 100593 2.11 MB/sec execute 191 sec latency 3732.322 ms oleg318-client.virtnet: 1 100597 2.10 MB/sec execute 192 sec latency 4725.246 ms oleg318-client.virtnet: 1 101244 2.11 MB/sec execute 193 sec latency 10.247 ms oleg318-client.virtnet: 1 101864 2.11 MB/sec execute 194 sec latency 13.911 ms oleg318-client.virtnet: 1 102982 2.15 MB/sec execute 195 sec latency 12.808 ms oleg318-client.virtnet: 1 103502 2.14 MB/sec execute 196 sec latency 10.135 ms oleg318-client.virtnet: 1 103899 2.14 MB/sec execute 197 sec latency 341.392 ms oleg318-client.virtnet: 1 104548 2.15 test_70b fail mds1 11 times Failing mds1 on oleg318-server Stopping /mnt/lustre-mds1 (opts:) on oleg318-server 06:46:26 (1744886786) shut down facet: mds1 facet_host: oleg318-server facet_failover_host: oleg318-server Failover mds1 to oleg318-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg318-server: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-MDT0000 06:46:39 (1744886799) targets are mounted 06:46:39 (1744886799) facet_failover done oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2436 1285252 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 12064 3593580 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 37764 3568736 2% /mnt/lustre[OST:1] filesystem_summary: 7666232 49828 7162316 1% /mnt/lustre test_70b fail mds2 12 times Failing mds2 on oleg318-server Stopping /mnt/lustre-mds2 (opts:) on oleg318-server 06:46:51 (1744886811) shut down facet: mds2 facet_host: oleg318-server facet_failover_host: oleg318-server Failover mds2 to oleg318-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg318-server: oleg318-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-MDT0001 06:47:05 (1744886825) targets are mounted 06:47:05 (1744886825) facet_failover done oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid MB/sec execute 198 sec latency 363.375 ms oleg318-client.virtnet: 1 105279 2.15 MB/sec execute 199 sec latency 9.025 ms oleg318-client.virtnet: 1 105318 2.14 MB/sec execute 200 sec latency 910.667 ms oleg318-client.virtnet: 1 105318 2.13 MB/sec execute 201 sec latency 1910.961 ms oleg318-client.virtnet: 1 105318 2.12 MB/sec execute 202 sec latency 2911.164 ms oleg318-client.virtnet: 1 105318 2.10 MB/sec execute 203 sec latency 3911.328 ms oleg318-client.virtnet: 1 105318 2.09 MB/sec execute 204 sec latency 4911.487 ms oleg318-client.virtnet: 1 105318 2.08 MB/sec execute 205 sec latency 5911.639 ms oleg318-client.virtnet: 1 105318 2.07 MB/sec execute 206 sec latency 6911.808 ms oleg318-client.virtnet: 1 105318 2.06 MB/sec execute 207 sec latency 7912.000 ms oleg318-client.virtnet: 1 105318 2.05 MB/sec execute 208 sec latency 8912.200 ms oleg318-client.virtnet: 1 105318 2.04 MB/sec execute 209 sec latency 9912.357 ms oleg318-client.virtnet: 1 105318 2.03 MB/sec execute 210 sec latency 10912.462 ms oleg318-client.virtnet: 1 105318 2.02 MB/sec execute 211 sec latency 11912.562 ms oleg318-client.virtnet: 1 105318 2.02 MB/sec execute 212 sec latency 12912.758 ms oleg318-client.virtnet: 1 105318 2.01 MB/sec execute 213 sec latency 13912.927 ms oleg318-client.virtnet: 1 105318 2.00 MB/sec execute 214 sec latency 14913.086 ms oleg318-client.virtnet: 1 105318 1.99 MB/sec execute 215 sec latency 15913.294 ms oleg318-client.virtnet: 1 105318 1.98 MB/sec execute 216 sec latency 16913.463 ms oleg318-client.virtnet: 1 105318 1.97 MB/sec execute 217 sec latency 17913.656 ms oleg318-client.virtnet: 1 106139 1.99 MB/sec execute 218 sec latency 18157.476 ms oleg318-client.virtnet: 1 106862 2.01 MB/sec execute 219 sec latency 8.443 ms oleg318-client.virtnet: 1 107446 2.00 MB/sec execute 220 sec latency 10.120 ms oleg318-client.virtnet: 1 108154 2.01 MB/sec execute 221 sec latency 9.268 ms oleg318-client.virtnet: 1 108868 2.01 MB/sec execute 222 sec latency 8.339 ms oleg318-client.virtnet: 1 110010 2.05 MB/sec execute 223 sec latency 9.788 ms oleg318-client.virtnet: 1 110517 2.05 MB/sec execute 224 sec latency 10.009 ms oleg318-client.virtnet: 1 111171 2.05 MB/sec execute 225 sec latency 10.435 ms oleg318-client.virtnet: 1 111840 2.05 MB/sec execute 226 sec latency 9.733 ms oleg318-client.virtnet: 1 112576 2.05 MB/sec execute 227 sec latency 8.394 ms oleg318-client.virtnet: 1 113714 2.09 MB/sec execute 228 sec latency 8.599 ms oleg318-client.virtnet: 1 114314 2.09 MB/sec execute 229 sec latency 10.291 ms oleg318-client.virtnet: 1 115012 2.10 MB/sec execute 230 sec latency 9.407 ms oleg318-client.virtnet: 1 115692 2.09 MB/sec execute 231 sec latency 8.688 ms oleg318-client.virtnet: 1 116717 2.11 MB/sec execute 232 sec latency 8.643 ms oleg318-client.virtnet: 1 117501 2.13 MB/sec execute 233 sec latency 8.162 ms oleg318-client.virtnet: 1 118136 2.13 MB/sec execute 234 sec latency 10.841 ms oleg318-client.virtnet: 1 118825 2.14 MB/sec execute 235 sec latency 9.013 ms oleg318-client.virtnet: 1 119545 2.13 MB/sec execute 236 sec latency 8.522 ms oleg318-client.virtnet: 1 120718 2.17 MB/sec execute 237 sec latency 10.056 ms oleg318-client.virtnet: 1 120909 2.17 MB/sec execute 238 sec latency 658.338 ms oleg318-client.virtnet: 1 120909 2.16 MB/sec execute 239 sec latency 1658.508 ms oleg318-client.virtnet: 1 120909 2.15 MB/sec execute 240 sec latency 2658.676 ms oleg318-client.virtnet: 1 120909 2.14 MB/sec execute 241 sec latency 3658.852 ms oleg318-client.virtnet: 1 120909 2.13 MB/sec execute 242 sec latency 4659.038 ms oleg318mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec -client.virtnet: 1 121315 2.12 MB/sec execute 243 sec latency 4932.508 ms oleg318-client.virtnet: 1 121959 2.13 MB/sec execute 244 sec latency 10.065 ms oleg318-client.virtnet: 1 122625 2.13 MB/sec execute 245 sec latency 10.820 ms oleg318-client.virtnet: 1 123524 2.14 MB/sec execute 246 sec latency 8.410 ms oleg318-client.virtnet: 1 124449 2.16 MB/sec execute 247 sec latency 8.380 ms oleg318-client.virtnet: 1 125013 2.16 MB/sec execute 248 sec latency 12.346 ms oleg318-client.virtnet: 1 125730 2.17 MB/sec execute 249 sec latency 8.958 ms oleg318-client.virtnet: 1 126360 2.16 MB/sec execute 250 sec latency 8.759 ms oleg318-client.virtnet: 1 127205 2.18 MB/sec execute 251 sec latency 16.457 ms oleg318-client.virtnet: 1 128049 2.20 MB/sec execute 252 sec latency 13.501 ms oleg318-client.virtnet: 1 128616 2.19 MB/sec execute 253 sec latency 14.461 ms oleg318-client.virtnet: 1 129293 2.20 MB/sec execute 254 sec latency 8.948 ms oleg318-client.virtnet: 1 129935 2.20 MB/sec execute 255 sec latency 9.631 ms oleg318-client.virtnet: 1 130982 2.22 MB/sec execute 256 sec latency 12.869 ms oleg318-client.virtnet: 1 131696 2.23 MB/sec execute 257 sec latency 10.515 ms oleg318-client.virtnet: 1 132271 2.23 MB/sec execute 258 sec latency 10.639 ms oleg318-client.virtnet: 1 132968 2.23 MB/sec execute 259 sec latency 9.955 ms oleg318-client.virtnet: 1 133674 2.23 MB/sec execute 260 sec latency 9.684 ms oleg318-client.virtnet: 1 134812 2.26 MB/sec execute 261 sec latency 8.819 ms oleg318-client.virtnet: 1 135380 2.26 MB/sec execute 262 sec latency 7.613 ms oleg318-client.virtnet: 1 136019 2.26 MB/sec execute 263 sec latency 11.528 ms oleg318-client.virtnet: 1 136656 2.26 MB/sec execute 264 sec latency 10.747 ms oleg318-client.virtnet: 1 137397 2.27 MB/sec execute 265 sec latency 9.183 ms oleg318-client.virtnet: 1 138500 2.29 MB/sec execute 266 sec latency 9.140 ms oleg318-client.virtnet: 1 139087 2.29 MB/sec execute 267 sec latency 11.111 ms oleg318-client.virtnet: 1 139763 2.30 MB/sec execute 268 sec latency 13.747 ms oleg318-client.virtnet: 1 140385 2.29 MB/sec execute 269 sec latency 9.849 ms oleg318-client.virtnet: 1 141288 2.31 MB/sec execute 270 sec latency 9.008 ms oleg318-client.virtnet: 1 142198 2.33 MB/sec execute 271 sec latency 8.569 ms oleg318-client.virtnet: 1 142785 2.32 MB/sec execute 272 sec latency 11.604 ms oleg318-client.virtnet: 1 143411 2.33 MB/sec execute 273 sec latency 9.475 ms oleg318-client.virtnet: 1 143921 2.32 MB/sec execute 274 sec latency 10.581 ms oleg318-client.virtnet: 1 144565 2.33 MB/sec execute 275 sec latency 10.437 ms oleg318-client.virtnet: 1 145679 2.35 MB/sec execute 276 sec latency 8.501 ms oleg318-client.virtnet: 1 146146 2.35 MB/sec execute 277 sec latency 16.693 ms oleg318-client.virtnet: 1 146599 2.34 MB/sec execute 278 sec latency 12.401 ms oleg318-client.virtnet: 1 147195 2.35 MB/sec execute 279 sec latency 13.435 ms oleg318-client.virtnet: 1 147896 2.35 MB/sec execute 280 sec latency 9.908 ms oleg318-client.virtnet: 1 149033 2.37 MB/sec execute 281 sec latency 10.032 ms oleg318-client.virtnet: 1 149575 2.37 MB/sec execute 282 sec latency 9.494 ms oleg318-client.virtnet: 1 150186 2.37 MB/sec execute 283 sec latency 14.637 ms oleg318-client.virtnet: 1 150795 2.37 MB/sec execute 284 sec latency 14.189 ms oleg318-client.virtnet: 1 151485 2.37 MB/sec execute 285 sec latency 10.182 ms oleg318-client.virtnet: 1 152642 2.40 MB/sec execute 286 sec latency 9.139 ms oleg318-client.virtnet: 1 153217 2.40 MB/sec execute 287 sec latency 8.174 ms oleg318-client.virtnet: 1 153765 2.40 MB/sec execute 288 sec latency 13.447 ms oleg318-client.virtnet: 1 154413 2.40 MB/sec execute 289 sec latency 8.942 ms oleg318-client.virtnet: 1 155125 2.40 MB/sec execute 290 sec latency 10.097 ms oleg318-client.virtnet: 1 156050 2.42 MB/sec execute 291 sec latency 14.183 ms oleg318-client.virtnet: 1 156595 2.42 MB/sec execute 292 sec latency 7.957 ms oleg318-client.virtnet: 1 157261 2.42 MB/sec execute 293 sec latency 10.195 ms oleg318-client.virtnet: 1 157931 2.42 MB/sec execute 294 sec latency 8.941 ms oleg318-client.virtnet: 1 158670 2.42 MB/sec execute 295 sec latency 8.980 ms oleg318-client.virtnet: 1 159776 2.45 MB/sec execute 296 sec latency 8.938 ms oleg318-client.virtnet: 1 160350 2.45 MB/sec execute 297 sec latency 10.209 ms oleg318-client.virtnet: 1 161037 2.45 MB/sec execute 298 sec latency 9.008 ms oleg318-client.virtnet: 1 161657 2.45 MB/sec execute 299 sec latency 8.612 ms oleg318-client.virtnet: 1 cleanup 300 sec oleg318-client.virtnet: 0 cleanup 300 sec oleg318-client.virtnet: oleg318-client.virtnet: Operation Count AvgLat MaxLat oleg318-client.virtnet: ---------------------------------------- oleg318-client.virtnet: NTCreateX 23650 6.951 18408.678 oleg318-client.virtnet: Close 17401 0.853 5.705 oleg318-client.virtnet: Rename 1004 7.052 18.093 oleg318-client.virtnet: Unlink 4750 1.568 6.897 oleg318-client.virtnet: Qpathinfo 21481 3.552 18727.081 oleg318-client.virtnet: Qfileinfo 3756 0.130 1.592 oleg318-client.virtnet: Qfsinfo 3917 0.189 5.264 oleg318-client.virtnet: Sfileinfo 1941 4.071 13.643 oleg318-client.virtnet: Find 8286 0.789 4176.855 oleg318-client.virtnet: WriteX 11705 0.735 6.634 oleg318-client.virtnet: ReadX 37142 0.017 3.600 oleg318-client.virtnet: LockX 78 0.847 1.780 oleg318-client.virtnet: UnlockX 78 0.899 2.011 oleg318-client.virtnet: Flush 1651 4.912 439.764 oleg318-client.virtnet: oleg318-client.virtnet: Throughput 2.44694 MB/sec 1 clients 1 procs max_latency=18727.102 ms oleg318-client.virtnet: stopping dbench on /mnt/lustre/d70b.replay-single/oleg318-client.virtnet at Thu Apr 17 06:48:06 EDT 2025 with return code 0 oleg318-client.virtnet: clean dbench files on /mnt/lustre/d70b.replay-single/oleg318-client.virtnet oleg318-client.virtnet: /mnt/lustre/d70b.replay-single/oleg318-client.virtnet /mnt/lustre/d70b.replay-single/oleg318-client.virtnet oleg318-client.virtnet: removed directory: 'clients/client0/~dmtmp/PWRPNT' oleg318-client.virtnet: removed directory: 'clients/client0/~dmtmp/SEED' oleg318-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORDPRO' oleg318-client.virtnet: removed directory: 'clients/client0/~dmtmp/PM' oleg318-client.virtnet: removed directory: 'clients/client0/~dmtmp/COREL' oleg318-client.virtnet: removed directory: 'clients/client0/~dmtmp/PARADOX' oleg318-client.virtnet: removed directory: 'clients/client0/~dmtmp/EXCEL' oleg318-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORD' oleg318-client.virtnet: removed directory: 'clients/client0/~dmtmp/ACCESS' oleg318-client.virtnet: removed directory: 'clients/client0/~dmtmp' oleg318-client.virtnet: removed directory: 'clients/client0' oleg318-client.virtnet: removed directory: 'clients' oleg318-client.virtnet: removed 'client.txt' oleg318-client.virtnet: /mnt/lustre/d70b.replay-single/oleg318-client.virtnet oleg318-client.virtnet: dbench successfully finished