== replay-single test 70b: dbench 2mdts recovery; 1 clients ========================================================== 18:02:35 (1746136955) Starting client oleg119-client.virtnet: -o user_xattr,flock 192.168.201.119@tcp:/lustre /mnt/lustre Started clients oleg119-client.virtnet: 192.168.201.119@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) striped dir -i0 -c2 -H all_char /mnt/lustre/d70b.replay-single + MISSING_DBENCH_OK= + PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin::/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests:/sbin:/usr/sbin:/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests/: + DBENCH_LIB= + TESTSUITE=replay-single + TESTNAME=test_70b + MOUNT=/mnt/lustre ++ hostname + DIR=/mnt/lustre/d70b.replay-single/oleg119-client.virtnet + LCTL=/home/green/git/lustre-release/lustre/utils/lctl + rundbench 1 -t 300 dbench: no process found dbench: no process found Started rundbench load pid=14810 ... UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2512 1285176 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2116 1285572 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 26152 3580868 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1572 3605448 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 27724 7186316 1% /mnt/lustre test_70b fail mds1 1 times Failing mds1 on oleg119-server Stopping /mnt/lustre-mds1 (opts:) on oleg119-server 18:02:52 (1746136972) shut down facet: mds1 facet_host: oleg119-server facet_failover_host: oleg119-server Failover mds1 to oleg119-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg119-client.virtnet: looking for dbench program oleg119-client.virtnet: /usr/bin/dbench oleg119-client.virtnet: creating output directory /mnt/lustre/d70b.replay-single/oleg119-client.virtnet oleg119-client.virtnet: mkdir: created directory '/mnt/lustre/d70b.replay-single/oleg119-client.virtnet' oleg119-client.virtnet: found dbench client file /usr/share/dbench/client.txt oleg119-client.virtnet: '/usr/share/dbench/client.txt' -> 'client.txt' oleg119-client.virtnet: running 'dbench 1 -t 300' on /mnt/lustre/d70b.replay-single/oleg119-client.virtnet at Thu May 1 18:02:38 EDT 2025 oleg119-client.virtnet: waiting for dbench pid 14841 oleg119-client.virtnet: dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 oleg119-client.virtnet: oleg119-client.virtnet: Running for 300 seconds with load 'client.txt' and minimum warmup 60 secs oleg119-client.virtnet: failed to create barrier semaphore oleg119-client.virtnet: 0 of 1 processes prepared for launch 0 sec oleg119-client.virtnet: 1 of 1 processes prepared for launch 0 sec oleg119-client.virtnet: releasing clients oleg119-client.virtnet: 1 111 5.23 MB/sec warmup 1 sec latency 79.302 ms oleg119-client.virtnet: 1 181 3.66 MB/sec warmup 2 sec latency 108.439 ms oleg119-client.virtnet: 1 305 3.74 MB/sec warmup 3 sec latency 59.449 ms oleg119-client.virtnet: 1 389 3.45 MB/sec warmup 4 sec latency 74.439 ms oleg119-client.virtnet: 1 503 3.49 MB/sec warmup 5 sec latency 56.756 ms oleg119-client.virtnet: 1 664 3.52 MB/sec warmup 6 sec latency 64.148 ms oleg119-client.virtnet: 1 721 3.04 MB/sec warmup 7 sec latency 155.087 ms oleg119-client.virtnet: 1 731 2.66 MB/sec warmup 8 sec latency 987.751 ms oleg119-client.virtnet: 1 794 2.38 MB/sec warmup 9 sec latency 100.654 ms oleg119-client.virtnet: 1 888 2.16 MB/sec warmup 10 sec latency 59.669 ms oleg119-client.virtnet: 1 952 1.98 MB/sec warmup 11 sec latency 83.508 ms oleg119-client.virtnet: 1 1061 1.90 MB/sec warmup 12 sec latency 58.311 ms oleg119-client.virtnet: 1 1134 1.77 MB/sec warmup 13 sec latency 367.926 ms oleg119-client.virtnet: 1 1134 1.64 MB/sec warmup 14 sec latency 1368.679 ms oleg119-client.virtnet: 1 1134 1.53 MB/sec warmup 15 sec latency 2369.091 ms oleg119-client.virtnet: 1 1134 1.43 MB/sec warmup 16 sec latency 3369.968 ms oleg119-client.virtnet: 1 1134 1.35 MB/sec warmup 17 sec latency 4370.178 ms oleg119-client.virtnet: 1 1134 1.28 MB/sec warmup 18 sec latency 5370.384 ms oleg119-client.virtnet: 1 1134 1.21 MB/sec warmup 19 sec latency 6370.628 ms oleg119-client.virtnet: 1 1134 1.15 MB/sec warmup 20 sec latency 7372.519 ms oleg119-client.virtnet: 1 1134 1.09 MB/sec warmup 21 sec latency 8372.915 ms oleg119-client.virtnet: 1 1134 1.04 MB/sec warmup 22 sec latency 9373.396 ms oleg119-client.virtnet: 1 1134 1.00 MB/sec warmup 23 sec latency 10373.622 ms oleg119-client.virtnet: 1 1134 0.96 MB/sec warmup 24 sec latency 11374.063 ms oleg119-client.virtnet: 1 1134 0.92 MB/sec warmup 25 sec latency 12374.746 ms oleg119-client.virtnet: 1 1134 0.88 MB/sec warmup 26 sec latency 13375.066 ms oleg119-client.virtnet: 1 1134 0.85 MB/sec warmup 27 sec latency 14375.391 ms oleg119-client.virtnet: 1 1134 0.82 MB/sec warmup 28 sec latency 15377.506 ms oleg119-client.virtnet: 1 1134 0.79 MB/sec warmup 29 sec latency 16377.906 ms oleg119-client.virtnet: 1 1134 0.77 MB/sec warmup 30 sec latency 17380.295 ms oleg119-client.virtnet: 1 1134 0.74 MB/sec warmup 31 sec latency 18380.525 ms oleg119-client.virtnet: 1 1134 0.72 MB/sec warmup 32 sec latency 19381.371 ms oleg119-client.virtnet: 1 1134 0.70 MB/sec warmup 33 sec latenoleg119-server: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg119-client: oleg119-server: ssh exited with exit code 1 Started lustre-MDT0000 18:03:14 (1746136994) targets are mounted 18:03:14 (1746136994) facet_failover done oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2480 1285208 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2076 1285612 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37424 3569384 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12092 3594872 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49516 7164256 1% /mnt/lustre test_70b fail mds2 2 times Failing mds2 on oleg119-server Stopping /mnt/lustre-mds2 (opts:) on oleg119-server 18:03:34 (1746137014) shut down facet: mds2 facet_host: oleg119-server facet_failover_host: oleg119-server Failover mds2 to oleg119-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg119-server: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg119-client: oleg119-server: ssh exited with exit code 1 Started lustre-MDT0001 18:03:56 (1746137036) targets are mounted 18:03:56 (1746137036) facet_failover done cy 20381.740 ms oleg119-client.virtnet: 1 1134 0.68 MB/sec warmup 34 sec latency 21383.086 ms oleg119-client.virtnet: 1 1134 0.66 MB/sec warmup 35 sec latency 22383.261 ms oleg119-client.virtnet: 1 1134 0.64 MB/sec warmup 36 sec latency 23383.516 ms oleg119-client.virtnet: 1 1134 0.62 MB/sec warmup 37 sec latency 24384.383 ms oleg119-client.virtnet: 1 1190 0.61 MB/sec warmup 38 sec latency 24686.368 ms oleg119-client.virtnet: 1 1270 0.59 MB/sec warmup 39 sec latency 58.206 ms oleg119-client.virtnet: 1 1434 0.65 MB/sec warmup 40 sec latency 70.271 ms oleg119-client.virtnet: 1 1504 0.63 MB/sec warmup 41 sec latency 68.369 ms oleg119-client.virtnet: 1 1618 0.62 MB/sec warmup 42 sec latency 43.998 ms oleg119-client.virtnet: 1 1698 0.61 MB/sec warmup 43 sec latency 60.818 ms oleg119-client.virtnet: 1 1868 0.60 MB/sec warmup 44 sec latency 36.148 ms oleg119-client.virtnet: 1 1995 0.59 MB/sec warmup 45 sec latency 56.476 ms oleg119-client.virtnet: 1 2098 0.58 MB/sec warmup 46 sec latency 69.930 ms oleg119-client.virtnet: 1 2193 0.57 MB/sec warmup 47 sec latency 99.374 ms oleg119-client.virtnet: 1 2346 0.58 MB/sec warmup 48 sec latency 59.489 ms oleg119-client.virtnet: 1 2426 0.57 MB/sec warmup 49 sec latency 79.479 ms oleg119-client.virtnet: 1 2553 0.58 MB/sec warmup 50 sec latency 60.981 ms oleg119-client.virtnet: 1 2776 0.63 MB/sec warmup 51 sec latency 39.854 ms oleg119-client.virtnet: 1 2931 0.64 MB/sec warmup 52 sec latency 97.626 ms oleg119-client.virtnet: 1 3024 0.63 MB/sec warmup 53 sec latency 77.085 ms oleg119-client.virtnet: 1 3277 0.65 MB/sec warmup 54 sec latency 70.192 ms oleg119-client.virtnet: 1 3494 0.70 MB/sec warmup 55 sec latency 59.327 ms oleg119-client.virtnet: 1 3634 0.69 MB/sec warmup 56 sec latency 60.234 ms oleg119-client.virtnet: 1 3785 0.70 MB/sec warmup 57 sec latency 33.138 ms oleg119-client.virtnet: 1 3883 0.68 MB/sec warmup 58 sec latency 48.948 ms oleg119-client.virtnet: 1 3963 0.68 MB/sec warmup 59 sec latency 55.829 ms oleg119-client.virtnet: 1 4083 0.01 MB/sec execute 1 sec latency 724.416 ms oleg119-client.virtnet: 1 4083 0.01 MB/sec execute 2 sec latency 1725.040 ms oleg119-client.virtnet: 1 4083 0.00 MB/sec execute 3 sec latency 2725.462 ms oleg119-client.virtnet: 1 4083 0.00 MB/sec execute 4 sec latency 3727.818 ms oleg119-client.virtnet: 1 4083 0.00 MB/sec execute 5 sec latency 4728.058 ms oleg119-client.virtnet: 1 4083 0.00 MB/sec execute 6 sec latency 5728.853 ms oleg119-client.virtnet: 1 4083 0.00 MB/sec execute 7 sec latency 6732.665 ms oleg119-client.virtnet: 1 4083 0.00 MB/sec execute 8 sec latency 7732.928 ms oleg119-client.virtnet: 1 4083 0.00 MB/sec execute 9 sec latency 8733.201 ms oleg119-client.virtnet: 1 4083 0.00 MB/sec execute 10 sec latency 9733.401 ms oleg119-client.virtnet: 1 4083 0.00 MB/sec execute 11 sec latency 10736.571 ms oleg119-client.virtnet: 1 4083 0.00 MB/sec execute 12 sec latency 11736.782 ms oleg119-client.virtnet: 1 4083 0.00 MB/sec execute 13 sec latency 12737.037 ms oleg119-client.virtnet: 1 4083 0.00 MB/sec execute 14 sec latency 13737.668 ms oleg119-client.virtnet: 1 4083 0.00 MB/sec execute 15 sec latency 14741.645 ms oleg119-client.virtnet: 1 4083 0.00 MB/sec execute 16 sec latency 15741.850 ms oleg119-client.virtnet: 1 4083 0.00 MB/sec execute 17 sec latency 16742.087 ms oleg119-client.virtnet: 1 4083 0.00 MB/sec execute 18 sec latency 17742.297 ms oleg119-client.virtnet: 1 4188 0oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2448 1285240 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37656 3569180 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 11908 3594872 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49564 7164052 1% /mnt/lustre test_70b fail mds1 3 times Failing mds1 on oleg119-server Stopping /mnt/lustre-mds1 (opts:) on oleg119-server 18:04:15 (1746137055) shut down facet: mds1 facet_host: oleg119-server facet_failover_host: oleg119-server Failover mds1 to oleg119-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg119-server: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg119-client: oleg119-server: ssh exited with exit code 1 Started lustre-MDT0000 18:04:35 (1746137075) targets are mounted 18:04:35 (1746137075) facet_failover done oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec .00 MB/sec execute 19 sec latency 17804.460 ms oleg119-client.virtnet: 1 4282 0.01 MB/sec execute 20 sec latency 62.646 ms oleg119-client.virtnet: 1 4362 0.02 MB/sec execute 21 sec latency 64.937 ms oleg119-client.virtnet: 1 4448 0.03 MB/sec execute 22 sec latency 99.586 ms oleg119-client.virtnet: 1 4509 0.04 MB/sec execute 23 sec latency 74.487 ms oleg119-client.virtnet: 1 4564 0.04 MB/sec execute 24 sec latency 122.605 ms oleg119-client.virtnet: 1 4701 0.08 MB/sec execute 25 sec latency 49.051 ms oleg119-client.virtnet: 1 4799 0.08 MB/sec execute 26 sec latency 62.341 ms oleg119-client.virtnet: 1 4959 0.19 MB/sec execute 27 sec latency 85.094 ms oleg119-client.virtnet: 1 5030 0.18 MB/sec execute 28 sec latency 62.300 ms oleg119-client.virtnet: 1 5122 0.18 MB/sec execute 29 sec latency 59.500 ms oleg119-client.virtnet: 1 5197 0.17 MB/sec execute 30 sec latency 75.751 ms oleg119-client.virtnet: 1 5280 0.17 MB/sec execute 31 sec latency 57.141 ms oleg119-client.virtnet: 1 5354 0.17 MB/sec execute 32 sec latency 67.131 ms oleg119-client.virtnet: 1 5467 0.17 MB/sec execute 33 sec latency 57.658 ms oleg119-client.virtnet: 1 5587 0.16 MB/sec execute 34 sec latency 63.870 ms oleg119-client.virtnet: 1 5716 0.16 MB/sec execute 35 sec latency 55.804 ms oleg119-client.virtnet: 1 5761 0.16 MB/sec execute 36 sec latency 436.108 ms oleg119-client.virtnet: 1 5761 0.16 MB/sec execute 37 sec latency 1436.373 ms oleg119-client.virtnet: 1 5761 0.15 MB/sec execute 38 sec latency 2438.222 ms oleg119-client.virtnet: 1 5761 0.15 MB/sec execute 39 sec latency 3439.516 ms oleg119-client.virtnet: 1 5761 0.14 MB/sec execute 40 sec latency 4441.704 ms oleg119-client.virtnet: 1 5761 0.14 MB/sec execute 41 sec latency 5441.929 ms oleg119-client.virtnet: 1 5761 0.14 MB/sec execute 42 sec latency 6442.862 ms oleg119-client.virtnet: 1 5761 0.13 MB/sec execute 43 sec latency 7444.003 ms oleg119-client.virtnet: 1 5761 0.13 MB/sec execute 44 sec latency 8444.252 ms oleg119-client.virtnet: 1 5761 0.13 MB/sec execute 45 sec latency 9446.034 ms oleg119-client.virtnet: 1 5761 0.13 MB/sec execute 46 sec latency 10447.748 ms oleg119-client.virtnet: 1 5761 0.12 MB/sec execute 47 sec latency 11449.274 ms oleg119-client.virtnet: 1 5761 0.12 MB/sec execute 48 sec latency 12449.470 ms oleg119-client.virtnet: 1 5761 0.12 MB/sec execute 49 sec latency 13449.983 ms oleg119-client.virtnet: 1 5761 0.12 MB/sec execute 50 sec latency 14450.505 ms oleg119-client.virtnet: 1 5761 0.11 MB/sec execute 51 sec latency 15450.704 ms oleg119-client.virtnet: 1 5761 0.11 MB/sec execute 52 sec latency 16450.924 ms oleg119-client.virtnet: 1 5761 0.11 MB/sec execute 53 sec latency 17451.362 ms oleg119-client.virtnet: 1 5761 0.11 MB/sec execute 54 sec latency 18451.924 ms oleg119-client.virtnet: 1 5761 0.11 MB/sec execute 55 sec latency 19452.453 ms oleg119-client.virtnet: 1 5761 0.10 MB/sec execute 56 sec latency 20453.069 ms oleg119-client.virtnet: 1 5761 0.10 MB/sec execute 57 sec latency 21458.791 ms oleg119-client.virtnet: 1 5761 0.10 MB/sec execute 58 sec latency 22459.242 ms oleg119-client.virtnet: 1 5761 0.10 MB/sec execute 59 sec latency 23460.309 ms oleg119-client.virtnet: 1 5914 0.12 MB/sec execute 60 sec latency 23530.033 ms oleg119-client.virtnet: 1 6001 0.12 MB/sec execute 61 sec latency 86.103 ms oleg119-client.virtnet: 1 6135 0.13 MB/sec execute 62 sec latency 47.724 ms oleg119-client.virtnet: 1 6437 0.18 MB/sec execute 63 UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2440 1285248 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37768 3568836 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 11924 3592236 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49692 7161072 1% /mnt/lustre test_70b fail mds2 4 times Failing mds2 on oleg119-server Stopping /mnt/lustre-mds2 (opts:) on oleg119-server 18:04:54 (1746137094) shut down facet: mds2 facet_host: oleg119-server facet_failover_host: oleg119-server Failover mds2 to oleg119-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg119-server: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg119-client: oleg119-server: ssh exited with exit code 1 Started lustre-MDT0001 18:05:16 (1746137116) targets are mounted 18:05:16 (1746137116) facet_failover done oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2448 1285240 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37456 3569124 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12068 3593932 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49524 7163056 1% /mnt/lustre sec latency 54.228 ms oleg119-client.virtnet: 1 6724 0.21 MB/sec execute 64 sec latency 51.912 ms oleg119-client.virtnet: 1 7049 0.27 MB/sec execute 65 sec latency 42.034 ms oleg119-client.virtnet: 1 7187 0.27 MB/sec execute 66 sec latency 45.213 ms oleg119-client.virtnet: 1 7319 0.28 MB/sec execute 67 sec latency 47.404 ms oleg119-client.virtnet: 1 7395 0.28 MB/sec execute 68 sec latency 71.855 ms oleg119-client.virtnet: 1 7488 0.28 MB/sec execute 69 sec latency 63.978 ms oleg119-client.virtnet: 1 7557 0.27 MB/sec execute 70 sec latency 53.562 ms oleg119-client.virtnet: 1 7628 0.27 MB/sec execute 71 sec latency 63.939 ms oleg119-client.virtnet: 1 7769 0.27 MB/sec execute 72 sec latency 48.759 ms oleg119-client.virtnet: 1 7823 0.26 MB/sec execute 73 sec latency 118.431 ms oleg119-client.virtnet: 1 7902 0.26 MB/sec execute 74 sec latency 67.117 ms oleg119-client.virtnet: 1 7997 0.26 MB/sec execute 75 sec latency 61.000 ms oleg119-client.virtnet: 1 8071 0.26 MB/sec execute 76 sec latency 75.381 ms oleg119-client.virtnet: 1 8193 0.27 MB/sec execute 77 sec latency 55.391 ms oleg119-client.virtnet: 1 8310 0.27 MB/sec execute 78 sec latency 61.936 ms oleg119-client.virtnet: 1 8512 0.31 MB/sec execute 79 sec latency 46.815 ms oleg119-client.virtnet: 1 8609 0.30 MB/sec execute 80 sec latency 63.732 ms oleg119-client.virtnet: 1 8711 0.30 MB/sec execute 81 sec latency 66.315 ms oleg119-client.virtnet: 1 8817 0.30 MB/sec execute 82 sec latency 62.143 ms oleg119-client.virtnet: 1 8903 0.29 MB/sec execute 83 sec latency 86.847 ms oleg119-client.virtnet: 1 8988 0.29 MB/sec execute 84 sec latency 97.305 ms oleg119-client.virtnet: 1 9105 0.29 MB/sec execute 85 sec latency 57.105 ms oleg119-client.virtnet: 1 9204 0.29 MB/sec execute 86 sec latency 64.094 ms oleg119-client.virtnet: 1 9296 0.29 MB/sec execute 87 sec latency 56.199 ms oleg119-client.virtnet: 1 9436 0.30 MB/sec execute 88 sec latency 84.834 ms oleg119-client.virtnet: 1 9521 0.29 MB/sec execute 89 sec latency 72.772 ms oleg119-client.virtnet: 1 9600 0.30 MB/sec execute 90 sec latency 69.502 ms oleg119-client.virtnet: 1 9855 0.33 MB/sec execute 91 sec latency 60.418 ms oleg119-client.virtnet: 1 10030 0.34 MB/sec execute 92 sec latency 66.525 ms oleg119-client.virtnet: 1 10276 0.35 MB/sec execute 93 sec latency 85.324 ms oleg119-client.virtnet: 1 10471 0.36 MB/sec execute 94 sec latency 44.304 ms oleg119-client.virtnet: 1 10486 0.35 MB/sec execute 95 sec latency 843.699 ms oleg119-client.virtnet: 1 10486 0.35 MB/sec execute 96 sec latency 1844.089 ms oleg119-client.virtnet: 1 10486 0.35 MB/sec execute 97 sec latency 2844.314 ms oleg119-client.virtnet: 1 10486 0.34 MB/sec execute 98 sec latency 3846.020 ms oleg119-client.virtnet: 1 10596 0.37 MB/sec execute 99 sec latency 4550.162 ms oleg119-client.virtnet: 1 10714 0.37 MB/sec execute 100 sec latency 58.125 ms oleg119-client.virtnet: 1 10798 0.37 MB/sec execute 101 sec latency 51.155 ms oleg119-client.virtnet: 1 10898 0.37 MB/sec execute 102 sec latency 102.495 ms oleg119-client.virtnet: 1 10954 0.37 MB/sec execute 103 sec latency 54.573 ms oleg119-client.virtnet: 1 11013 0.37 MB/sec execute 104 sec latency 88.604 ms oleg119-client.virtnet: 1 11085 0.36 MB/sec execute 105 sec latency 90.821 ms oleg119-client.virtnet: 1 11138 0.36 MB/sec execute 106 sec latency 68.468 ms oleg119-client.virtnet: 1 11273 0.36 MB/sec execute 107 sec latency 46.650 ms oleg119-client.virtnet: 1 11364 0.36 MB/sec extest_70b fail mds1 5 times Failing mds1 on oleg119-server Stopping /mnt/lustre-mds1 (opts:) on oleg119-server 18:05:38 (1746137138) shut down facet: mds1 facet_host: oleg119-server facet_failover_host: oleg119-server Failover mds1 to oleg119-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg119-server: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg119-client: oleg119-server: ssh exited with exit code 1 Started lustre-MDT0000 18:06:07 (1746137167) targets are mounted 18:06:07 (1746137167) facet_failover done ecute 108 sec latency 72.966 ms oleg119-client.virtnet: 1 11427 0.35 MB/sec execute 109 sec latency 76.209 ms oleg119-client.virtnet: 1 11476 0.35 MB/sec execute 110 sec latency 124.027 ms oleg119-client.virtnet: 1 11559 0.35 MB/sec execute 111 sec latency 80.354 ms oleg119-client.virtnet: 1 11572 0.35 MB/sec execute 112 sec latency 663.969 ms oleg119-client.virtnet: 1 11577 0.35 MB/sec execute 113 sec latency 1534.287 ms oleg119-client.virtnet: 1 11649 0.34 MB/sec execute 114 sec latency 69.420 ms oleg119-client.virtnet: 1 11794 0.35 MB/sec execute 115 sec latency 55.902 ms oleg119-client.virtnet: 1 11880 0.35 MB/sec execute 116 sec latency 57.025 ms oleg119-client.virtnet: 1 12034 0.37 MB/sec execute 117 sec latency 69.557 ms oleg119-client.virtnet: 1 12102 0.37 MB/sec execute 118 sec latency 228.502 ms oleg119-client.virtnet: 1 12102 0.36 MB/sec execute 119 sec latency 1229.834 ms oleg119-client.virtnet: 1 12102 0.36 MB/sec execute 120 sec latency 2230.482 ms oleg119-client.virtnet: 1 12102 0.36 MB/sec execute 121 sec latency 3231.034 ms oleg119-client.virtnet: 1 12102 0.35 MB/sec execute 122 sec latency 4231.327 ms oleg119-client.virtnet: 1 12102 0.35 MB/sec execute 123 sec latency 5235.333 ms oleg119-client.virtnet: 1 12102 0.35 MB/sec execute 124 sec latency 6244.645 ms oleg119-client.virtnet: 1 12102 0.35 MB/sec execute 125 sec latency 7245.552 ms oleg119-client.virtnet: 1 12102 0.34 MB/sec execute 126 sec latency 8246.134 ms oleg119-client.virtnet: 1 12102 0.34 MB/sec execute 127 sec latency 9246.360 ms oleg119-client.virtnet: 1 12102 0.34 MB/sec execute 128 sec latency 10247.776 ms oleg119-client.virtnet: 1 12102 0.34 MB/sec execute 129 sec latency 11248.003 ms oleg119-client.virtnet: 1 12102 0.33 MB/sec execute 130 sec latency 12252.309 ms oleg119-client.virtnet: 1 12102 0.33 MB/sec execute 131 sec latency 13253.570 ms oleg119-client.virtnet: 1 12102 0.33 MB/sec execute 132 sec latency 14253.763 ms oleg119-client.virtnet: 1 12102 0.33 MB/sec execute 133 sec latency 15253.998 ms oleg119-client.virtnet: 1 12102 0.32 MB/sec execute 134 sec latency 16263.023 ms oleg119-client.virtnet: 1 12102 0.32 MB/sec execute 135 sec latency 17263.373 ms oleg119-client.virtnet: 1 12102 0.32 MB/sec execute 136 sec latency 18263.915 ms oleg119-client.virtnet: 1 12102 0.32 MB/sec execute 137 sec latency 19264.176 ms oleg119-client.virtnet: 1 12102 0.31 MB/sec execute 138 sec latency 20265.852 ms oleg119-client.virtnet: 1 12102 0.31 MB/sec execute 139 sec latency 21266.236 ms oleg119-client.virtnet: 1 12102 0.31 MB/sec execute 140 sec latency 22266.481 ms oleg119-client.virtnet: 1 12102 0.31 MB/sec execute 141 sec latency 23266.970 ms oleg119-client.virtnet: 1 12102 0.30 MB/sec execute 142 sec latency 24267.368 ms oleg119-client.virtnet: 1 12102 0.30 MB/sec execute 143 sec latency 25269.252 ms oleg119-client.virtnet: 1 12102 0.30 MB/sec execute 144 sec latency 26269.482 ms oleg119-client.virtnet: 1 12102 0.30 MB/sec execute 145 sec latency 27270.113 ms oleg119-client.virtnet: 1 12102 0.30 MB/sec execute 146 sec latency 28277.144 ms oleg119-client.virtnet: 1 12102 0.29 MB/sec execute 147 sec latency 29277.410 ms oleg119-client.virtnet: 1 12102 0.29 MB/sec execute 148 sec latency 30278.058 ms oleg119-client.virtnet: 1 12102 0.29 MB/sec execute 149 sec latency 31278.428 ms oleg119-client.virtnet: 1 12102 0.29 MB/sec execute 150 sec latency 32278.870 ms oleg119-client.virtnet: 1 12113 0.29 MB/sec execute 151 sec latency 33043.935 ms oleg119-client.virtnet: 1 12208 0.29 MB/sec oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2440 1285248 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37424 3569508 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12108 3594828 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49532 7164336 1% /mnt/lustre test_70b fail mds2 6 times Failing mds2 on oleg119-server Stopping /mnt/lustre-mds2 (opts:) on oleg119-server 18:06:28 (1746137188) shut down facet: mds2 facet_host: oleg119-server facet_failover_host: oleg119-server Failover mds2 to oleg119-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg119-server: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg119-client: oleg119-server: ssh exited with exit code 1 Started lustre-MDT0001 18:06:51 (1746137211) targets are mounted 18:06:51 (1746137211) facet_failover done oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid execute 152 sec latency 57.030 ms oleg119-client.virtnet: 1 12300 0.28 MB/sec execute 153 sec latency 65.315 ms oleg119-client.virtnet: 1 12372 0.28 MB/sec execute 154 sec latency 69.478 ms oleg119-client.virtnet: 1 12472 0.28 MB/sec execute 155 sec latency 71.279 ms oleg119-client.virtnet: 1 12584 0.28 MB/sec execute 156 sec latency 43.924 ms oleg119-client.virtnet: 1 12687 0.28 MB/sec execute 157 sec latency 61.094 ms oleg119-client.virtnet: 1 12809 0.28 MB/sec execute 158 sec latency 59.536 ms oleg119-client.virtnet: 1 12918 0.28 MB/sec execute 159 sec latency 81.376 ms oleg119-client.virtnet: 1 13023 0.28 MB/sec execute 160 sec latency 111.861 ms oleg119-client.virtnet: 1 13095 0.28 MB/sec execute 161 sec latency 66.043 ms oleg119-client.virtnet: 1 13226 0.29 MB/sec execute 162 sec latency 55.862 ms oleg119-client.virtnet: 1 13400 0.30 MB/sec execute 163 sec latency 85.861 ms oleg119-client.virtnet: 1 13572 0.31 MB/sec execute 164 sec latency 57.740 ms oleg119-client.virtnet: 1 13675 0.31 MB/sec execute 165 sec latency 74.677 ms oleg119-client.virtnet: 1 13978 0.32 MB/sec execute 166 sec latency 102.218 ms oleg119-client.virtnet: 1 14081 0.32 MB/sec execute 167 sec latency 89.722 ms oleg119-client.virtnet: 1 14232 0.33 MB/sec execute 168 sec latency 93.636 ms oleg119-client.virtnet: 1 14354 0.33 MB/sec execute 169 sec latency 44.737 ms oleg119-client.virtnet: 1 14444 0.34 MB/sec execute 170 sec latency 84.100 ms oleg119-client.virtnet: 1 14544 0.33 MB/sec execute 171 sec latency 48.878 ms oleg119-client.virtnet: 1 14631 0.33 MB/sec execute 172 sec latency 62.634 ms oleg119-client.virtnet: 1 14708 0.33 MB/sec execute 173 sec latency 52.078 ms oleg119-client.virtnet: 1 14858 0.33 MB/sec execute 174 sec latency 49.187 ms oleg119-client.virtnet: 1 14938 0.33 MB/sec execute 175 sec latency 65.947 ms oleg119-client.virtnet: 1 15022 0.33 MB/sec execute 176 sec latency 59.285 ms oleg119-client.virtnet: 1 15135 0.33 MB/sec execute 177 sec latency 52.892 ms oleg119-client.virtnet: 1 15271 0.33 MB/sec execute 178 sec latency 43.503 ms oleg119-client.virtnet: 1 15394 0.33 MB/sec execute 179 sec latency 58.121 ms oleg119-client.virtnet: 1 15498 0.33 MB/sec execute 180 sec latency 55.750 ms oleg119-client.virtnet: 1 15649 0.34 MB/sec execute 181 sec latency 67.967 ms oleg119-client.virtnet: 1 15734 0.34 MB/sec execute 182 sec latency 57.786 ms oleg119-client.virtnet: 1 15792 0.34 MB/sec execute 183 sec latency 90.294 ms oleg119-client.virtnet: 1 15859 0.34 MB/sec execute 184 sec latency 74.551 ms oleg119-client.virtnet: 1 15920 0.34 MB/sec execute 185 sec latency 74.001 ms oleg119-client.virtnet: 1 15986 0.34 MB/sec execute 186 sec latency 85.923 ms oleg119-client.virtnet: 1 16038 0.34 MB/sec execute 187 sec latency 93.402 ms oleg119-client.virtnet: 1 16145 0.33 MB/sec execute 188 sec latency 58.756 ms oleg119-client.virtnet: 1 16222 0.33 MB/sec execute 189 sec latency 58.118 ms oleg119-client.virtnet: 1 16337 0.33 MB/sec execute 190 sec latency 113.376 ms oleg119-client.virtnet: 1 16354 0.33 MB/sec execute 191 sec latency 824.256 ms oleg119-client.virtnet: 1 16354 0.33 MB/sec execute 192 sec latency 1828.638 ms oleg119-client.virtnet: 1 16354 0.33 MB/sec execute 193 sec latency 2828.827 ms oleg119-client.virtnet: 1 16362 0.32 MB/sec execute 194 sec latency 3661.797 ms oleg119-client.virtnet: 1 16470 0.33 MB/sec execute 195 sec latency 67.917 ms oleg119-client.virtnet: 1 16585 0.33 MB/sec execute 196 sec latency 56.307 ms oleg119-client.virtnet: 1 16641 0.3mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2448 1285240 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37536 3568756 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12068 3591928 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49604 7160684 1% /mnt/lustre test_70b fail mds1 7 times Failing mds1 on oleg119-server Stopping /mnt/lustre-mds1 (opts:) on oleg119-server 18:07:14 (1746137234) shut down facet: mds1 facet_host: oleg119-server facet_failover_host: oleg119-server Failover mds1 to oleg119-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg119-server: oleg119-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg119-client: oleg119-server: ssh exited with exit code 1 Started lustre-MDT0000 18:07:37 (1746137257) targets are mounted 18:07:37 (1746137257) facet_failover done 3 MB/sec execute 197 sec latency 77.962 ms oleg119-client.virtnet: 1 16761 0.33 MB/sec execute 198 sec latency 60.284 ms oleg119-client.virtnet: 1 16850 0.33 MB/sec execute 199 sec latency 83.848 ms oleg119-client.virtnet: 1 17121 0.35 MB/sec execute 200 sec latency 45.863 ms oleg119-client.virtnet: 1 17216 0.35 MB/sec execute 201 sec latency 72.991 ms oleg119-client.virtnet: 1 17545 0.35 MB/sec execute 202 sec latency 54.716 ms oleg119-client.virtnet: 1 17722 0.37 MB/sec execute 203 sec latency 61.976 ms oleg119-client.virtnet: 1 17809 0.37 MB/sec execute 204 sec latency 74.221 ms oleg119-client.virtnet: 1 17880 0.37 MB/sec execute 205 sec latency 106.084 ms oleg119-client.virtnet: 1 17985 0.37 MB/sec execute 206 sec latency 84.672 ms oleg119-client.virtnet: 1 18037 0.37 MB/sec execute 207 sec latency 62.157 ms oleg119-client.virtnet: 1 18087 0.37 MB/sec execute 208 sec latency 102.282 ms oleg119-client.virtnet: 1 18137 0.37 MB/sec execute 209 sec latency 111.847 ms oleg119-client.virtnet: 1 18223 0.36 MB/sec execute 210 sec latency 50.432 ms oleg119-client.virtnet: 1 18274 0.36 MB/sec execute 211 sec latency 71.143 ms oleg119-client.virtnet: 1 18363 0.36 MB/sec execute 212 sec latency 77.502 ms oleg119-client.virtnet: 1 18444 0.36 MB/sec execute 213 sec latency 92.469 ms oleg119-client.virtnet: 1 18490 0.36 MB/sec execute 214 sec latency 282.682 ms oleg119-client.virtnet: 1 18490 0.36 MB/sec execute 215 sec latency 1283.657 ms oleg119-client.virtnet: 1 18490 0.36 MB/sec execute 216 sec latency 2284.391 ms oleg119-client.virtnet: 1 18490 0.35 MB/sec execute 217 sec latency 3285.691 ms oleg119-client.virtnet: 1 18490 0.35 MB/sec execute 218 sec latency 4285.937 ms oleg119-client.virtnet: 1 18490 0.35 MB/sec execute 219 sec latency 5287.020 ms oleg119-client.virtnet: 1 18490 0.35 MB/sec execute 220 sec latency 6287.359 ms oleg119-client.virtnet: 1 18490 0.35 MB/sec execute 221 sec latency 7289.416 ms oleg119-client.virtnet: 1 18490 0.35 MB/sec execute 222 sec latency 8289.868 ms oleg119-client.virtnet: 1 18490 0.34 MB/sec execute 223 sec latency 9290.692 ms oleg119-client.virtnet: 1 18490 0.34 MB/sec execute 224 sec latency 10291.271 ms oleg119-client.virtnet: 1 18490 0.34 MB/sec execute 225 sec latency 11294.888 ms oleg119-client.virtnet: 1 18490 0.34 MB/sec execute 226 sec latency 12295.452 ms oleg119-client.virtnet: 1 18490 0.34 MB/sec execute 227 sec latency 13296.056 ms oleg119-client.virtnet: 1 18490 0.34 MB/sec execute 228 sec latency 14296.317 ms oleg119-client.virtnet: 1 18490 0.34 MB/sec execute 229 sec latency 15299.791 ms oleg119-client.virtnet: 1 18490 0.33 MB/sec execute 230 sec latency 16300.218 ms oleg119-client.virtnet: 1 18490 0.33 MB/sec execute 231 sec latency 17300.416 ms oleg119-client.virtnet: 1 18490 0.33 MB/sec execute 232 sec latency 18301.891 ms oleg119-client.virtnet: 1 18490 0.33 MB/sec execute 233 sec latency 19302.086 ms oleg119-client.virtnet: 1 18490 0.33 MB/sec execute 234 sec latency 20304.694 ms oleg119-client.virtnet: 1 18490 0.33 MB/sec execute 235 sec latency 21306.592 ms oleg119-client.virtnet: 1 18490 0.33 MB/sec execute 236 sec latency 22306.822 ms oleg119-client.virtnet: 1 18490 0.32 MB/sec execute 237 sec latency 23307.203 ms oleg119-client.virtnet: 1 18490 0.32 MB/sec execute 238 sec latency 24308.356 ms oleg119-client.virtnet: 1 18490 0.32 MB/sec execute 239 sec latency 25309.359 ms oleg119-client.virtnet: 1 18490 0.32 MB/sec execute 240 sec latency 26311.829 ms oleg119-client.virtnet: 1 18543 0.32 MB/sec execute 2oleg119-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 41 sec latency 26631.810 ms oleg119-client.virtnet: 1 18647 0.32 MB/sec execute 242 sec latency 65.960 ms oleg119-client.virtnet: 1 18730 0.32 MB/sec execute 243 sec latency 62.301 ms oleg119-client.virtnet: 1 18834 0.32 MB/sec execute 244 sec latency 69.685 ms oleg119-client.virtnet: 1 18952 0.32 MB/sec execute 245 sec latency 39.010 ms oleg119-client.virtnet: 1 19138 0.33 MB/sec execute 246 sec latency 49.015 ms oleg119-client.virtnet: 1 19250 0.33 MB/sec execute 247 sec latency 46.109 ms oleg119-client.virtnet: 1 19363 0.33 MB/sec execute 248 sec latency 44.957 ms oleg119-client.virtnet: 1 19471 0.33 MB/sec execute 249 sec latency 52.025 ms oleg119-client.virtnet: 1 19568 0.33 MB/sec execute 250 sec latency 40.359 ms oleg119-client.virtnet: 1 19705 0.33 MB/sec execute 251 sec latency 61.521 ms oleg119-client.virtnet: 1 19817 0.33 MB/sec execute 252 sec latency 37.053 ms oleg119-client.virtnet: 1 19939 0.32 MB/sec execute 253 sec latency 49.409 ms oleg119-client.virtnet: 1 20050 0.32 MB/sec execute 254 sec latency 48.910 ms oleg119-client.virtnet: 1 20181 0.33 MB/sec execute 255 sec latency 53.364 ms oleg119-client.virtnet: 1 20275 0.33 MB/sec execute 256 sec latency 49.006 ms oleg119-client.virtnet: 1 20556 0.34 MB/sec execute 257 sec latency 40.890 ms oleg119-client.virtnet: 1 20780 0.34 MB/sec execute 258 sec latency 40.051 ms oleg119-client.virtnet: 1 21108 0.35 MB/sec execute 259 sec latency 44.141 ms oleg119-client.virtnet: 1 21330 0.36 MB/sec execute 260 sec latency 48.444 ms oleg119-client.virtnet: 1 21455 0.36 MB/sec execute 261 sec latency 43.042 ms oleg119-client.virtnet: 1 21589 0.36 MB/sec execute 262 sec latency 37.305 ms oleg119-client.virtnet: 1 21714 0.36 MB/sec execute 263 sec latency 54.096 ms oleg119-client.virtnet: 1 21802 0.36 MB/sec execute 264 sec latency 48.275 ms oleg119-client.virtnet: 1 21945 0.36 MB/sec execute 265 sec latency 40.339 ms oleg119-client.virtnet: 1 22057 0.36 MB/sec execute 266 sec latency 66.922 ms oleg119-client.virtnet: 1 22209 0.36 MB/sec execute 267 sec latency 41.366 ms oleg119-client.virtnet: 1 22317 0.36 MB/sec execute 268 sec latency 41.605 ms oleg119-client.virtnet: 1 22477 0.36 MB/sec execute 269 sec latency 33.797 ms oleg119-client.virtnet: 1 22606 0.36 MB/sec execute 270 sec latency 43.245 ms oleg119-client.virtnet: 1 22801 0.37 MB/sec execute 271 sec latency 55.402 ms oleg119-client.virtnet: 1 22916 0.37 MB/sec execute 272 sec latency 41.624 ms oleg119-client.virtnet: 1 23010 0.37 MB/sec execute 273 sec latency 51.673 ms oleg119-client.virtnet: 1 23111 0.37 MB/sec execute 274 sec latency 63.312 ms oleg119-client.virtnet: 1 23198 0.37 MB/sec execute 275 sec latency 56.166 ms oleg119-client.virtnet: 1 23346 0.37 MB/sec execute 276 sec latency 40.492 ms oleg119-client.virtnet: 1 23508 0.37 MB/sec execute 277 sec latency 56.138 ms oleg119-client.virtnet: 1 23684 0.37 MB/sec execute 278 sec latency 43.410 ms oleg119-client.virtnet: 1 23772 0.37 MB/sec execute 279 sec latency 62.939 ms oleg119-client.virtnet: 1 23916 0.37 MB/sec execute 280 sec latency 53.952 ms oleg119-client.virtnet: 1 24226 0.38 MB/sec execute 281 sec latency 57.677 ms oleg119-client.virtnet: 1 24499 0.39 MB/sec execute 282 sec latency 47.513 ms oleg119-client.virtnet: 1 24755 0.39 MB/sec execute 283 sec latency 40.893 ms oleg119-client.virtnet: 1 24943 0.40 MB/sec execute 284 sec latency 41.349 ms oleg119-client.virtnet: 1 25075 0.40 MB/sec execute 285 sec latency 48.279 ms oleg119-client.virtnet: 1 25167 0.40 MB/sec execute 286 sec latency 44.779 ms oleg119-client.virtnet: 1 25267 0.40 MB/sec execute 287 sec latency 40.885 ms oleg119-client.virtnet: 1 25359 0.40 MB/sec execute 288 sec latency 73.809 ms oleg119-client.virtnet: 1 25499 0.40 MB/sec execute 289 sec latency 43.053 ms oleg119-client.virtnet: 1 25615 0.40 MB/sec execute 290 sec latency 67.672 ms oleg119-client.virtnet: 1 25710 0.40 MB/sec execute 291 sec latency 50.659 ms oleg119-client.virtnet: 1 25848 0.40 MB/sec execute 292 sec latency 54.166 ms oleg119-client.virtnet: 1 26008 0.40 MB/sec execute 293 sec latency 52.666 ms oleg119-client.virtnet: 1 26109 0.40 MB/sec execute 294 sec latency 41.285 ms oleg119-client.virtnet: 1 26280 0.41 MB/sec execute 295 sec latency 58.090 ms oleg119-client.virtnet: 1 26375 0.40 MB/sec execute 296 sec latency 53.700 ms oleg119-client.virtnet: 1 26478 0.40 MB/sec execute 297 sec latency 55.832 ms oleg119-client.virtnet: 1 26580 0.40 MB/sec execute 298 sec latency 72.286 ms oleg119-client.virtnet: 1 26686 0.40 MB/sec execute 299 sec latency 43.112 ms oleg119-client.virtnet: 1 cleanup 300 sec oleg119-client.virtnet: 0 cleanup 301 sec oleg119-client.virtnet: oleg119-client.virtnet: Operation Count AvgLat MaxLat oleg119-client.virtnet: ---------------------------------------- oleg119-client.virtnet: NTCreateX 3953 40.583 26631.791 oleg119-client.virtnet: Close 2925 5.875 76.192 oleg119-client.virtnet: Rename 167 37.465 102.191 oleg119-client.virtnet: Unlink 796 8.975 43.946 oleg119-client.virtnet: Qpathinfo 3597 21.760 33043.900 oleg119-client.virtnet: Qfileinfo 623 0.381 15.025 oleg119-client.virtnet: Qfsinfo 646 1.044 37.728 oleg119-client.virtnet: Sfileinfo 334 26.644 74.518 oleg119-client.virtnet: Find 1377 1.502 77.482 oleg119-client.virtnet: WriteX 1946 3.947 45.659 oleg119-client.virtnet: ReadX 6086 0.059 32.197 oleg119-client.virtnet: LockX 12 6.846 11.910 oleg119-client.virtnet: UnlockX 12 5.233 8.527 oleg119-client.virtnet: Flush 288 36.281 1534.262 oleg119-client.virtnet: oleg119-client.virtnet: Throughput 0.401362 MB/sec 1 clients 1 procs max_latency=33043.935 ms oleg119-client.virtnet: stopping dbench on /mnt/lustre/d70b.replay-single/oleg119-client.virtnet at Thu May 1 18:08:39 EDT 2025 with return code 0 oleg119-client.virtnet: clean dbench files on /mnt/lustre/d70b.replay-single/oleg119-client.virtnet oleg119-client.virtnet: /mnt/lustre/d70b.replay-single/oleg119-client.virtnet /mnt/lustre/d70b.replay-single/oleg119-client.virtnet oleg119-client.virtnet: removed directory: 'clients/client0/~dmtmp/PARADOX' oleg119-client.virtnet: removed directory: 'clients/client0/~dmtmp/ACCESS' oleg119-client.virtnet: removed directory: 'clients/client0/~dmtmp/COREL' oleg119-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORDPRO' oleg119-client.virtnet: removed directory: 'clients/client0/~dmtmp/PWRPNT' oleg119-client.virtnet: removed directory: 'clients/client0/~dmtmp/PM' oleg119-client.virtnet: removed directory: 'clients/client0/~dmtmp/SEED' oleg119-client.virtnet: removed directory: 'clients/client0/~dmtmp/EXCEL' oleg119-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORD' oleg119-client.virtnet: removed directory: 'clients/client0/~dmtmp' oleg119-client.virtnet: removed directory: 'clients/client0' oleg119-client.virtnet: removed directory: 'clients' oleg119-client.virtnet: removed 'client.txt' oleg119-client.virtnet: /mnt/lustre/d70b.replay-single/oleg119-client.virtnet oleg119-client.virtnet: dbench successfully finished