== replay-single test 70b: dbench 1mdts recovery; 1 clients ========================================================== 21:52:19 (1746150739) Starting client oleg315-client.virtnet: -o user_xattr,flock 192.168.203.115@tcp:/lustre /mnt/lustre Started clients oleg315-client.virtnet: 192.168.203.115@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) + MISSING_DBENCH_OK= + PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin::/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests:/sbin:/usr/sbin:/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests/: + DBENCH_LIB= + TESTSUITE=replay-single + TESTNAME=test_70b + MOUNT=/mnt/lustre ++ hostname + DIR=/mnt/lustre/d70b.replay-single/oleg315-client.virtnet + LCTL=/home/green/git/lustre-release/lustre/utils/lctl + rundbench 1 -t 300 dbench: no process found Started rundbench load pid=6424 ... UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3712 2204672 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3736576 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3721216 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 6144 7457792 1% /mnt/lustre test_70b fail mds1 1 times Failing mds1 on oleg315-server Stopping /mnt/lustre-mds1 (opts:) on oleg315-server 21:52:29 (1746150749) shut down facet: mds1 facet_host: oleg315-server facet_failover_host: oleg315-server Failover mds1 to oleg315-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg315-server: oleg315-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg315-client: oleg315-server: ssh exited with exit code 1 Started lustre-MDT0000 21:52:44 (1746150764) targets are mounted 21:52:44 (1746150764) facet_failover done oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3968 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 32768 3688448 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 11264 3727360 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 44032 7415808 1% /mnt/lustre test_70b fail mds1 2 times Failing mds1 on oleg315-server Stopping /mnt/lustre-mds1 (opts:) on oleg315-server oleg315-client.virtnet: looking for dbench program oleg315-client.virtnet: /usr/bin/dbench oleg315-client.virtnet: creating output directory /mnt/lustre/d70b.replay-single/oleg315-client.virtnet oleg315-client.virtnet: mkdir: created directory '/mnt/lustre/d70b.replay-single' oleg315-client.virtnet: mkdir: created directory '/mnt/lustre/d70b.replay-single/oleg315-client.virtnet' oleg315-client.virtnet: found dbench client file /usr/share/dbench/client.txt oleg315-client.virtnet: '/usr/share/dbench/client.txt' -> 'client.txt' oleg315-client.virtnet: running 'dbench 1 -t 300' on /mnt/lustre/d70b.replay-single/oleg315-client.virtnet at Thu May 1 21:52:21 EDT 2025 oleg315-client.virtnet: waiting for dbench pid 6449 oleg315-client.virtnet: dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 oleg315-client.virtnet: oleg315-client.virtnet: Running for 300 seconds with load 'client.txt' and minimum warmup 60 secs oleg315-client.virtnet: failed to create barrier semaphore oleg315-client.virtnet: 0 of 1 processes prepared for launch 0 sec oleg315-client.virtnet: 1 of 1 processes prepared for launch 0 sec oleg315-client.virtnet: releasing clients oleg315-client.virtnet: 1 155 6.54 MB/sec warmup 1 sec latency 60.840 ms oleg315-client.virtnet: 1 339 6.13 MB/sec warmup 2 sec latency 45.989 ms oleg315-client.virtnet: 1 503 5.82 MB/sec warmup 3 sec latency 49.125 ms oleg315-client.virtnet: 1 676 5.29 MB/sec warmup 4 sec latency 62.975 ms oleg315-client.virtnet: 1 789 4.29 MB/sec warmup 5 sec latency 98.605 ms oleg315-client.virtnet: 1 910 3.62 MB/sec warmup 6 sec latency 85.351 ms oleg315-client.virtnet: 1 969 3.11 MB/sec warmup 7 sec latency 531.906 ms oleg315-client.virtnet: 1 969 2.72 MB/sec warmup 8 sec latency 1532.899 ms oleg315-client.virtnet: 1 969 2.42 MB/sec warmup 9 sec latency 2533.661 ms oleg315-client.virtnet: 1 969 2.18 MB/sec warmup 10 sec latency 3534.433 ms oleg315-client.virtnet: 1 969 1.98 MB/sec warmup 11 sec latency 4536.118 ms oleg315-client.virtnet: 1 969 1.82 MB/sec warmup 12 sec latency 5537.210 ms oleg315-client.virtnet: 1 969 1.68 MB/sec warmup 13 sec latency 6539.019 ms oleg315-client.virtnet: 1 969 1.56 MB/sec warmup 14 sec latency 7539.488 ms oleg315-client.virtnet: 1 969 1.45 MB/sec warmup 15 sec latency 8539.997 ms oleg315-client.virtnet: 1 969 1.36 MB/sec warmup 16 sec latency 9540.161 ms oleg315-client.virtnet: 1 969 1.28 MB/sec warmup 17 sec latency 10540.700 ms oleg315-client.virtnet: 1 969 1.21 MB/sec warmup 18 sec latency 11541.185 ms oleg315-client.virtnet: 1 969 1.15 MB/sec warmup 19 sec latency 12541.688 ms oleg315-client.virtnet: 1 969 1.09 MB/sec warmup 20 sec latency 13542.140 ms oleg315-client.virtnet: 1 969 1.04 MB/sec warmup 21 sec latency 14543.704 ms oleg315-client.virtnet: 1 969 0.99 MB/sec warmup 22 sec latency 15546.192 ms oleg315-client.virtnet: 1 969 0.95 MB/sec warmup 23 sec latency 16546.443 ms oleg315-client.virtnet: 1 969 0.91 MB/sec warmup 24 sec latency 17547.061 ms oleg315-client.virtnet: 1 1127 0.92 MB/sec warmup 25 sec latency 17642.784 ms oleg315-client.virtnet: 1 1263 0.89 MB/sec warmup 26 sec latency 41.565 ms oleg315-client.virtnet: 1 1516 0.96 MB/sec warmup 27 sec latency 79.934 ms oleg315-client.virtnet: 1 1689 0.94 MB/sec warmup 28 sec latency 90.575 ms oleg315-client.virtnet: 1 1884 0.91 MB/sec warmup 29 sec latency 62.774 ms oleg315-client.virtnet: 1 2088 0.89 MB/sec warmup 30 sec latency 83.451 ms oleg315-client.virtnet: 1 2297 0.89 MB/sec warmup 31 sec latency 41.058 ms oleg315-client.virtnet: 1 2423 0.88 MB/sec warmup 32 sec latency 112.601 ms ol21:52:56 (1746150776) shut down facet: mds1 facet_host: oleg315-server facet_failover_host: oleg315-server Failover mds1 to oleg315-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg315-server: oleg315-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg315-client: oleg315-server: ssh exited with exit code 1 Started lustre-MDT0000 21:53:11 (1746150791) targets are mounted 21:53:11 (1746150791) facet_failover done oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3968 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 40960 3723264 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 13312 3730432 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 54272 7453696 1% /mnt/lustre test_70b fail mds1 3 times Failing mds1 on oleg315-server Stopping /mnt/lustre-mds1 (opts:) on oleg315-server 21:53:23 (1746150803) shut down facet: mds1 facet_host: oleg315-server facet_failover_host: oleg315-server Failover mds1 to oleg315-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg315-server: oleg315-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg315-client: oleg315-server: ssh exited with exit code 1 Started lustre-MDT0000 21:53:38 (1746150818) targets are mounted 21:53:38 (1746150818) facet_failover done eg315-client.virtnet: 1 2594 0.88 MB/sec warmup 33 sec latency 119.643 ms oleg315-client.virtnet: 1 2762 0.94 MB/sec warmup 34 sec latency 702.595 ms oleg315-client.virtnet: 1 2762 0.91 MB/sec warmup 35 sec latency 1703.087 ms oleg315-client.virtnet: 1 2762 0.89 MB/sec warmup 36 sec latency 2704.298 ms oleg315-client.virtnet: 1 2762 0.86 MB/sec warmup 37 sec latency 3704.905 ms oleg315-client.virtnet: 1 2762 0.84 MB/sec warmup 38 sec latency 4705.911 ms oleg315-client.virtnet: 1 2762 0.82 MB/sec warmup 39 sec latency 5707.715 ms oleg315-client.virtnet: 1 2762 0.80 MB/sec warmup 40 sec latency 6708.883 ms oleg315-client.virtnet: 1 2762 0.78 MB/sec warmup 41 sec latency 7709.284 ms oleg315-client.virtnet: 1 2762 0.76 MB/sec warmup 42 sec latency 8709.990 ms oleg315-client.virtnet: 1 2762 0.74 MB/sec warmup 43 sec latency 9710.212 ms oleg315-client.virtnet: 1 2762 0.73 MB/sec warmup 44 sec latency 10710.524 ms oleg315-client.virtnet: 1 2762 0.71 MB/sec warmup 45 sec latency 11718.080 ms oleg315-client.virtnet: 1 2762 0.69 MB/sec warmup 46 sec latency 12718.948 ms oleg315-client.virtnet: 1 2762 0.68 MB/sec warmup 47 sec latency 13719.183 ms oleg315-client.virtnet: 1 2762 0.67 MB/sec warmup 48 sec latency 14719.372 ms oleg315-client.virtnet: 1 2762 0.65 MB/sec warmup 49 sec latency 15719.492 ms oleg315-client.virtnet: 1 2973 0.67 MB/sec warmup 50 sec latency 15959.626 ms oleg315-client.virtnet: 1 3237 0.68 MB/sec warmup 51 sec latency 104.056 ms oleg315-client.virtnet: 1 3572 0.74 MB/sec warmup 52 sec latency 41.831 ms oleg315-client.virtnet: 1 3780 0.75 MB/sec warmup 53 sec latency 34.479 ms oleg315-client.virtnet: 1 3945 0.74 MB/sec warmup 54 sec latency 31.280 ms oleg315-client.virtnet: 1 4065 0.73 MB/sec warmup 55 sec latency 35.692 ms oleg315-client.virtnet: 1 4228 0.72 MB/sec warmup 56 sec latency 71.998 ms oleg315-client.virtnet: 1 4301 0.71 MB/sec warmup 57 sec latency 150.582 ms oleg315-client.virtnet: 1 4377 0.70 MB/sec warmup 58 sec latency 125.209 ms oleg315-client.virtnet: 1 4486 0.69 MB/sec warmup 59 sec latency 88.615 ms oleg315-client.virtnet: 1 4693 0.10 MB/sec execute 1 sec latency 671.212 ms oleg315-client.virtnet: 1 4693 0.05 MB/sec execute 2 sec latency 1671.637 ms oleg315-client.virtnet: 1 4693 0.03 MB/sec execute 3 sec latency 2673.216 ms oleg315-client.virtnet: 1 4693 0.02 MB/sec execute 4 sec latency 3673.386 ms oleg315-client.virtnet: 1 4693 0.02 MB/sec execute 5 sec latency 4675.144 ms oleg315-client.virtnet: 1 4693 0.02 MB/sec execute 6 sec latency 5675.518 ms oleg315-client.virtnet: 1 4693 0.01 MB/sec execute 7 sec latency 6675.749 ms oleg315-client.virtnet: 1 4693 0.01 MB/sec execute 8 sec latency 7676.388 ms oleg315-client.virtnet: 1 4693 0.01 MB/sec execute 9 sec latency 8678.610 ms oleg315-client.virtnet: 1 4693 0.01 MB/sec execute 10 sec latency 9680.591 ms oleg315-client.virtnet: 1 4693 0.01 MB/sec execute 11 sec latency 10680.999 ms oleg315-client.virtnet: 1 4693 0.01 MB/sec execute 12 sec latency 11681.181 ms oleg315-client.virtnet: 1 4693 0.01 MB/sec execute 13 sec latency 12681.363 ms oleg315-client.virtnet: 1 4693 0.01 MB/sec execute 14 sec latency 13681.734 ms oleg315-client.virtnet: 1 4693 0.01 MB/sec execute 15 sec latency 14682.092 ms oleg315-client.virtnet: 1 4693 0.01 MB/sec execute 16 sec latency 15682.311 ms oleg315-client.virtnet: 1 4693 0.01 MB/sec execute 17 sec latency 16682.823 ms oleg315-client.virtnet: 1 469oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3968 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 40960 3721216 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 14336 3744768 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 55296 7465984 1% /mnt/lustre test_70b fail mds1 4 times Failing mds1 on oleg315-server Stopping /mnt/lustre-mds1 (opts:) on oleg315-server 21:53:52 (1746150832) shut down facet: mds1 facet_host: oleg315-server facet_failover_host: oleg315-server Failover mds1 to oleg315-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg315-server: oleg315-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg315-client: oleg315-server: ssh exited with exit code 1 Started lustre-MDT0000 21:54:08 (1746150848) targets are mounted 21:54:08 (1746150848) facet_failover done oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3968 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 40960 3681280 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 14336 3736576 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 55296 7417856 1% /mnt/lustre test_70b fail mds1 5 times Failing mds1 on oleg315-server Stopping /mnt/lustre-mds1 (opts:) on oleg315-server 21:54:22 (1746150862) shut down facet: mds1 facet_host: oleg315-server facet_failover_host: oleg315-server 3 0.01 MB/sec execute 18 sec latency 17684.096 ms oleg315-client.virtnet: 1 4694 0.01 MB/sec execute 19 sec latency 18669.200 ms oleg315-client.virtnet: 1 4799 0.01 MB/sec execute 20 sec latency 72.772 ms oleg315-client.virtnet: 1 4984 0.15 MB/sec execute 21 sec latency 53.255 ms oleg315-client.virtnet: 1 5082 0.14 MB/sec execute 22 sec latency 151.995 ms oleg315-client.virtnet: 1 5220 0.14 MB/sec execute 23 sec latency 105.795 ms oleg315-client.virtnet: 1 5354 0.14 MB/sec execute 24 sec latency 51.193 ms oleg315-client.virtnet: 1 5449 0.14 MB/sec execute 25 sec latency 81.865 ms oleg315-client.virtnet: 1 5577 0.14 MB/sec execute 26 sec latency 99.297 ms oleg315-client.virtnet: 1 5726 0.14 MB/sec execute 27 sec latency 47.589 ms oleg315-client.virtnet: 1 5943 0.18 MB/sec execute 28 sec latency 83.507 ms oleg315-client.virtnet: 1 6037 0.20 MB/sec execute 29 sec latency 129.842 ms oleg315-client.virtnet: 1 6129 0.20 MB/sec execute 30 sec latency 564.903 ms oleg315-client.virtnet: 1 6129 0.20 MB/sec execute 31 sec latency 1565.405 ms oleg315-client.virtnet: 1 6129 0.19 MB/sec execute 32 sec latency 2565.663 ms oleg315-client.virtnet: 1 6129 0.18 MB/sec execute 33 sec latency 3566.494 ms oleg315-client.virtnet: 1 6129 0.18 MB/sec execute 34 sec latency 4567.913 ms oleg315-client.virtnet: 1 6129 0.17 MB/sec execute 35 sec latency 5568.771 ms oleg315-client.virtnet: 1 6129 0.17 MB/sec execute 36 sec latency 6570.438 ms oleg315-client.virtnet: 1 6129 0.16 MB/sec execute 37 sec latency 7570.612 ms oleg315-client.virtnet: 1 6129 0.16 MB/sec execute 38 sec latency 8572.181 ms oleg315-client.virtnet: 1 6129 0.16 MB/sec execute 39 sec latency 9572.358 ms oleg315-client.virtnet: 1 6129 0.15 MB/sec execute 40 sec latency 10572.893 ms oleg315-client.virtnet: 1 6129 0.15 MB/sec execute 41 sec latency 11573.098 ms oleg315-client.virtnet: 1 6129 0.14 MB/sec execute 42 sec latency 12573.252 ms oleg315-client.virtnet: 1 6129 0.14 MB/sec execute 43 sec latency 13573.405 ms oleg315-client.virtnet: 1 6129 0.14 MB/sec execute 44 sec latency 14574.267 ms oleg315-client.virtnet: 1 6129 0.13 MB/sec execute 45 sec latency 15575.904 ms oleg315-client.virtnet: 1 6129 0.13 MB/sec execute 46 sec latency 16576.112 ms oleg315-client.virtnet: 1 6129 0.13 MB/sec execute 47 sec latency 17576.284 ms oleg315-client.virtnet: 1 6129 0.13 MB/sec execute 48 sec latency 18577.701 ms oleg315-client.virtnet: 1 6129 0.12 MB/sec execute 49 sec latency 19577.902 ms oleg315-client.virtnet: 1 6324 0.18 MB/sec execute 50 sec latency 19693.045 ms oleg315-client.virtnet: 1 6523 0.21 MB/sec execute 51 sec latency 206.007 ms oleg315-client.virtnet: 1 6752 0.22 MB/sec execute 52 sec latency 153.042 ms oleg315-client.virtnet: 1 7057 0.29 MB/sec execute 53 sec latency 85.357 ms oleg315-client.virtnet: 1 7211 0.29 MB/sec execute 54 sec latency 39.587 ms oleg315-client.virtnet: 1 7364 0.31 MB/sec execute 55 sec latency 33.923 ms oleg315-client.virtnet: 1 7492 0.31 MB/sec execute 56 sec latency 44.580 ms oleg315-client.virtnet: 1 7619 0.30 MB/sec execute 57 sec latency 30.407 ms oleg315-client.virtnet: 1 7788 0.30 MB/sec execute 58 sec latency 101.571 ms oleg315-client.virtnet: 1 7875 0.30 MB/sec execute 59 sec latency 73.587 ms oleg315-client.virtnet: 1 7893 0.29 MB/sec execute 60 sec latency 815.459 ms oleg315-client.virtnet: 1 7893 0.29 MB/sec execute 61 sec latency 1817.866 ms oleg315-client.virtnet: 1 7893 0.28 MB/sec executFailover mds1 to oleg315-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg315-server: oleg315-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg315-client: oleg315-server: ssh exited with exit code 1 Started lustre-MDT0000 21:54:37 (1746150877) targets are mounted 21:54:37 (1746150877) facet_failover done oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3968 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 40960 3716096 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 14336 3747840 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 55296 7463936 1% /mnt/lustre test_70b fail mds1 6 times Failing mds1 on oleg315-server Stopping /mnt/lustre-mds1 (opts:) on oleg315-server 21:54:52 (1746150892) shut down facet: mds1 facet_host: oleg315-server facet_failover_host: oleg315-server Failover mds1 to oleg315-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg315-server: oleg315-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all e 62 sec latency 2818.114 ms oleg315-client.virtnet: 1 7893 0.28 MB/sec execute 63 sec latency 3819.027 ms oleg315-client.virtnet: 1 7893 0.27 MB/sec execute 64 sec latency 4819.191 ms oleg315-client.virtnet: 1 7893 0.27 MB/sec execute 65 sec latency 5822.970 ms oleg315-client.virtnet: 1 7893 0.27 MB/sec execute 66 sec latency 6823.145 ms oleg315-client.virtnet: 1 7893 0.26 MB/sec execute 67 sec latency 7823.342 ms oleg315-client.virtnet: 1 7893 0.26 MB/sec execute 68 sec latency 8823.751 ms oleg315-client.virtnet: 1 7893 0.25 MB/sec execute 69 sec latency 9824.326 ms oleg315-client.virtnet: 1 7893 0.25 MB/sec execute 70 sec latency 10824.514 ms oleg315-client.virtnet: 1 7893 0.25 MB/sec execute 71 sec latency 11830.006 ms oleg315-client.virtnet: 1 7893 0.24 MB/sec execute 72 sec latency 12830.603 ms oleg315-client.virtnet: 1 7893 0.24 MB/sec execute 73 sec latency 13831.670 ms oleg315-client.virtnet: 1 7893 0.24 MB/sec execute 74 sec latency 14831.943 ms oleg315-client.virtnet: 1 7893 0.23 MB/sec execute 75 sec latency 15832.136 ms oleg315-client.virtnet: 1 7893 0.23 MB/sec execute 76 sec latency 16832.507 ms oleg315-client.virtnet: 1 7893 0.23 MB/sec execute 77 sec latency 17836.325 ms oleg315-client.virtnet: 1 7893 0.23 MB/sec execute 78 sec latency 18836.528 ms oleg315-client.virtnet: 1 7893 0.22 MB/sec execute 79 sec latency 19837.140 ms oleg315-client.virtnet: 1 7958 0.22 MB/sec execute 80 sec latency 20045.930 ms oleg315-client.virtnet: 1 8028 0.22 MB/sec execute 81 sec latency 111.556 ms oleg315-client.virtnet: 1 8091 0.22 MB/sec execute 82 sec latency 180.761 ms oleg315-client.virtnet: 1 8264 0.23 MB/sec execute 83 sec latency 192.926 ms oleg315-client.virtnet: 1 8500 0.26 MB/sec execute 84 sec latency 41.947 ms oleg315-client.virtnet: 1 8618 0.26 MB/sec execute 85 sec latency 183.929 ms oleg315-client.virtnet: 1 8740 0.26 MB/sec execute 86 sec latency 81.650 ms oleg315-client.virtnet: 1 8839 0.26 MB/sec execute 87 sec latency 79.619 ms oleg315-client.virtnet: 1 8970 0.26 MB/sec execute 88 sec latency 51.262 ms oleg315-client.virtnet: 1 9103 0.26 MB/sec execute 89 sec latency 119.028 ms oleg315-client.virtnet: 1 9147 0.25 MB/sec execute 90 sec latency 559.929 ms oleg315-client.virtnet: 1 9147 0.25 MB/sec execute 91 sec latency 1562.860 ms oleg315-client.virtnet: 1 9147 0.25 MB/sec execute 92 sec latency 2565.710 ms oleg315-client.virtnet: 1 9147 0.25 MB/sec execute 93 sec latency 3567.875 ms oleg315-client.virtnet: 1 9147 0.24 MB/sec execute 94 sec latency 4570.764 ms oleg315-client.virtnet: 1 9147 0.24 MB/sec execute 95 sec latency 5570.998 ms oleg315-client.virtnet: 1 9147 0.24 MB/sec execute 96 sec latency 6571.223 ms oleg315-client.virtnet: 1 9147 0.24 MB/sec execute 97 sec latency 7573.806 ms oleg315-client.virtnet: 1 9147 0.23 MB/sec execute 98 sec latency 8573.960 ms oleg315-client.virtnet: 1 9147 0.23 MB/sec execute 99 sec latency 9574.115 ms oleg315-client.virtnet: 1 9147 0.23 MB/sec execute 100 sec latency 10574.276 ms oleg315-client.virtnet: 1 9147 0.23 MB/sec execute 101 sec latency 11576.355 ms oleg315-client.virtnet: 1 9147 0.22 MB/sec execute 102 sec latency 12576.578 ms oleg315-client.virtnet: 1 9147 0.22 MB/sec execute 103 sec latency 13577.784 ms oleg315-client.virtnet: 1 9147 0.22 MB/sec execute 104 sec latency 14578.217 ms oleg315-client.virtnet: 1 9147 0.22 MB/sec execute 105 sec latency 15580.932 ms oleg315-client.virtnet: 1 9147 0.22 MB/sec executepdsh@oleg315-client: oleg315-server: ssh exited with exit code 1 Started lustre-MDT0000 21:55:08 (1746150908) targets are mounted 21:55:08 (1746150908) facet_failover done oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3968 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 40960 3715072 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 14336 3742720 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 55296 7457792 1% /mnt/lustre test_70b fail mds1 7 times Failing mds1 on oleg315-server Stopping /mnt/lustre-mds1 (opts:) on oleg315-server 21:55:22 (1746150922) shut down facet: mds1 facet_host: oleg315-server facet_failover_host: oleg315-server Failover mds1 to oleg315-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg315-server: oleg315-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg315-client: oleg315-server: ssh exited with exit code 1 Started lustre-MDT0000 21:55:39 (1746150939) targets are mounted 21:55:39 (1746150939) facet_failover done oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3968 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 40960 3726336 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 13312 3745792 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 54272 7472128 1% /mnt/lustre test_70b fail mds1 8 times Failing mds1 on oleg315-server 106 sec latency 16581.265 ms oleg315-client.virtnet: 1 9147 0.21 MB/sec execute 107 sec latency 17581.469 ms oleg315-client.virtnet: 1 9147 0.21 MB/sec execute 108 sec latency 18583.861 ms oleg315-client.virtnet: 1 9148 0.21 MB/sec execute 109 sec latency 19533.126 ms oleg315-client.virtnet: 1 9287 0.21 MB/sec execute 110 sec latency 83.837 ms oleg315-client.virtnet: 1 9449 0.22 MB/sec execute 111 sec latency 57.454 ms oleg315-client.virtnet: 1 9510 0.22 MB/sec execute 112 sec latency 140.052 ms oleg315-client.virtnet: 1 9673 0.22 MB/sec execute 113 sec latency 132.707 ms oleg315-client.virtnet: 1 9996 0.25 MB/sec execute 114 sec latency 110.424 ms oleg315-client.virtnet: 1 10111 0.26 MB/sec execute 115 sec latency 126.381 ms oleg315-client.virtnet: 1 10370 0.27 MB/sec execute 116 sec latency 173.970 ms oleg315-client.virtnet: 1 10622 0.30 MB/sec execute 117 sec latency 60.989 ms oleg315-client.virtnet: 1 10797 0.30 MB/sec execute 118 sec latency 33.860 ms oleg315-client.virtnet: 1 10918 0.30 MB/sec execute 119 sec latency 40.653 ms oleg315-client.virtnet: 1 10974 0.30 MB/sec execute 120 sec latency 392.973 ms oleg315-client.virtnet: 1 10974 0.30 MB/sec execute 121 sec latency 1393.103 ms oleg315-client.virtnet: 1 10974 0.29 MB/sec execute 122 sec latency 2393.281 ms oleg315-client.virtnet: 1 10974 0.29 MB/sec execute 123 sec latency 3393.442 ms oleg315-client.virtnet: 1 10974 0.29 MB/sec execute 124 sec latency 4393.823 ms oleg315-client.virtnet: 1 10974 0.29 MB/sec execute 125 sec latency 5393.980 ms oleg315-client.virtnet: 1 10974 0.29 MB/sec execute 126 sec latency 6394.147 ms oleg315-client.virtnet: 1 10974 0.28 MB/sec execute 127 sec latency 7394.631 ms oleg315-client.virtnet: 1 10974 0.28 MB/sec execute 128 sec latency 8395.169 ms oleg315-client.virtnet: 1 10974 0.28 MB/sec execute 129 sec latency 9395.821 ms oleg315-client.virtnet: 1 10974 0.28 MB/sec execute 130 sec latency 10396.624 ms oleg315-client.virtnet: 1 10974 0.27 MB/sec execute 131 sec latency 11396.844 ms oleg315-client.virtnet: 1 10974 0.27 MB/sec execute 132 sec latency 12398.639 ms oleg315-client.virtnet: 1 10974 0.27 MB/sec execute 133 sec latency 13398.858 ms oleg315-client.virtnet: 1 10974 0.27 MB/sec execute 134 sec latency 14399.081 ms oleg315-client.virtnet: 1 10974 0.27 MB/sec execute 135 sec latency 15399.248 ms oleg315-client.virtnet: 1 10974 0.26 MB/sec execute 136 sec latency 16399.659 ms oleg315-client.virtnet: 1 10974 0.26 MB/sec execute 137 sec latency 17400.603 ms oleg315-client.virtnet: 1 10974 0.26 MB/sec execute 138 sec latency 18402.850 ms oleg315-client.virtnet: 1 10974 0.26 MB/sec execute 139 sec latency 19402.991 ms oleg315-client.virtnet: 1 11010 0.26 MB/sec execute 140 sec latency 19980.021 ms oleg315-client.virtnet: 1 11086 0.26 MB/sec execute 141 sec latency 64.733 ms oleg315-client.virtnet: 1 11179 0.26 MB/sec execute 142 sec latency 62.210 ms oleg315-client.virtnet: 1 11310 0.25 MB/sec execute 143 sec latency 38.122 ms oleg315-client.virtnet: 1 11372 0.25 MB/sec execute 144 sec latency 163.114 ms oleg315-client.virtnet: 1 11449 0.25 MB/sec execute 145 sec latency 253.689 ms oleg315-client.virtnet: 1 11533 0.25 MB/sec execute 146 sec latency 142.006 ms oleg315-client.virtnet: 1 11615 0.25 MB/sec execute 147 sec latency 120.544 ms oleg315-client.virtnet: 1 11707 0.26 MB/sec execute 148 sec latency 112.458 ms oleg315-client.virtnet: 1 11864 0.26 MB/sec execute 149 sec latency 38.612 ms oleg315-client.virtnet: 1 12038 0.28 MB/sec execute 150 sec latency Stopping /mnt/lustre-mds1 (opts:) on oleg315-server 21:55:54 (1746150954) shut down facet: mds1 facet_host: oleg315-server facet_failover_host: oleg315-server Failover mds1 to oleg315-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg315-server: oleg315-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg315-client: oleg315-server: ssh exited with exit code 1 Started lustre-MDT0000 21:56:09 (1746150969) targets are mounted 21:56:09 (1746150969) facet_failover done oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3968 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 40960 3716096 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 13312 3748864 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 54272 7464960 1% /mnt/lustre test_70b fail mds1 9 times Failing mds1 on oleg315-server Stopping /mnt/lustre-mds1 (opts:) on oleg315-server 21:56:28 (1746150988) shut down facet: mds1 facet_host: oleg315-server facet_failover_host: oleg315-server 60.768 ms oleg315-client.virtnet: 1 12111 0.27 MB/sec execute 151 sec latency 307.989 ms oleg315-client.virtnet: 1 12111 0.27 MB/sec execute 152 sec latency 1308.171 ms oleg315-client.virtnet: 1 12111 0.27 MB/sec execute 153 sec latency 2308.810 ms oleg315-client.virtnet: 1 12111 0.27 MB/sec execute 154 sec latency 3308.965 ms oleg315-client.virtnet: 1 12111 0.27 MB/sec execute 155 sec latency 4309.175 ms oleg315-client.virtnet: 1 12111 0.26 MB/sec execute 156 sec latency 5310.044 ms oleg315-client.virtnet: 1 12111 0.26 MB/sec execute 157 sec latency 6310.464 ms oleg315-client.virtnet: 1 12111 0.26 MB/sec execute 158 sec latency 7312.252 ms oleg315-client.virtnet: 1 12111 0.26 MB/sec execute 159 sec latency 8312.719 ms oleg315-client.virtnet: 1 12111 0.26 MB/sec execute 160 sec latency 9313.413 ms oleg315-client.virtnet: 1 12111 0.26 MB/sec execute 161 sec latency 10313.749 ms oleg315-client.virtnet: 1 12111 0.26 MB/sec execute 162 sec latency 11313.996 ms oleg315-client.virtnet: 1 12111 0.25 MB/sec execute 163 sec latency 12319.419 ms oleg315-client.virtnet: 1 12111 0.25 MB/sec execute 164 sec latency 13319.631 ms oleg315-client.virtnet: 1 12111 0.25 MB/sec execute 165 sec latency 14321.096 ms oleg315-client.virtnet: 1 12111 0.25 MB/sec execute 166 sec latency 15321.537 ms oleg315-client.virtnet: 1 12111 0.25 MB/sec execute 167 sec latency 16321.859 ms oleg315-client.virtnet: 1 12111 0.25 MB/sec execute 168 sec latency 17322.071 ms oleg315-client.virtnet: 1 12111 0.24 MB/sec execute 169 sec latency 18322.444 ms oleg315-client.virtnet: 1 12215 0.24 MB/sec execute 170 sec latency 18329.292 ms oleg315-client.virtnet: 1 12308 0.24 MB/sec execute 171 sec latency 78.773 ms oleg315-client.virtnet: 1 12432 0.24 MB/sec execute 172 sec latency 57.368 ms oleg315-client.virtnet: 1 12542 0.24 MB/sec execute 173 sec latency 46.283 ms oleg315-client.virtnet: 1 12674 0.24 MB/sec execute 174 sec latency 125.106 ms oleg315-client.virtnet: 1 12803 0.24 MB/sec execute 175 sec latency 117.995 ms oleg315-client.virtnet: 1 12957 0.24 MB/sec execute 176 sec latency 56.536 ms oleg315-client.virtnet: 1 13076 0.25 MB/sec execute 177 sec latency 111.962 ms oleg315-client.virtnet: 1 13261 0.25 MB/sec execute 178 sec latency 75.491 ms oleg315-client.virtnet: 1 13581 0.27 MB/sec execute 179 sec latency 91.696 ms oleg315-client.virtnet: 1 13637 0.27 MB/sec execute 180 sec latency 715.638 ms oleg315-client.virtnet: 1 13637 0.27 MB/sec execute 181 sec latency 1715.813 ms oleg315-client.virtnet: 1 13637 0.27 MB/sec execute 182 sec latency 2715.988 ms oleg315-client.virtnet: 1 13637 0.27 MB/sec execute 183 sec latency 3716.607 ms oleg315-client.virtnet: 1 13637 0.27 MB/sec execute 184 sec latency 4717.058 ms oleg315-client.virtnet: 1 13637 0.26 MB/sec execute 185 sec latency 5717.312 ms oleg315-client.virtnet: 1 13637 0.26 MB/sec execute 186 sec latency 6717.514 ms oleg315-client.virtnet: 1 13637 0.26 MB/sec execute 187 sec latency 7720.154 ms oleg315-client.virtnet: 1 13637 0.26 MB/sec execute 188 sec latency 8720.406 ms oleg315-client.virtnet: 1 13637 0.26 MB/sec execute 189 sec latency 9720.695 ms oleg315-client.virtnet: 1 13637 0.26 MB/sec execute 190 sec latency 10721.696 ms oleg315-client.virtnet: 1 13637 0.26 MB/sec execute 191 sec latency 11721.861 ms oleg315-client.virtnet: 1 13637 0.25 MB/sec execute 192 sec latency 12722.093 ms oleg315-client.virtnet: 1 13637 0.25 MB/sec execute 193 sec latency 13722.464 ms oleg315-client.virtnet: 1 13637 0.25 MB/sec execute 194 sec latency 14722.622Failover mds1 to oleg315-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg315-server: oleg315-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg315-client: oleg315-server: ssh exited with exit code 1 Started lustre-MDT0000 21:56:43 (1746151003) targets are mounted 21:56:43 (1746151003) facet_failover done oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3968 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 40960 3708928 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 14336 3752960 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 55296 7461888 1% /mnt/lustre test_70b fail mds1 10 times Failing mds1 on oleg315-server Stopping /mnt/lustre-mds1 (opts:) on oleg315-server 21:57:03 (1746151023) shut down facet: mds1 facet_host: oleg315-server facet_failover_host: oleg315-server Failover mds1 to oleg315-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg315-server: oleg315-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg315-client: oleg315-server: ssh exited with exit code 1 Started lustre-MDT0000 21:57:19 (1746151039) targets are mounted 21:57:19 (1746151039) facet_failover done ms oleg315-client.virtnet: 1 13637 0.25 MB/sec execute 195 sec latency 15722.814 ms oleg315-client.virtnet: 1 13637 0.25 MB/sec execute 196 sec latency 16723.054 ms oleg315-client.virtnet: 1 13637 0.25 MB/sec execute 197 sec latency 17725.976 ms oleg315-client.virtnet: 1 13637 0.25 MB/sec execute 198 sec latency 18728.426 ms oleg315-client.virtnet: 1 13637 0.25 MB/sec execute 199 sec latency 19728.668 ms oleg315-client.virtnet: 1 13637 0.24 MB/sec execute 200 sec latency 20729.319 ms oleg315-client.virtnet: 1 13637 0.24 MB/sec execute 201 sec latency 21729.518 ms oleg315-client.virtnet: 1 13637 0.24 MB/sec execute 202 sec latency 22731.650 ms oleg315-client.virtnet: 1 13637 0.24 MB/sec execute 203 sec latency 23739.101 ms oleg315-client.virtnet: 1 13637 0.24 MB/sec execute 204 sec latency 24739.300 ms oleg315-client.virtnet: 1 13637 0.24 MB/sec execute 205 sec latency 25739.541 ms oleg315-client.virtnet: 1 13637 0.24 MB/sec execute 206 sec latency 26739.697 ms oleg315-client.virtnet: 1 13637 0.24 MB/sec execute 207 sec latency 27740.093 ms oleg315-client.virtnet: 1 13637 0.23 MB/sec execute 208 sec latency 28741.237 ms oleg315-client.virtnet: 1 13637 0.23 MB/sec execute 209 sec latency 29741.638 ms oleg315-client.virtnet: 1 13826 0.24 MB/sec execute 210 sec latency 29923.528 ms oleg315-client.virtnet: 1 13983 0.24 MB/sec execute 211 sec latency 117.966 ms oleg315-client.virtnet: 1 14180 0.25 MB/sec execute 212 sec latency 67.766 ms oleg315-client.virtnet: 1 14356 0.25 MB/sec execute 213 sec latency 36.858 ms oleg315-client.virtnet: 1 14491 0.26 MB/sec execute 214 sec latency 68.888 ms oleg315-client.virtnet: 1 14614 0.26 MB/sec execute 215 sec latency 48.799 ms oleg315-client.virtnet: 1 14703 0.26 MB/sec execute 216 sec latency 80.232 ms oleg315-client.virtnet: 1 14857 0.26 MB/sec execute 217 sec latency 54.826 ms oleg315-client.virtnet: 1 14935 0.26 MB/sec execute 218 sec latency 106.174 ms oleg315-client.virtnet: 1 15008 0.26 MB/sec execute 219 sec latency 144.721 ms oleg315-client.virtnet: 1 15108 0.25 MB/sec execute 220 sec latency 108.541 ms oleg315-client.virtnet: 1 15119 0.25 MB/sec execute 221 sec latency 904.537 ms oleg315-client.virtnet: 1 15119 0.25 MB/sec execute 222 sec latency 1904.958 ms oleg315-client.virtnet: 1 15119 0.25 MB/sec execute 223 sec latency 2905.141 ms oleg315-client.virtnet: 1 15119 0.25 MB/sec execute 224 sec latency 3905.303 ms oleg315-client.virtnet: 1 15119 0.25 MB/sec execute 225 sec latency 4905.506 ms oleg315-client.virtnet: 1 15119 0.25 MB/sec execute 226 sec latency 5906.488 ms oleg315-client.virtnet: 1 15119 0.25 MB/sec execute 227 sec latency 6916.108 ms oleg315-client.virtnet: 1 15119 0.25 MB/sec execute 228 sec latency 7916.735 ms oleg315-client.virtnet: 1 15119 0.25 MB/sec execute 229 sec latency 8917.488 ms oleg315-client.virtnet: 1 15119 0.24 MB/sec execute 230 sec latency 9917.754 ms oleg315-client.virtnet: 1 15119 0.24 MB/sec execute 231 sec latency 10917.924 ms oleg315-client.virtnet: 1 15119 0.24 MB/sec execute 232 sec latency 11919.013 ms oleg315-client.virtnet: 1 15119 0.24 MB/sec execute 233 sec latency 12920.387 ms oleg315-client.virtnet: 1 15119 0.24 MB/sec execute 234 sec latency 13920.569 ms oleg315-client.virtnet: 1 15119 0.24 MB/sec execute 235 sec latency 14923.370 ms oleg315-client.virtnet: 1 15119 0.24 MB/sec execute 236 sec latency 15923.552 ms oleg315-client.virtnet: 1 15119 0.24 MB/sec execute 237 sec latency 16925.737 ms oleg315-client.virtnet: 1 15119 0.24 MB/sec execute 238 sec latency 17925.oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 945 ms oleg315-client.virtnet: 1 15119 0.24 MB/sec execute 239 sec latency 18926.091 ms oleg315-client.virtnet: 1 15211 0.24 MB/sec execute 240 sec latency 18970.964 ms oleg315-client.virtnet: 1 15426 0.24 MB/sec execute 241 sec latency 39.953 ms oleg315-client.virtnet: 1 15691 0.25 MB/sec execute 242 sec latency 90.448 ms oleg315-client.virtnet: 1 15888 0.25 MB/sec execute 243 sec latency 70.996 ms oleg315-client.virtnet: 1 16109 0.25 MB/sec execute 244 sec latency 54.724 ms oleg315-client.virtnet: 1 16388 0.25 MB/sec execute 245 sec latency 24.054 ms oleg315-client.virtnet: 1 16612 0.25 MB/sec execute 246 sec latency 89.129 ms oleg315-client.virtnet: 1 16772 0.26 MB/sec execute 247 sec latency 154.156 ms oleg315-client.virtnet: 1 17116 0.27 MB/sec execute 248 sec latency 93.012 ms oleg315-client.virtnet: 1 17429 0.28 MB/sec execute 249 sec latency 83.649 ms oleg315-client.virtnet: 1 17732 0.29 MB/sec execute 250 sec latency 55.553 ms oleg315-client.virtnet: 1 17943 0.30 MB/sec execute 251 sec latency 44.692 ms oleg315-client.virtnet: 1 18080 0.30 MB/sec execute 252 sec latency 50.564 ms oleg315-client.virtnet: 1 18206 0.29 MB/sec execute 253 sec latency 33.170 ms oleg315-client.virtnet: 1 18372 0.29 MB/sec execute 254 sec latency 90.305 ms oleg315-client.virtnet: 1 18476 0.29 MB/sec execute 255 sec latency 185.657 ms oleg315-client.virtnet: 1 18553 0.29 MB/sec execute 256 sec latency 126.454 ms oleg315-client.virtnet: 1 18667 0.29 MB/sec execute 257 sec latency 112.585 ms oleg315-client.virtnet: 1 18816 0.30 MB/sec execute 258 sec latency 118.750 ms oleg315-client.virtnet: 1 18979 0.30 MB/sec execute 259 sec latency 43.329 ms oleg315-client.virtnet: 1 19219 0.31 MB/sec execute 260 sec latency 33.077 ms oleg315-client.virtnet: 1 19341 0.31 MB/sec execute 261 sec latency 103.740 ms oleg315-client.virtnet: 1 19475 0.30 MB/sec execute 262 sec latency 97.565 ms oleg315-client.virtnet: 1 19635 0.30 MB/sec execute 263 sec latency 56.320 ms oleg315-client.virtnet: 1 19835 0.30 MB/sec execute 264 sec latency 126.453 ms oleg315-client.virtnet: 1 20013 0.30 MB/sec execute 265 sec latency 33.739 ms oleg315-client.virtnet: 1 20192 0.31 MB/sec execute 266 sec latency 86.642 ms oleg315-client.virtnet: 1 20354 0.31 MB/sec execute 267 sec latency 119.880 ms oleg315-client.virtnet: 1 20702 0.32 MB/sec execute 268 sec latency 136.692 ms oleg315-client.virtnet: 1 20977 0.33 MB/sec execute 269 sec latency 91.370 ms oleg315-client.virtnet: 1 21281 0.34 MB/sec execute 270 sec latency 104.818 ms oleg315-client.virtnet: 1 21463 0.34 MB/sec execute 271 sec latency 32.796 ms oleg315-client.virtnet: 1 21641 0.34 MB/sec execute 272 sec latency 26.118 ms oleg315-client.virtnet: 1 21768 0.34 MB/sec execute 273 sec latency 39.458 ms oleg315-client.virtnet: 1 21901 0.34 MB/sec execute 274 sec latency 45.911 ms oleg315-client.virtnet: 1 22046 0.34 MB/sec execute 275 sec latency 120.004 ms oleg315-client.virtnet: 1 22137 0.34 MB/sec execute 276 sec latency 112.729 ms oleg315-client.virtnet: 1 22249 0.34 MB/sec execute 277 sec latency 110.130 ms oleg315-client.virtnet: 1 22378 0.34 MB/sec execute 278 sec latency 110.497 ms oleg315-client.virtnet: 1 22560 0.34 MB/sec execute 279 sec latency 31.891 ms oleg315-client.virtnet: 1 22786 0.35 MB/sec execute 280 sec latency 44.631 ms oleg315-client.virtnet: 1 22929 0.35 MB/sec execute 281 sec latency 110.096 ms oleg315-client.virtnet: 1 23059 0.35 MB/sec execute 282 sec latency 100.392 ms oleg315-client.virtnet: 1 23214 0.35 MB/sec execute 283 sec latency 35.281 ms oleg315-client.virtnet: 1 23390 0.35 MB/sec execute 284 sec latency 88.100 ms oleg315-client.virtnet: 1 23621 0.35 MB/sec execute 285 sec latency 36.392 ms oleg315-client.virtnet: 1 23746 0.35 MB/sec execute 286 sec latency 109.317 ms oleg315-client.virtnet: 1 23929 0.35 MB/sec execute 287 sec latency 77.595 ms oleg315-client.virtnet: 1 24255 0.37 MB/sec execute 288 sec latency 128.464 ms oleg315-client.virtnet: 1 24555 0.37 MB/sec execute 289 sec latency 100.688 ms oleg315-client.virtnet: 1 24864 0.38 MB/sec execute 290 sec latency 60.328 ms oleg315-client.virtnet: 1 25036 0.38 MB/sec execute 291 sec latency 60.187 ms oleg315-client.virtnet: 1 25220 0.39 MB/sec execute 292 sec latency 23.768 ms oleg315-client.virtnet: 1 25359 0.38 MB/sec execute 293 sec latency 30.501 ms oleg315-client.virtnet: 1 25551 0.38 MB/sec execute 294 sec latency 42.510 ms oleg315-client.virtnet: 1 25651 0.38 MB/sec execute 295 sec latency 127.507 ms oleg315-client.virtnet: 1 25770 0.38 MB/sec execute 296 sec latency 74.259 ms oleg315-client.virtnet: 1 25875 0.38 MB/sec execute 297 sec latency 91.287 ms oleg315-client.virtnet: 1 26112 0.39 MB/sec execute 298 sec latency 22.499 ms oleg315-client.virtnet: 1 26347 0.39 MB/sec execute 299 sec latency 34.945 ms oleg315-client.virtnet: 1 cleanup 300 sec oleg315-client.virtnet: 0 cleanup 301 sec oleg315-client.virtnet: oleg315-client.virtnet: Operation Count AvgLat MaxLat oleg315-client.virtnet: ---------------------------------------- oleg315-client.virtnet: NTCreateX 3796 41.074 29923.519 oleg315-client.virtnet: Close 2792 3.980 73.806 oleg315-client.virtnet: Rename 162 26.665 162.571 oleg315-client.virtnet: Unlink 755 6.482 117.975 oleg315-client.virtnet: Qpathinfo 3444 10.208 19693.030 oleg315-client.virtnet: Qfileinfo 601 0.275 4.811 oleg315-client.virtnet: Qfsinfo 622 0.464 18.573 oleg315-client.virtnet: Sfileinfo 316 76.745 18669.186 oleg315-client.virtnet: Find 1320 1.033 37.987 oleg315-client.virtnet: WriteX 1859 12.930 19533.112 oleg315-client.virtnet: ReadX 5941 0.062 21.463 oleg315-client.virtnet: LockX 12 4.577 11.373 oleg315-client.virtnet: UnlockX 12 3.397 4.981 oleg315-client.virtnet: Flush 260 145.576 18970.951 oleg315-client.virtnet: oleg315-client.virtnet: Throughput 0.394046 MB/sec 1 clients 1 procs max_latency=29923.528 ms oleg315-client.virtnet: stopping dbench on /mnt/lustre/d70b.replay-single/oleg315-client.virtnet at Thu May 1 21:58:22 EDT 2025 with return code 0 oleg315-client.virtnet: clean dbench files on /mnt/lustre/d70b.replay-single/oleg315-client.virtnet oleg315-client.virtnet: /mnt/lustre/d70b.replay-single/oleg315-client.virtnet /mnt/lustre/d70b.replay-single/oleg315-client.virtnet oleg315-client.virtnet: removed directory: 'clients/client0/~dmtmp/ACCESS' oleg315-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORDPRO' oleg315-client.virtnet: removed directory: 'clients/client0/~dmtmp/SEED' oleg315-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORD' oleg315-client.virtnet: removed directory: 'clients/client0/~dmtmp/PM' oleg315-client.virtnet: removed directory: 'clients/client0/~dmtmp/COREL' oleg315-client.virtnet: removed directory: 'clients/client0/~dmtmp/PARADOX' oleg315-client.virtnet: removed directory: 'clients/client0/~dmtmp/PWRPNT' oleg315-client.virtnet: removed directory: 'clients/client0/~dmtmp/EXCEL' oleg315-client.virtnet: removed directory: 'clients/client0/~dmtmp' oleg315-client.virtnet: removed directory: 'clients/client0' oleg315-client.virtnet: removed directory: 'clients' oleg315-client.virtnet: removed 'client.txt' oleg315-client.virtnet: /mnt/lustre/d70b.replay-single/oleg315-client.virtnet oleg315-client.virtnet: dbench successfully finished