== replay-single test 70b: dbench 1mdts recovery; 1 clients ========================================================== 21:48:27 (1745459307) Starting client oleg342-client.virtnet: -o user_xattr,flock 192.168.203.142@tcp:/lustre /mnt/lustre Started clients oleg342-client.virtnet: 192.168.203.142@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,noencrypt,flock,lazystatfs,lruresize,nolock,statfs_project,nouser_fid2path,user_xattr,verbose) + MISSING_DBENCH_OK= + PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin::/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests:/sbin:/usr/sbin:/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests/: + DBENCH_LIB= + TESTSUITE=replay-single + TESTNAME=test_70b + MOUNT=/mnt/lustre ++ hostname + DIR=/mnt/lustre/d70b.replay-single/oleg342-client.virtnet + LCTL=/home/green/git/lustre-release/lustre/utils/lctl + rundbench 1 -t 300 dbench: no process found Started rundbench load pid=5595 ... UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3840 2204544 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 7168 3726336 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3685376 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 39936 7411712 1% /mnt/lustre test_70b fail mds1 1 times Failing mds1 on oleg342-server Stopping /mnt/lustre-mds1 (opts:) on oleg342-server 21:48:33 (1745459313) shut down facet: mds1 facet_host: oleg342-server facet_failover_host: oleg342-server Failover mds1 to oleg342-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg342-server: oleg342-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg342-client: oleg342-server: ssh exited with exit code 1 Started lustre-MDT0000 21:48:46 (1745459326) targets are mounted 21:48:46 (1745459326) facet_failover done oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3968 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 14336 3727360 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 32768 3679232 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 47104 7406592 1% /mnt/lustre test_70b fail mds1 2 times Failing mds1 on oleg342-server Stopping /mnt/lustre-mds1 (opts:) on oleg342-server 21:48:58 (1745459338) shut down facet: mds1 facet_host: oleg342-server facet_failover_host: oleg342-server oleg342-client.virtnet: looking for dbench program oleg342-client.virtnet: /usr/bin/dbench oleg342-client.virtnet: creating output directory /mnt/lustre/d70b.replay-single/oleg342-client.virtnet oleg342-client.virtnet: mkdir: created directory '/mnt/lustre/d70b.replay-single' oleg342-client.virtnet: mkdir: created directory '/mnt/lustre/d70b.replay-single/oleg342-client.virtnet' oleg342-client.virtnet: found dbench client file /usr/share/dbench/client.txt oleg342-client.virtnet: '/usr/share/dbench/client.txt' -> 'client.txt' oleg342-client.virtnet: running 'dbench 1 -t 300' on /mnt/lustre/d70b.replay-single/oleg342-client.virtnet at Wed Apr 23 21:48:28 EDT 2025 oleg342-client.virtnet: waiting for dbench pid 5619 oleg342-client.virtnet: dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 oleg342-client.virtnet: oleg342-client.virtnet: Running for 300 seconds with load 'client.txt' and minimum warmup 60 secs oleg342-client.virtnet: failed to create barrier semaphore oleg342-client.virtnet: 0 of 1 processes prepared for launch 0 sec oleg342-client.virtnet: 1 of 1 processes prepared for launch 0 sec oleg342-client.virtnet: releasing clients oleg342-client.virtnet: 1 504 17.46 MB/sec warmup 1 sec latency 17.418 ms oleg342-client.virtnet: 1 981 10.95 MB/sec warmup 2 sec latency 47.718 ms oleg342-client.virtnet: 1 1667 8.74 MB/sec warmup 3 sec latency 21.422 ms oleg342-client.virtnet: 1 2376 6.99 MB/sec warmup 4 sec latency 21.548 ms oleg342-client.virtnet: 1 2982 6.70 MB/sec warmup 5 sec latency 273.142 ms oleg342-client.virtnet: 1 2982 5.59 MB/sec warmup 6 sec latency 1273.298 ms oleg342-client.virtnet: 1 2982 4.79 MB/sec warmup 7 sec latency 2273.469 ms oleg342-client.virtnet: 1 2982 4.19 MB/sec warmup 8 sec latency 3273.616 ms oleg342-client.virtnet: 1 2982 3.72 MB/sec warmup 9 sec latency 4273.770 ms oleg342-client.virtnet: 1 2982 3.35 MB/sec warmup 10 sec latency 5273.913 ms oleg342-client.virtnet: 1 2982 3.05 MB/sec warmup 11 sec latency 6274.090 ms oleg342-client.virtnet: 1 2982 2.79 MB/sec warmup 12 sec latency 7274.335 ms oleg342-client.virtnet: 1 2982 2.58 MB/sec warmup 13 sec latency 8274.678 ms oleg342-client.virtnet: 1 2982 2.39 MB/sec warmup 14 sec latency 9274.962 ms oleg342-client.virtnet: 1 2982 2.23 MB/sec warmup 15 sec latency 10275.184 ms oleg342-client.virtnet: 1 2982 2.09 MB/sec warmup 16 sec latency 11275.386 ms oleg342-client.virtnet: 1 2982 1.97 MB/sec warmup 17 sec latency 12275.582 ms oleg342-client.virtnet: 1 2982 1.86 MB/sec warmup 18 sec latency 13275.782 ms oleg342-client.virtnet: 1 2982 1.76 MB/sec warmup 19 sec latency 14275.959 ms oleg342-client.virtnet: 1 2982 1.68 MB/sec warmup 20 sec latency 15276.171 ms oleg342-client.virtnet: 1 2982 1.60 MB/sec warmup 21 sec latency 16276.394 ms oleg342-client.virtnet: 1 2982 1.52 MB/sec warmup 22 sec latency 17276.605 ms oleg342-client.virtnet: 1 2982 1.46 MB/sec warmup 23 sec latency 18276.937 ms oleg342-client.virtnet: 1 3624 1.61 MB/sec warmup 24 sec latency 18666.914 ms oleg342-client.virtnet: 1 4142 1.60 MB/sec warmup 25 sec latency 9.808 ms oleg342-client.virtnet: 1 4628 1.62 MB/sec warmup 26 sec latency 27.635 ms oleg342-client.virtnet: 1 5278 1.68 MB/sec warmup 27 sec latency 25.604 ms oleg342-client.virtnet: 1 5960 1.68 MB/sec warmup 28 sec latency 22.176 ms oleg342-client.virtnet: 1 7049 1.98 MB/sec warmup 29 sec latency 17.790 ms oleg342-client.virtnet: 1 7408 1.96 MB/sec warmup 30 sec latency 409.819 ms oleg342-client.virtnet: 1 7408 1.90 MB/sec warmup 31 sec latency 1409.980 ms oleg342-client.virtnet: 1 7408 1.84 MB/sec warmup 32 sec latency 2410.134Failover mds1 to oleg342-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg342-server: oleg342-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg342-client: oleg342-server: ssh exited with exit code 1 Started lustre-MDT0000 21:49:11 (1745459351) targets are mounted 21:49:11 (1745459351) facet_failover done oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3968 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 14336 3735552 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 40960 3715072 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 55296 7450624 1% /mnt/lustre test_70b fail mds1 3 times Failing mds1 on oleg342-server Stopping /mnt/lustre-mds1 (opts:) on oleg342-server 21:49:23 (1745459363) shut down facet: mds1 facet_host: oleg342-server facet_failover_host: oleg342-server Failover mds1 to oleg342-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg342-server: oleg342-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg342-client: oleg342-server: ssh exited with exit code 1 Started lustre-MDT0000 21:49:36 (1745459376) targets are mounted 21:49:36 (1745459376) facet_failover done oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3968 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 13312 3737600 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 40960 3693568 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 54272 7431168 1% /mnt/lustre ms oleg342-client.virtnet: 1 7408 1.78 MB/sec warmup 33 sec latency 3410.294 ms oleg342-client.virtnet: 1 7408 1.73 MB/sec warmup 34 sec latency 4410.434 ms oleg342-client.virtnet: 1 7408 1.68 MB/sec warmup 35 sec latency 5410.588 ms oleg342-client.virtnet: 1 7408 1.64 MB/sec warmup 36 sec latency 6410.733 ms oleg342-client.virtnet: 1 7408 1.59 MB/sec warmup 37 sec latency 7410.950 ms oleg342-client.virtnet: 1 7408 1.55 MB/sec warmup 38 sec latency 8411.141 ms oleg342-client.virtnet: 1 7408 1.51 MB/sec warmup 39 sec latency 9411.365 ms oleg342-client.virtnet: 1 7408 1.47 MB/sec warmup 40 sec latency 10411.590 ms oleg342-client.virtnet: 1 7408 1.44 MB/sec warmup 41 sec latency 11411.722 ms oleg342-client.virtnet: 1 7408 1.40 MB/sec warmup 42 sec latency 12411.912 ms oleg342-client.virtnet: 1 7408 1.37 MB/sec warmup 43 sec latency 13412.119 ms oleg342-client.virtnet: 1 7408 1.34 MB/sec warmup 44 sec latency 14412.344 ms oleg342-client.virtnet: 1 7408 1.31 MB/sec warmup 45 sec latency 15412.534 ms oleg342-client.virtnet: 1 7408 1.28 MB/sec warmup 46 sec latency 16412.748 ms oleg342-client.virtnet: 1 7408 1.25 MB/sec warmup 47 sec latency 17412.957 ms oleg342-client.virtnet: 1 7408 1.23 MB/sec warmup 48 sec latency 18413.200 ms oleg342-client.virtnet: 1 7659 1.21 MB/sec warmup 49 sec latency 18871.231 ms oleg342-client.virtnet: 1 8091 1.20 MB/sec warmup 50 sec latency 40.282 ms oleg342-client.virtnet: 1 8761 1.26 MB/sec warmup 51 sec latency 24.739 ms oleg342-client.virtnet: 1 9454 1.27 MB/sec warmup 52 sec latency 26.570 ms oleg342-client.virtnet: 1 10374 1.39 MB/sec warmup 53 sec latency 23.272 ms oleg342-client.virtnet: 1 11057 1.45 MB/sec warmup 54 sec latency 10.137 ms oleg342-client.virtnet: 1 11391 1.43 MB/sec warmup 55 sec latency 321.539 ms oleg342-client.virtnet: 1 11391 1.40 MB/sec warmup 56 sec latency 1321.679 ms oleg342-client.virtnet: 1 11391 1.38 MB/sec warmup 57 sec latency 2321.834 ms oleg342-client.virtnet: 1 11391 1.36 MB/sec warmup 58 sec latency 3321.997 ms oleg342-client.virtnet: 1 11391 1.33 MB/sec warmup 59 sec latency 4322.184 ms oleg342-client.virtnet: 1 11391 0.00 MB/sec execute 1 sec latency 6322.437 ms oleg342-client.virtnet: 1 11391 0.00 MB/sec execute 2 sec latency 7322.655 ms oleg342-client.virtnet: 1 11391 0.00 MB/sec execute 3 sec latency 8322.861 ms oleg342-client.virtnet: 1 11391 0.00 MB/sec execute 4 sec latency 9323.154 ms oleg342-client.virtnet: 1 11391 0.00 MB/sec execute 5 sec latency 10323.290 ms oleg342-client.virtnet: 1 11391 0.00 MB/sec execute 6 sec latency 11323.423 ms oleg342-client.virtnet: 1 11391 0.00 MB/sec execute 7 sec latency 12323.637 ms oleg342-client.virtnet: 1 11391 0.00 MB/sec execute 8 sec latency 13323.821 ms oleg342-client.virtnet: 1 11391 0.00 MB/sec execute 9 sec latency 14324.148 ms oleg342-client.virtnet: 1 11391 0.00 MB/sec execute 10 sec latency 15324.395 ms oleg342-client.virtnet: 1 11391 0.00 MB/sec execute 11 sec latency 16324.678 ms oleg342-client.virtnet: 1 11391 0.00 MB/sec execute 12 sec latency 17324.872 ms oleg342-client.virtnet: 1 11391 0.00 MB/sec execute 13 sec latency 18325.141 ms oleg342-client.virtnet: 1 11606 0.03 MB/sec execute 14 sec latency 18827.419 ms oleg342-client.virtnet: 1 12233 0.32 MB/sec execute 15 sec latency 34.822 ms oleg342-client.virtnet: 1 12830 0.34 MB/sec execute 16 sec latency 23.074 ms oleg342-client.virtnet: 1 13662 0.72 MB/sec execute 17 sec latency 22.929 ms oleg342-client.virtnet: test_70b fail mds1 4 times Failing mds1 on oleg342-server Stopping /mnt/lustre-mds1 (opts:) on oleg342-server 21:49:48 (1745459388) shut down facet: mds1 facet_host: oleg342-server facet_failover_host: oleg342-server Failover mds1 to oleg342-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg342-server: oleg342-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg342-client: oleg342-server: ssh exited with exit code 1 Started lustre-MDT0000 21:50:01 (1745459401) targets are mounted 21:50:01 (1745459401) facet_failover done oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3968 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 14336 3726336 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 40960 3709952 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 55296 7436288 1% /mnt/lustre test_70b fail mds1 5 times Failing mds1 on oleg342-server Stopping /mnt/lustre-mds1 (opts:) on oleg342-server 21:50:13 (1745459413) shut down facet: mds1 facet_host: oleg342-server facet_failover_host: oleg342-server Failover mds1 to oleg342-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg342-server: oleg342-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg342-client: oleg342-server: ssh exited with exit code 1 Started lustre-MDT0000 21:50:26 (1745459426) targets are mounted 21:50:26 (1745459426) facet_failover done 1 14529 1.03 MB/sec execute 18 sec latency 17.356 ms oleg342-client.virtnet: 1 15063 1.02 MB/sec execute 19 sec latency 29.912 ms oleg342-client.virtnet: 1 15427 1.04 MB/sec execute 20 sec latency 306.071 ms oleg342-client.virtnet: 1 15427 0.99 MB/sec execute 21 sec latency 1306.240 ms oleg342-client.virtnet: 1 15427 0.95 MB/sec execute 22 sec latency 2306.420 ms oleg342-client.virtnet: 1 15427 0.91 MB/sec execute 23 sec latency 3306.593 ms oleg342-client.virtnet: 1 15427 0.87 MB/sec execute 24 sec latency 4306.759 ms oleg342-client.virtnet: 1 15427 0.84 MB/sec execute 25 sec latency 5306.916 ms oleg342-client.virtnet: 1 15427 0.80 MB/sec execute 26 sec latency 6307.246 ms oleg342-client.virtnet: 1 15427 0.77 MB/sec execute 27 sec latency 7307.409 ms oleg342-client.virtnet: 1 15427 0.75 MB/sec execute 28 sec latency 8307.541 ms oleg342-client.virtnet: 1 15427 0.72 MB/sec execute 29 sec latency 9307.707 ms oleg342-client.virtnet: 1 15427 0.70 MB/sec execute 30 sec latency 10307.892 ms oleg342-client.virtnet: 1 15427 0.67 MB/sec execute 31 sec latency 11308.056 ms oleg342-client.virtnet: 1 15427 0.65 MB/sec execute 32 sec latency 12308.182 ms oleg342-client.virtnet: 1 15427 0.63 MB/sec execute 33 sec latency 13308.346 ms oleg342-client.virtnet: 1 15427 0.61 MB/sec execute 34 sec latency 14308.562 ms oleg342-client.virtnet: 1 15427 0.60 MB/sec execute 35 sec latency 15308.722 ms oleg342-client.virtnet: 1 15427 0.58 MB/sec execute 36 sec latency 16308.986 ms oleg342-client.virtnet: 1 15427 0.56 MB/sec execute 37 sec latency 17309.196 ms oleg342-client.virtnet: 1 15427 0.55 MB/sec execute 38 sec latency 18309.428 ms oleg342-client.virtnet: 1 15740 0.61 MB/sec execute 39 sec latency 18860.243 ms oleg342-client.virtnet: 1 16337 0.61 MB/sec execute 40 sec latency 25.870 ms oleg342-client.virtnet: 1 17174 0.76 MB/sec execute 41 sec latency 21.423 ms oleg342-client.virtnet: 1 18026 0.89 MB/sec execute 42 sec latency 17.397 ms oleg342-client.virtnet: 1 18514 0.89 MB/sec execute 43 sec latency 16.138 ms oleg342-client.virtnet: 1 19142 0.98 MB/sec execute 44 sec latency 19.832 ms oleg342-client.virtnet: 1 19564 0.96 MB/sec execute 45 sec latency 217.141 ms oleg342-client.virtnet: 1 19564 0.94 MB/sec execute 46 sec latency 1217.335 ms oleg342-client.virtnet: 1 19564 0.92 MB/sec execute 47 sec latency 2217.524 ms oleg342-client.virtnet: 1 19564 0.90 MB/sec execute 48 sec latency 3217.805 ms oleg342-client.virtnet: 1 19564 0.88 MB/sec execute 49 sec latency 4218.025 ms oleg342-client.virtnet: 1 19564 0.87 MB/sec execute 50 sec latency 5218.183 ms oleg342-client.virtnet: 1 19564 0.85 MB/sec execute 51 sec latency 6218.387 ms oleg342-client.virtnet: 1 19564 0.83 MB/sec execute 52 sec latency 7218.604 ms oleg342-client.virtnet: 1 19564 0.82 MB/sec execute 53 sec latency 8218.849 ms oleg342-client.virtnet: 1 19564 0.80 MB/sec execute 54 sec latency 9219.038 ms oleg342-client.virtnet: 1 19564 0.79 MB/sec execute 55 sec latency 10219.242 ms oleg342-client.virtnet: 1 19564 0.77 MB/sec execute 56 sec latency 11219.405 ms oleg342-client.virtnet: 1 19564 0.76 MB/sec execute 57 sec latency 12219.523 ms oleg342-client.virtnet: 1 19564 0.75 MB/sec execute 58 sec latency 13219.661 ms oleg342-client.virtnet: 1 19564 0.73 MB/sec execute 59 sec latency 14219.793 ms oleg342-client.virtnet: 1 19564 0.72 MB/sec execute 60 sec latency 15219.954 ms oleg342-client.virtnet: 1 19564 0.71 MB/sec execute 61 sec latency 16220.106 ms oleg342-client.virtnet: oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3968 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 14336 3736576 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 40960 3708928 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 55296 7445504 1% /mnt/lustre test_70b fail mds1 6 times Failing mds1 on oleg342-server Stopping /mnt/lustre-mds1 (opts:) on oleg342-server 21:50:38 (1745459438) shut down facet: mds1 facet_host: oleg342-server facet_failover_host: oleg342-server Failover mds1 to oleg342-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg342-server: oleg342-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg342-client: oleg342-server: ssh exited with exit code 1 Started lustre-MDT0000 21:50:51 (1745459451) targets are mounted 21:50:51 (1745459451) facet_failover done oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3968 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 14336 3728384 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 40960 3720192 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 55296 7448576 1% /mnt/lustre test_70b fail mds1 7 times Failing mds1 on oleg342-server Stopping /mnt/lustre-mds1 (opts:) on oleg342-server 21:51:03 (1745459463) shut down facet: mds1 facet_host: oleg342-server facet_failover_host: oleg342-server Failover mds1 to oleg342-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 19564 0.70 MB/sec execute 62 sec latency 17220.283 ms oleg342-client.virtnet: 1 19564 0.69 MB/sec execute 63 sec latency 18220.484 ms oleg342-client.virtnet: 1 19831 0.68 MB/sec execute 64 sec latency 18707.980 ms oleg342-client.virtnet: 1 20531 0.75 MB/sec execute 65 sec latency 22.880 ms oleg342-client.virtnet: 1 21455 0.84 MB/sec execute 66 sec latency 24.975 ms oleg342-client.virtnet: 1 21993 0.85 MB/sec execute 67 sec latency 9.540 ms oleg342-client.virtnet: 1 22387 0.87 MB/sec execute 68 sec latency 30.544 ms oleg342-client.virtnet: 1 23043 0.90 MB/sec execute 69 sec latency 23.519 ms oleg342-client.virtnet: 1 23516 0.90 MB/sec execute 70 sec latency 243.498 ms oleg342-client.virtnet: 1 23516 0.89 MB/sec execute 71 sec latency 1243.617 ms oleg342-client.virtnet: 1 23516 0.87 MB/sec execute 72 sec latency 2243.778 ms oleg342-client.virtnet: 1 23516 0.86 MB/sec execute 73 sec latency 3243.917 ms oleg342-client.virtnet: 1 23516 0.85 MB/sec execute 74 sec latency 4244.176 ms oleg342-client.virtnet: 1 23516 0.84 MB/sec execute 75 sec latency 5244.432 ms oleg342-client.virtnet: 1 23516 0.83 MB/sec execute 76 sec latency 6244.654 ms oleg342-client.virtnet: 1 23516 0.82 MB/sec execute 77 sec latency 7244.945 ms oleg342-client.virtnet: 1 23516 0.81 MB/sec execute 78 sec latency 8245.065 ms oleg342-client.virtnet: 1 23516 0.80 MB/sec execute 79 sec latency 9245.252 ms oleg342-client.virtnet: 1 23516 0.79 MB/sec execute 80 sec latency 10245.434 ms oleg342-client.virtnet: 1 23516 0.78 MB/sec execute 81 sec latency 11245.636 ms oleg342-client.virtnet: 1 23516 0.77 MB/sec execute 82 sec latency 12245.851 ms oleg342-client.virtnet: 1 23516 0.76 MB/sec execute 83 sec latency 13246.015 ms oleg342-client.virtnet: 1 23516 0.75 MB/sec execute 84 sec latency 14246.149 ms oleg342-client.virtnet: 1 23516 0.74 MB/sec execute 85 sec latency 15246.271 ms oleg342-client.virtnet: 1 23516 0.73 MB/sec execute 86 sec latency 16246.456 ms oleg342-client.virtnet: 1 23516 0.72 MB/sec execute 87 sec latency 17246.653 ms oleg342-client.virtnet: 1 23516 0.71 MB/sec execute 88 sec latency 18246.874 ms oleg342-client.virtnet: 1 23828 0.73 MB/sec execute 89 sec latency 18705.153 ms oleg342-client.virtnet: 1 24904 0.83 MB/sec execute 90 sec latency 21.771 ms oleg342-client.virtnet: 1 25438 0.84 MB/sec execute 91 sec latency 8.137 ms oleg342-client.virtnet: 1 25942 0.85 MB/sec execute 92 sec latency 26.285 ms oleg342-client.virtnet: 1 26598 0.88 MB/sec execute 93 sec latency 26.348 ms oleg342-client.virtnet: 1 27271 0.89 MB/sec execute 94 sec latency 21.012 ms oleg342-client.virtnet: 1 27808 0.93 MB/sec execute 95 sec latency 389.333 ms oleg342-client.virtnet: 1 27808 0.92 MB/sec execute 96 sec latency 1389.478 ms oleg342-client.virtnet: 1 27808 0.91 MB/sec execute 97 sec latency 2389.652 ms oleg342-client.virtnet: 1 27808 0.91 MB/sec execute 98 sec latency 3389.798 ms oleg342-client.virtnet: 1 27808 0.90 MB/sec execute 99 sec latency 4389.986 ms oleg342-client.virtnet: 1 27808 0.89 MB/sec execute 100 sec latency 5390.103 ms oleg342-client.virtnet: 1 27808 0.88 MB/sec execute 101 sec latency 6390.256 ms oleg342-client.virtnet: 1 27808 0.87 MB/sec execute 102 sec latency 7390.427 ms oleg342-client.virtnet: 1 27808 0.86 MB/sec execute 103 sec latency 8390.615 ms oleg342-client.virtnet: 1 27808 0.85 MB/sec execute 104 sec latency 9390.791 ms oleg342-client.virtnet: 1 27808 0.84 MB/sec execute 105 sec latency 10390.904 ms oleg342-client.virtnet: 1 27808 oleg342-server: oleg342-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg342-client: oleg342-server: ssh exited with exit code 1 Started lustre-MDT0000 21:51:15 (1745459475) targets are mounted 21:51:15 (1745459475) facet_failover done oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3968 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 14336 3736576 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 44032 3691520 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 58368 7428096 1% /mnt/lustre test_70b fail mds1 8 times Failing mds1 on oleg342-server Stopping /mnt/lustre-mds1 (opts:) on oleg342-server 21:51:28 (1745459488) shut down facet: mds1 facet_host: oleg342-server facet_failover_host: oleg342-server Failover mds1 to oleg342-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg342-server: oleg342-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg342-client: oleg342-server: ssh exited with exit code 1 Started lustre-MDT0000 21:51:41 (1745459501) targets are mounted 21:51:41 (1745459501) facet_failover done oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3968 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 14336 3734528 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 43008 3712000 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 57344 7446528 1% /mnt/lustre test_70b fail mds1 9 times Failing mds1 on oleg342-server Stopping /mnt/lustre-mds1 (opts:) on oleg342-server 21:51:53 (1745459513) shut down facet: mds1 facet_host: oleg342-server facet_failover_host: oleg342-server 0.84 MB/sec execute 106 sec latency 11391.085 ms oleg342-client.virtnet: 1 27808 0.83 MB/sec execute 107 sec latency 12391.209 ms oleg342-client.virtnet: 1 27808 0.82 MB/sec execute 108 sec latency 13391.379 ms oleg342-client.virtnet: 1 27808 0.81 MB/sec execute 109 sec latency 14391.550 ms oleg342-client.virtnet: 1 27808 0.81 MB/sec execute 110 sec latency 15391.775 ms oleg342-client.virtnet: 1 27808 0.80 MB/sec execute 111 sec latency 16391.887 ms oleg342-client.virtnet: 1 27808 0.79 MB/sec execute 112 sec latency 17392.046 ms oleg342-client.virtnet: 1 27808 0.79 MB/sec execute 113 sec latency 18392.208 ms oleg342-client.virtnet: 1 28268 0.80 MB/sec execute 114 sec latency 18900.462 ms oleg342-client.virtnet: 1 28858 0.83 MB/sec execute 115 sec latency 10.047 ms oleg342-client.virtnet: 1 29330 0.83 MB/sec execute 116 sec latency 30.732 ms oleg342-client.virtnet: 1 29997 0.86 MB/sec execute 117 sec latency 22.235 ms oleg342-client.virtnet: 1 30604 0.86 MB/sec execute 118 sec latency 24.864 ms oleg342-client.virtnet: 1 31427 0.91 MB/sec execute 119 sec latency 20.011 ms oleg342-client.virtnet: 1 32174 0.95 MB/sec execute 120 sec latency 224.314 ms oleg342-client.virtnet: 1 32174 0.94 MB/sec execute 121 sec latency 1224.467 ms oleg342-client.virtnet: 1 32174 0.94 MB/sec execute 122 sec latency 2224.604 ms oleg342-client.virtnet: 1 32174 0.93 MB/sec execute 123 sec latency 3224.729 ms oleg342-client.virtnet: 1 32174 0.92 MB/sec execute 124 sec latency 4224.904 ms oleg342-client.virtnet: 1 32174 0.91 MB/sec execute 125 sec latency 5225.039 ms oleg342-client.virtnet: 1 32174 0.91 MB/sec execute 126 sec latency 6225.203 ms oleg342-client.virtnet: 1 32174 0.90 MB/sec execute 127 sec latency 7225.411 ms oleg342-client.virtnet: 1 32174 0.89 MB/sec execute 128 sec latency 8225.579 ms oleg342-client.virtnet: 1 32174 0.88 MB/sec execute 129 sec latency 9225.727 ms oleg342-client.virtnet: 1 32174 0.88 MB/sec execute 130 sec latency 10225.949 ms oleg342-client.virtnet: 1 32174 0.87 MB/sec execute 131 sec latency 11226.110 ms oleg342-client.virtnet: 1 32174 0.86 MB/sec execute 132 sec latency 12226.315 ms oleg342-client.virtnet: 1 32174 0.86 MB/sec execute 133 sec latency 13226.554 ms oleg342-client.virtnet: 1 32174 0.85 MB/sec execute 134 sec latency 14226.819 ms oleg342-client.virtnet: 1 32174 0.85 MB/sec execute 135 sec latency 15227.060 ms oleg342-client.virtnet: 1 32174 0.84 MB/sec execute 136 sec latency 16227.303 ms oleg342-client.virtnet: 1 32174 0.83 MB/sec execute 137 sec latency 17227.626 ms oleg342-client.virtnet: 1 32174 0.83 MB/sec execute 138 sec latency 18227.874 ms oleg342-client.virtnet: 1 32341 0.82 MB/sec execute 139 sec latency 18860.938 ms oleg342-client.virtnet: 1 32774 0.82 MB/sec execute 140 sec latency 34.982 ms oleg342-client.virtnet: 1 33418 0.85 MB/sec execute 141 sec latency 18.650 ms oleg342-client.virtnet: 1 33976 0.84 MB/sec execute 142 sec latency 21.612 ms oleg342-client.virtnet: 1 34744 0.88 MB/sec execute 143 sec latency 19.148 ms oleg342-client.virtnet: 1 35725 0.93 MB/sec execute 144 sec latency 21.639 ms oleg342-client.virtnet: 1 36089 0.92 MB/sec execute 145 sec latency 246.983 ms oleg342-client.virtnet: 1 36089 0.91 MB/sec execute 146 sec latency 1247.174 ms oleg342-client.virtnet: 1 36089 0.91 MB/sec execute 147 sec latency 2247.335 ms oleg342-client.virtnet: 1 36089 0.90 MB/sec execute 148 sec latency 3247.506 ms oleg342-client.virtnet: 1 36089 0.90 MB/sec execute 149 sec latency 4247.604 ms oleg342-client.virtnet: 1 36089 0.89 Failover mds1 to oleg342-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg342-server: oleg342-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg342-client: oleg342-server: ssh exited with exit code 1 Started lustre-MDT0000 21:52:06 (1745459526) targets are mounted 21:52:06 (1745459526) facet_failover done oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3968 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 13312 3748864 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 40960 3715072 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 54272 7463936 1% /mnt/lustre test_70b fail mds1 10 times Failing mds1 on oleg342-server Stopping /mnt/lustre-mds1 (opts:) on oleg342-server 21:52:21 (1745459541) shut down facet: mds1 facet_host: oleg342-server facet_failover_host: oleg342-server Failover mds1 to oleg342-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg342-server: oleg342-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg342-client: oleg342-server: ssh exited with exit code 1 Started lustre-MDT0000 21:52:33 (1745459553) targets are mounted 21:52:33 (1745459553) facet_failover done oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3968 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 13312 3744768 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 40960 3709952 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 54272 7454720 1% /mnt/lustre MB/sec execute 150 sec latency 5247.759 ms oleg342-client.virtnet: 1 36089 0.88 MB/sec execute 151 sec latency 6247.969 ms oleg342-client.virtnet: 1 36089 0.88 MB/sec execute 152 sec latency 7248.141 ms oleg342-client.virtnet: 1 36089 0.87 MB/sec execute 153 sec latency 8248.258 ms oleg342-client.virtnet: 1 36089 0.87 MB/sec execute 154 sec latency 9248.371 ms oleg342-client.virtnet: 1 36089 0.86 MB/sec execute 155 sec latency 10248.536 ms oleg342-client.virtnet: 1 36089 0.86 MB/sec execute 156 sec latency 11248.666 ms oleg342-client.virtnet: 1 36089 0.85 MB/sec execute 157 sec latency 12248.791 ms oleg342-client.virtnet: 1 36089 0.85 MB/sec execute 158 sec latency 13249.038 ms oleg342-client.virtnet: 1 36089 0.84 MB/sec execute 159 sec latency 14249.201 ms oleg342-client.virtnet: 1 36089 0.83 MB/sec execute 160 sec latency 15249.405 ms oleg342-client.virtnet: 1 36089 0.83 MB/sec execute 161 sec latency 16249.565 ms oleg342-client.virtnet: 1 36089 0.82 MB/sec execute 162 sec latency 17249.755 ms oleg342-client.virtnet: 1 36089 0.82 MB/sec execute 163 sec latency 18250.030 ms oleg342-client.virtnet: 1 36096 0.81 MB/sec execute 164 sec latency 19215.431 ms oleg342-client.virtnet: 1 36488 0.81 MB/sec execute 165 sec latency 35.739 ms oleg342-client.virtnet: 1 37146 0.84 MB/sec execute 166 sec latency 21.321 ms oleg342-client.virtnet: 1 37839 0.84 MB/sec execute 167 sec latency 25.915 ms oleg342-client.virtnet: 1 38711 0.87 MB/sec execute 168 sec latency 22.696 ms oleg342-client.virtnet: 1 39411 0.90 MB/sec execute 169 sec latency 12.326 ms oleg342-client.virtnet: 1 39923 0.90 MB/sec execute 170 sec latency 30.515 ms oleg342-client.virtnet: 1 39987 0.90 MB/sec execute 171 sec latency 822.372 ms oleg342-client.virtnet: 1 39987 0.89 MB/sec execute 172 sec latency 1822.512 ms oleg342-client.virtnet: 1 39987 0.89 MB/sec execute 173 sec latency 2822.632 ms oleg342-client.virtnet: 1 39987 0.88 MB/sec execute 174 sec latency 3822.812 ms oleg342-client.virtnet: 1 39987 0.88 MB/sec execute 175 sec latency 4822.956 ms oleg342-client.virtnet: 1 39987 0.87 MB/sec execute 176 sec latency 5823.107 ms oleg342-client.virtnet: 1 39987 0.87 MB/sec execute 177 sec latency 6823.230 ms oleg342-client.virtnet: 1 39987 0.86 MB/sec execute 178 sec latency 7823.529 ms oleg342-client.virtnet: 1 39987 0.86 MB/sec execute 179 sec latency 8823.749 ms oleg342-client.virtnet: 1 39987 0.85 MB/sec execute 180 sec latency 9823.958 ms oleg342-client.virtnet: 1 39987 0.85 MB/sec execute 181 sec latency 10824.110 ms oleg342-client.virtnet: 1 39987 0.84 MB/sec execute 182 sec latency 11824.344 ms oleg342-client.virtnet: 1 39987 0.84 MB/sec execute 183 sec latency 12824.524 ms oleg342-client.virtnet: 1 39987 0.83 MB/sec execute 184 sec latency 13824.687 ms oleg342-client.virtnet: 1 39987 0.83 MB/sec execute 185 sec latency 14824.905 ms oleg342-client.virtnet: 1 39987 0.83 MB/sec execute 186 sec latency 15825.214 ms oleg342-client.virtnet: 1 39987 0.82 MB/sec execute 187 sec latency 16825.445 ms oleg342-client.virtnet: 1 39987 0.82 MB/sec execute 188 sec latency 17825.637 ms oleg342-client.virtnet: 1 40141 0.82 MB/sec execute 189 sec latency 18547.416 ms oleg342-client.virtnet: 1 40748 0.83 MB/sec execute 190 sec latency 31.290 ms oleg342-client.virtnet: 1 41404 0.84 MB/sec execute 191 sec latency 17.754 ms oleg342-client.virtnet: 1 42255 0.86 MB/sec execute 192 sec latency 24.182 ms oleg342-client.virtnet: 1 42958 0.89 MB/sec execute 193 sec latency 15.297 ms oleg342-client.virtnet: 1 43412 0.89 MB/sectest_70b fail mds1 11 times Failing mds1 on oleg342-server Stopping /mnt/lustre-mds1 (opts:) on oleg342-server 21:52:45 (1745459565) shut down facet: mds1 facet_host: oleg342-server facet_failover_host: oleg342-server Failover mds1 to oleg342-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg342-server: oleg342-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg342-client: oleg342-server: ssh exited with exit code 1 Started lustre-MDT0000 21:52:58 (1745459578) targets are mounted 21:52:58 (1745459578) facet_failover done oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3968 2204416 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 14336 3733504 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 40960 3706880 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 55296 7440384 1% /mnt/lustre test_70b fail mds1 12 times Failing mds1 on oleg342-server Stopping /mnt/lustre-mds1 (opts:) on oleg342-server 21:53:08 (1745459588) shut down facet: mds1 facet_host: oleg342-server facet_failover_host: oleg342-server Failover mds1 to oleg342-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg342-server: oleg342-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg342-client: oleg342-server: ssh exited with exit code 1 Started lustre-MDT0000 21:53:21 (1745459601) targets are mounted 21:53:21 (1745459601) facet_failover done execute 194 sec latency 40.249 ms oleg342-client.virtnet: 1 43900 0.89 MB/sec execute 195 sec latency 58.182 ms oleg342-client.virtnet: 1 43900 0.89 MB/sec execute 196 sec latency 1058.301 ms oleg342-client.virtnet: 1 43900 0.88 MB/sec execute 197 sec latency 2058.464 ms oleg342-client.virtnet: 1 43900 0.88 MB/sec execute 198 sec latency 3058.644 ms oleg342-client.virtnet: 1 43900 0.88 MB/sec execute 199 sec latency 4058.791 ms oleg342-client.virtnet: 1 43900 0.87 MB/sec execute 200 sec latency 5058.940 ms oleg342-client.virtnet: 1 43900 0.87 MB/sec execute 201 sec latency 6059.104 ms oleg342-client.virtnet: 1 43900 0.86 MB/sec execute 202 sec latency 7059.273 ms oleg342-client.virtnet: 1 43900 0.86 MB/sec execute 203 sec latency 8059.451 ms oleg342-client.virtnet: 1 43900 0.85 MB/sec execute 204 sec latency 9059.654 ms oleg342-client.virtnet: 1 43900 0.85 MB/sec execute 205 sec latency 10059.778 ms oleg342-client.virtnet: 1 43900 0.85 MB/sec execute 206 sec latency 11059.911 ms oleg342-client.virtnet: 1 43900 0.84 MB/sec execute 207 sec latency 12060.046 ms oleg342-client.virtnet: 1 43900 0.84 MB/sec execute 208 sec latency 13060.209 ms oleg342-client.virtnet: 1 43900 0.83 MB/sec execute 209 sec latency 14060.318 ms oleg342-client.virtnet: 1 43900 0.83 MB/sec execute 210 sec latency 15060.509 ms oleg342-client.virtnet: 1 43900 0.83 MB/sec execute 211 sec latency 16060.636 ms oleg342-client.virtnet: 1 43900 0.82 MB/sec execute 212 sec latency 17060.779 ms oleg342-client.virtnet: 1 43900 0.82 MB/sec execute 213 sec latency 18060.929 ms oleg342-client.virtnet: 1 44159 0.83 MB/sec execute 214 sec latency 18736.974 ms oleg342-client.virtnet: 1 44748 0.83 MB/sec execute 215 sec latency 24.574 ms oleg342-client.virtnet: 1 45553 0.85 MB/sec execute 216 sec latency 26.921 ms oleg342-client.virtnet: 1 46423 0.88 MB/sec execute 217 sec latency 19.195 ms oleg342-client.virtnet: 1 46924 0.88 MB/sec execute 218 sec latency 31.372 ms oleg342-client.virtnet: 1 47538 0.89 MB/sec execute 219 sec latency 22.540 ms oleg342-client.virtnet: 1 48030 0.89 MB/sec execute 220 sec latency 84.975 ms oleg342-client.virtnet: 1 48030 0.89 MB/sec execute 221 sec latency 1085.110 ms oleg342-client.virtnet: 1 48030 0.88 MB/sec execute 222 sec latency 2085.266 ms oleg342-client.virtnet: 1 48030 0.88 MB/sec execute 223 sec latency 3085.434 ms oleg342-client.virtnet: 1 48030 0.88 MB/sec execute 224 sec latency 4085.624 ms oleg342-client.virtnet: 1 48030 0.87 MB/sec execute 225 sec latency 5085.784 ms oleg342-client.virtnet: 1 48030 0.87 MB/sec execute 226 sec latency 6085.946 ms oleg342-client.virtnet: 1 48030 0.86 MB/sec execute 227 sec latency 7086.109 ms oleg342-client.virtnet: 1 48030 0.86 MB/sec execute 228 sec latency 8086.247 ms oleg342-client.virtnet: 1 48030 0.86 MB/sec execute 229 sec latency 9086.387 ms oleg342-client.virtnet: 1 48030 0.85 MB/sec execute 230 sec latency 10086.588 ms oleg342-client.virtnet: 1 48030 0.85 MB/sec execute 231 sec latency 11086.770 ms oleg342-client.virtnet: 1 48030 0.85 MB/sec execute 232 sec latency 12086.886 ms oleg342-client.virtnet: 1 48030 0.84 MB/sec execute 233 sec latency 13087.002 ms oleg342-client.virtnet: 1 48030 0.84 MB/sec execute 234 sec latency 14087.103 ms oleg342-client.virtnet: 1 48030 0.84 MB/sec execute 235 sec latency 15087.244 ms oleg342-client.virtnet: 1 48030 0.83 MB/sec execute 236 sec latency 16087.439 ms oleg342-client.virtnet: 1 48030 0.83 MB/sec execute 237 sec latency 17087.574 ms oleg342-client.virtnet: 1 48030 0.82 MB/sec oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec execute 238 sec latency 18087.769 ms oleg342-client.virtnet: 1 48252 0.82 MB/sec execute 239 sec latency 18748.654 ms oleg342-client.virtnet: 1 49074 0.85 MB/sec execute 240 sec latency 22.414 ms oleg342-client.virtnet: 1 49960 0.87 MB/sec execute 241 sec latency 20.768 ms oleg342-client.virtnet: 1 50478 0.87 MB/sec execute 242 sec latency 28.540 ms oleg342-client.virtnet: 1 51096 0.88 MB/sec execute 243 sec latency 23.252 ms oleg342-client.virtnet: 1 51681 0.88 MB/sec execute 244 sec latency 27.024 ms oleg342-client.virtnet: 1 52456 0.90 MB/sec execute 245 sec latency 21.343 ms oleg342-client.virtnet: 1 53438 0.93 MB/sec execute 246 sec latency 19.207 ms oleg342-client.virtnet: 1 53958 0.93 MB/sec execute 247 sec latency 33.737 ms oleg342-client.virtnet: 1 54451 0.93 MB/sec execute 248 sec latency 23.740 ms oleg342-client.virtnet: 1 55062 0.94 MB/sec execute 249 sec latency 21.504 ms oleg342-client.virtnet: 1 55679 0.95 MB/sec execute 250 sec latency 19.736 ms oleg342-client.virtnet: 1 56772 0.98 MB/sec execute 251 sec latency 20.486 ms oleg342-client.virtnet: 1 57311 0.99 MB/sec execute 252 sec latency 8.600 ms oleg342-client.virtnet: 1 57865 0.99 MB/sec execute 253 sec latency 28.955 ms oleg342-client.virtnet: 1 58515 1.00 MB/sec execute 254 sec latency 28.023 ms oleg342-client.virtnet: 1 59198 1.00 MB/sec execute 255 sec latency 19.305 ms oleg342-client.virtnet: 1 60288 1.04 MB/sec execute 256 sec latency 20.092 ms oleg342-client.virtnet: 1 60840 1.04 MB/sec execute 257 sec latency 7.665 ms oleg342-client.virtnet: 1 61378 1.04 MB/sec execute 258 sec latency 29.220 ms oleg342-client.virtnet: 1 62033 1.05 MB/sec execute 259 sec latency 26.544 ms oleg342-client.virtnet: 1 62739 1.06 MB/sec execute 260 sec latency 18.487 ms oleg342-client.virtnet: 1 63830 1.09 MB/sec execute 261 sec latency 17.025 ms oleg342-client.virtnet: 1 64379 1.09 MB/sec execute 262 sec latency 9.432 ms oleg342-client.virtnet: 1 64828 1.09 MB/sec execute 263 sec latency 36.908 ms oleg342-client.virtnet: 1 65128 1.09 MB/sec execute 264 sec latency 31.964 ms oleg342-client.virtnet: 1 65706 1.10 MB/sec execute 265 sec latency 27.522 ms oleg342-client.virtnet: 1 66378 1.11 MB/sec execute 266 sec latency 19.904 ms oleg342-client.virtnet: 1 67500 1.14 MB/sec execute 267 sec latency 17.760 ms oleg342-client.virtnet: 1 68087 1.14 MB/sec execute 268 sec latency 7.835 ms oleg342-client.virtnet: 1 68573 1.14 MB/sec execute 269 sec latency 29.424 ms oleg342-client.virtnet: 1 69211 1.15 MB/sec execute 270 sec latency 22.065 ms oleg342-client.virtnet: 1 69875 1.15 MB/sec execute 271 sec latency 20.489 ms oleg342-client.virtnet: 1 70971 1.19 MB/sec execute 272 sec latency 21.301 ms oleg342-client.virtnet: 1 71519 1.19 MB/sec execute 273 sec latency 9.277 ms oleg342-client.virtnet: 1 72030 1.19 MB/sec execute 274 sec latency 36.344 ms oleg342-client.virtnet: 1 72678 1.20 MB/sec execute 275 sec latency 27.024 ms oleg342-client.virtnet: 1 73351 1.20 MB/sec execute 276 sec latency 23.194 ms oleg342-client.virtnet: 1 74309 1.22 MB/sec execute 277 sec latency 19.686 ms oleg342-client.virtnet: 1 74955 1.24 MB/sec execute 278 sec latency 9.045 ms oleg342-client.virtnet: 1 75456 1.24 MB/sec execute 279 sec latency 33.004 ms oleg342-client.virtnet: 1 76119 1.25 MB/sec execute 280 sec latency 27.212 ms oleg342-client.virtnet: 1 76751 1.24 MB/sec execute 281 sec latency 21.426 ms oleg342-client.virtnet: 1 77561 1.26 MB/sec execute 282 sec latency 26.303 ms oleg342-client.virtnet: 1 78427 1.28 MB/sec execute 283 sec latency 15.843 ms oleg342-client.virtnet: 1 78904 1.28 MB/sec execute 284 sec latency 27.653 ms oleg342-client.virtnet: 1 79498 1.29 MB/sec execute 285 sec latency 19.207 ms oleg342-client.virtnet: 1 79938 1.29 MB/sec execute 286 sec latency 40.967 ms oleg342-client.virtnet: 1 80477 1.29 MB/sec execute 287 sec latency 40.053 ms oleg342-client.virtnet: 1 81254 1.31 MB/sec execute 288 sec latency 31.325 ms oleg342-client.virtnet: 1 81879 1.32 MB/sec execute 289 sec latency 19.827 ms oleg342-client.virtnet: 1 82355 1.32 MB/sec execute 290 sec latency 27.408 ms oleg342-client.virtnet: 1 82877 1.32 MB/sec execute 291 sec latency 20.135 ms oleg342-client.virtnet: 1 83514 1.33 MB/sec execute 292 sec latency 20.687 ms oleg342-client.virtnet: 1 84026 1.33 MB/sec execute 293 sec latency 34.271 ms oleg342-client.virtnet: 1 84910 1.35 MB/sec execute 294 sec latency 23.612 ms oleg342-client.virtnet: 1 85606 1.36 MB/sec execute 295 sec latency 9.337 ms oleg342-client.virtnet: 1 86043 1.36 MB/sec execute 296 sec latency 36.832 ms oleg342-client.virtnet: 1 86567 1.37 MB/sec execute 297 sec latency 18.141 ms oleg342-client.virtnet: 1 86975 1.36 MB/sec execute 298 sec latency 31.927 ms oleg342-client.virtnet: 1 87554 1.37 MB/sec execute 299 sec latency 20.897 ms oleg342-client.virtnet: 1 cleanup 300 sec oleg342-client.virtnet: 0 cleanup 300 sec oleg342-client.virtnet: oleg342-client.virtnet: Operation Count AvgLat MaxLat oleg342-client.virtnet: ---------------------------------------- oleg342-client.virtnet: NTCreateX 13274 12.958 18900.443 oleg342-client.virtnet: Close 9788 2.799 18705.119 oleg342-client.virtnet: Rename 566 7.248 19.103 oleg342-client.virtnet: Unlink 2648 8.926 19215.408 oleg342-client.virtnet: Qpathinfo 12052 2.862 18860.235 oleg342-client.virtnet: Qfileinfo 2116 0.129 2.071 oleg342-client.virtnet: Qfsinfo 2188 0.089 2.922 oleg342-client.virtnet: Sfileinfo 1096 21.296 18827.403 oleg342-client.virtnet: Find 4639 0.296 12.018 oleg342-client.virtnet: WriteX 6580 0.656 5.529 oleg342-client.virtnet: ReadX 20833 0.018 3.237 oleg342-client.virtnet: LockX 44 0.857 1.264 oleg342-client.virtnet: UnlockX 44 0.900 1.264 oleg342-client.virtnet: Flush 930 14.355 45.347 oleg342-client.virtnet: oleg342-client.virtnet: Throughput 1.36536 MB/sec 1 clients 1 procs max_latency=19215.431 ms oleg342-client.virtnet: stopping dbench on /mnt/lustre/d70b.replay-single/oleg342-client.virtnet at Wed Apr 23 21:54:28 EDT 2025 with return code 0 oleg342-client.virtnet: clean dbench files on /mnt/lustre/d70b.replay-single/oleg342-client.virtnet oleg342-client.virtnet: /mnt/lustre/d70b.replay-single/oleg342-client.virtnet /mnt/lustre/d70b.replay-single/oleg342-client.virtnet oleg342-client.virtnet: removed directory: 'clients/client0/~dmtmp/PWRPNT' oleg342-client.virtnet: removed directory: 'clients/client0/~dmtmp/EXCEL' oleg342-client.virtnet: removed directory: 'clients/client0/~dmtmp/PARADOX' oleg342-client.virtnet: removed directory: 'clients/client0/~dmtmp/PM' oleg342-client.virtnet: removed directory: 'clients/client0/~dmtmp/SEED' oleg342-client.virtnet: removed directory: 'clients/client0/~dmtmp/ACCESS' oleg342-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORD' oleg342-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORDPRO' oleg342-client.virtnet: removed directory: 'clients/client0/~dmtmp/COREL' oleg342-client.virtnet: removed directory: 'clients/client0/~dmtmp' oleg342-client.virtnet: removed directory: 'clients/client0' oleg342-client.virtnet: removed directory: 'clients' oleg342-client.virtnet: removed 'client.txt' oleg342-client.virtnet: /mnt/lustre/d70b.replay-single/oleg342-client.virtnet oleg342-client.virtnet: dbench successfully finished