== replay-single test 70b: dbench 2mdts recovery; 1 clients ========================================================== 00:00:50 (1735102850) Starting client oleg229-client.virtnet: -o user_xattr,flock 192.168.202.129@tcp:/lustre /mnt/lustre Started clients oleg229-client.virtnet: 192.168.202.129@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) striped dir -i0 -c2 -H crush2 /mnt/lustre/d70b.replay-single + MISSING_DBENCH_OK= + PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin::/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests:/sbin:/usr/sbin:/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests/: + DBENCH_LIB= + TESTSUITE=replay-single + TESTNAME=test_70b + MOUNT=/mnt/lustre ++ hostname + DIR=/mnt/lustre/d70b.replay-single/oleg229-client.virtnet + LCTL=/home/green/git/lustre-release/lustre/utils/lctl + rundbench 1 -t 300 dbench: no process found Started rundbench load pid=13500 ... UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2528 1285160 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2116 1285572 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 26156 3556144 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1964 3594020 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 28120 7150164 1% /mnt/lustre test_70b fail mds1 1 times Failing mds1 on oleg229-server Stopping /mnt/lustre-mds1 (opts:) on oleg229-server 00:00:58 (1735102858) shut down facet: mds1 facet_host: oleg229-server facet_failover_host: oleg229-server Failover mds1 to oleg229-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg229-server: oleg229-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Started lustre-MDT0000 00:01:11 (1735102871) targets are mounted 00:01:11 (1735102871) facet_failover done oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2480 1285208 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2076 1285612 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 26412 3568672 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 2092 3593892 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 28504 7162564 1% /mnt/lustre test_70b fail mds2 2 times Failing mds2 on oleg229-server Stopping /mnt/lustre-mds2 (opts:) on oleg229-server 00:01:24 (1735102884) shut down facet: mds2 facet_host: oleg229-server facet_failover_host: oleg229-server oleg229-client.virtnet: looking for dbench program oleg229-client.virtnet: /usr/bin/dbench oleg229-client.virtnet: creating output directory /mnt/lustre/d70b.replay-single/oleg229-client.virtnet oleg229-client.virtnet: mkdir: created directory '/mnt/lustre/d70b.replay-single/oleg229-client.virtnet' oleg229-client.virtnet: found dbench client file /usr/share/dbench/client.txt oleg229-client.virtnet: '/usr/share/dbench/client.txt' -> 'client.txt' oleg229-client.virtnet: running 'dbench 1 -t 300' on /mnt/lustre/d70b.replay-single/oleg229-client.virtnet at Wed Dec 25 00:00:51 EST 2024 oleg229-client.virtnet: waiting for dbench pid 13525 oleg229-client.virtnet: dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 oleg229-client.virtnet: oleg229-client.virtnet: Running for 300 seconds with load 'client.txt' and minimum warmup 60 secs oleg229-client.virtnet: failed to create barrier semaphore oleg229-client.virtnet: 0 of 1 processes prepared for launch 0 sec oleg229-client.virtnet: 1 of 1 processes prepared for launch 0 sec oleg229-client.virtnet: releasing clients oleg229-client.virtnet: 1 510 17.73 MB/sec warmup 1 sec latency 13.821 ms oleg229-client.virtnet: 1 1143 11.50 MB/sec warmup 2 sec latency 11.489 ms oleg229-client.virtnet: 1 1799 8.78 MB/sec warmup 3 sec latency 10.067 ms oleg229-client.virtnet: 1 2200 6.69 MB/sec warmup 4 sec latency 391.956 ms oleg229-client.virtnet: 1 3277 7.07 MB/sec warmup 5 sec latency 11.419 ms oleg229-client.virtnet: 1 3829 6.63 MB/sec warmup 6 sec latency 356.815 ms oleg229-client.virtnet: 1 3829 5.68 MB/sec warmup 7 sec latency 1356.957 ms oleg229-client.virtnet: 1 3829 4.97 MB/sec warmup 8 sec latency 2357.141 ms oleg229-client.virtnet: 1 3829 4.42 MB/sec warmup 9 sec latency 3357.311 ms oleg229-client.virtnet: 1 3829 3.98 MB/sec warmup 10 sec latency 4357.505 ms oleg229-client.virtnet: 1 3829 3.61 MB/sec warmup 11 sec latency 5357.679 ms oleg229-client.virtnet: 1 3829 3.31 MB/sec warmup 12 sec latency 6357.843 ms oleg229-client.virtnet: 1 3829 3.06 MB/sec warmup 13 sec latency 7358.022 ms oleg229-client.virtnet: 1 3829 2.84 MB/sec warmup 14 sec latency 8358.303 ms oleg229-client.virtnet: 1 3829 2.65 MB/sec warmup 15 sec latency 9358.464 ms oleg229-client.virtnet: 1 3829 2.48 MB/sec warmup 16 sec latency 10358.633 ms oleg229-client.virtnet: 1 3829 2.34 MB/sec warmup 17 sec latency 11358.745 ms oleg229-client.virtnet: 1 3829 2.21 MB/sec warmup 18 sec latency 12358.908 ms oleg229-client.virtnet: 1 3829 2.09 MB/sec warmup 19 sec latency 13359.095 ms oleg229-client.virtnet: 1 3829 1.99 MB/sec warmup 20 sec latency 14359.275 ms oleg229-client.virtnet: 1 3829 1.89 MB/sec warmup 21 sec latency 15359.462 ms oleg229-client.virtnet: 1 3829 1.81 MB/sec warmup 22 sec latency 16359.674 ms oleg229-client.virtnet: 1 3829 1.73 MB/sec warmup 23 sec latency 17359.819 ms oleg229-client.virtnet: 1 3861 1.66 MB/sec warmup 24 sec latency 18263.765 ms oleg229-client.virtnet: 1 4311 1.62 MB/sec warmup 25 sec latency 11.367 ms oleg229-client.virtnet: 1 4787 1.62 MB/sec warmup 26 sec latency 13.337 ms oleg229-client.virtnet: 1 5218 1.68 MB/sec warmup 27 sec latency 18.757 ms oleg229-client.virtnet: 1 5583 1.63 MB/sec warmup 28 sec latency 14.457 ms oleg229-client.virtnet: 1 6030 1.65 MB/sec warmup 29 sec latency 16.592 ms oleg229-client.virtnet: 1 6916 1.82 MB/sec warmup 30 sec latency 15.627 ms oleg229-client.virtnet: 1 7428 1.90 MB/sec warmup 31 sec latency 13.809 ms oleg229-client.virtnet: 1 7807 1.86 MB/sec warmup 32 sec latency 18.676 ms oleg229-client.virtnet: 1 8174 1.85 MB/sec warmup 33 sec latency 16.Failover mds2 to oleg229-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg229-server: oleg229-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Started lustre-MDT0001 00:01:38 (1735102898) targets are mounted 00:01:38 (1735102898) facet_failover done oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2472 1285216 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37432 3568812 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12072 3593732 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49504 7162544 1% /mnt/lustre test_70b fail mds1 3 times Failing mds1 on oleg229-server Stopping /mnt/lustre-mds1 (opts:) on oleg229-server 00:01:50 (1735102910) shut down facet: mds1 facet_host: oleg229-server facet_failover_host: oleg229-server Failover mds1 to oleg229-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg229-server: oleg229-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Started lustre-MDT0000 00:02:04 (1735102924) targets are mounted 00:02:04 (1735102924) facet_failover done oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 692 ms oleg229-client.virtnet: 1 8617 1.89 MB/sec warmup 34 sec latency 17.220 ms oleg229-client.virtnet: 1 8951 1.85 MB/sec warmup 35 sec latency 23.986 ms oleg229-client.virtnet: 1 9493 1.84 MB/sec warmup 36 sec latency 12.717 ms oleg229-client.virtnet: 1 10275 1.96 MB/sec warmup 37 sec latency 18.150 ms oleg229-client.virtnet: 1 10891 2.05 MB/sec warmup 38 sec latency 11.853 ms oleg229-client.virtnet: 1 11306 2.01 MB/sec warmup 39 sec latency 11.998 ms oleg229-client.virtnet: 1 11739 2.01 MB/sec warmup 40 sec latency 16.843 ms oleg229-client.virtnet: 1 12209 2.04 MB/sec warmup 41 sec latency 14.200 ms oleg229-client.virtnet: 1 12517 1.99 MB/sec warmup 42 sec latency 16.873 ms oleg229-client.virtnet: 1 13002 1.98 MB/sec warmup 43 sec latency 13.753 ms oleg229-client.virtnet: 1 13564 2.06 MB/sec warmup 44 sec latency 19.208 ms oleg229-client.virtnet: 1 14273 2.13 MB/sec warmup 45 sec latency 16.536 ms oleg229-client.virtnet: 1 14578 2.12 MB/sec warmup 46 sec latency 16.407 ms oleg229-client.virtnet: 1 14752 2.07 MB/sec warmup 47 sec latency 323.362 ms oleg229-client.virtnet: 1 14752 2.03 MB/sec warmup 48 sec latency 1323.487 ms oleg229-client.virtnet: 1 14752 1.99 MB/sec warmup 49 sec latency 2323.663 ms oleg229-client.virtnet: 1 14752 1.95 MB/sec warmup 50 sec latency 3323.868 ms oleg229-client.virtnet: 1 15090 1.92 MB/sec warmup 51 sec latency 3631.308 ms oleg229-client.virtnet: 1 15612 1.97 MB/sec warmup 52 sec latency 14.894 ms oleg229-client.virtnet: 1 16089 1.94 MB/sec warmup 53 sec latency 11.910 ms oleg229-client.virtnet: 1 16682 1.95 MB/sec warmup 54 sec latency 11.802 ms oleg229-client.virtnet: 1 17695 2.09 MB/sec warmup 55 sec latency 13.312 ms oleg229-client.virtnet: 1 18158 2.08 MB/sec warmup 56 sec latency 10.909 ms oleg229-client.virtnet: 1 18559 2.05 MB/sec warmup 57 sec latency 16.508 ms oleg229-client.virtnet: 1 18911 2.04 MB/sec warmup 58 sec latency 147.517 ms oleg229-client.virtnet: 1 18911 2.01 MB/sec warmup 59 sec latency 1147.682 ms oleg229-client.virtnet: 1 18911 0.00 MB/sec execute 1 sec latency 3147.950 ms oleg229-client.virtnet: 1 18911 0.00 MB/sec execute 2 sec latency 4148.100 ms oleg229-client.virtnet: 1 18911 0.00 MB/sec execute 3 sec latency 5148.271 ms oleg229-client.virtnet: 1 18911 0.00 MB/sec execute 4 sec latency 6148.403 ms oleg229-client.virtnet: 1 18911 0.00 MB/sec execute 5 sec latency 7148.552 ms oleg229-client.virtnet: 1 18911 0.00 MB/sec execute 6 sec latency 8148.706 ms oleg229-client.virtnet: 1 18911 0.00 MB/sec execute 7 sec latency 9148.865 ms oleg229-client.virtnet: 1 18911 0.00 MB/sec execute 8 sec latency 10149.055 ms oleg229-client.virtnet: 1 18911 0.00 MB/sec execute 9 sec latency 11149.298 ms oleg229-client.virtnet: 1 18911 0.00 MB/sec execute 10 sec latency 12149.439 ms oleg229-client.virtnet: 1 18911 0.00 MB/sec execute 11 sec latency 13149.591 ms oleg229-client.virtnet: 1 18911 0.00 MB/sec execute 12 sec latency 14149.729 ms oleg229-client.virtnet: 1 18911 0.00 MB/sec execute 13 sec latency 15149.905 ms oleg229-client.virtnet: 1 18911 0.00 MB/sec execute 14 sec latency 16150.201 ms oleg229-client.virtnet: 1 18911 0.00 MB/sec execute 15 sec latency 17150.596 ms oleg229-client.virtnet: 1 18911 0.00 MB/sec execute 16 sec latency 18150.756 ms oleg229-client.virtnet: 1 19171 0.18 MB/sec execute 17 sec latency 18736.981 ms oleg229-client.virtnet: 1 19624 0.19 MB/sec execute 18 sec latency 11.024 ms oleg229-client.virtnet: 1 20201 0.27 MB/sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2440 1285248 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37424 3569092 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12088 3593804 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49512 7162896 1% /mnt/lustre test_70b fail mds2 4 times Failing mds2 on oleg229-server Stopping /mnt/lustre-mds2 (opts:) on oleg229-server 00:02:16 (1735102936) shut down facet: mds2 facet_host: oleg229-server facet_failover_host: oleg229-server Failover mds2 to oleg229-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg229-server: oleg229-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Started lustre-MDT0001 00:02:31 (1735102951) targets are mounted 00:02:31 (1735102951) facet_failover done oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2472 1285216 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37512 3568672 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12076 3591856 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49588 7160528 1% /mnt/lustre test_70b fail mds1 5 times Failing mds1 on oleg229-server Stopping /mnt/lustre-mds1 (opts:) on oleg229-server 00:02:42 (1735102962) shut down facet: mds1 facet_host: oleg229-server facet_failover_host: oleg229-server Failover mds1 to oleg229-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 execute 19 sec latency 10.668 ms oleg229-client.virtnet: 1 20957 0.56 MB/sec execute 20 sec latency 15.253 ms oleg229-client.virtnet: 1 21542 0.80 MB/sec execute 21 sec latency 12.204 ms oleg229-client.virtnet: 1 21845 0.78 MB/sec execute 22 sec latency 14.049 ms oleg229-client.virtnet: 1 22227 0.78 MB/sec execute 23 sec latency 16.085 ms oleg229-client.virtnet: 1 22722 0.92 MB/sec execute 24 sec latency 14.586 ms oleg229-client.virtnet: 1 23061 0.90 MB/sec execute 25 sec latency 16.740 ms oleg229-client.virtnet: 1 23425 0.88 MB/sec execute 26 sec latency 14.198 ms oleg229-client.virtnet: 1 23822 0.92 MB/sec execute 27 sec latency 19.120 ms oleg229-client.virtnet: 1 24862 1.23 MB/sec execute 28 sec latency 15.343 ms oleg229-client.virtnet: 1 25254 1.25 MB/sec execute 29 sec latency 11.908 ms oleg229-client.virtnet: 1 25694 1.23 MB/sec execute 30 sec latency 14.629 ms oleg229-client.virtnet: 1 26363 1.33 MB/sec execute 31 sec latency 9.673 ms oleg229-client.virtnet: 1 26905 1.31 MB/sec execute 32 sec latency 10.493 ms oleg229-client.virtnet: 1 27428 1.34 MB/sec execute 33 sec latency 13.281 ms oleg229-client.virtnet: 1 28369 1.58 MB/sec execute 34 sec latency 11.745 ms oleg229-client.virtnet: 1 28822 1.58 MB/sec execute 35 sec latency 10.484 ms oleg229-client.virtnet: 1 29295 1.56 MB/sec execute 36 sec latency 12.238 ms oleg229-client.virtnet: 1 29804 1.63 MB/sec execute 37 sec latency 15.625 ms oleg229-client.virtnet: 1 30251 1.60 MB/sec execute 38 sec latency 12.079 ms oleg229-client.virtnet: 1 30848 1.60 MB/sec execute 39 sec latency 62.154 ms oleg229-client.virtnet: 1 30848 1.56 MB/sec execute 40 sec latency 1062.348 ms oleg229-client.virtnet: 1 30848 1.52 MB/sec execute 41 sec latency 2062.537 ms oleg229-client.virtnet: 1 30848 1.49 MB/sec execute 42 sec latency 3062.743 ms oleg229-client.virtnet: 1 30908 1.47 MB/sec execute 43 sec latency 3920.798 ms oleg229-client.virtnet: 1 31968 1.66 MB/sec execute 44 sec latency 13.991 ms oleg229-client.virtnet: 1 32358 1.65 MB/sec execute 45 sec latency 15.694 ms oleg229-client.virtnet: 1 32823 1.63 MB/sec execute 46 sec latency 14.305 ms oleg229-client.virtnet: 1 33399 1.69 MB/sec execute 47 sec latency 11.778 ms oleg229-client.virtnet: 1 33854 1.66 MB/sec execute 48 sec latency 24.811 ms oleg229-client.virtnet: 1 34382 1.67 MB/sec execute 49 sec latency 138.788 ms oleg229-client.virtnet: 1 35327 1.78 MB/sec execute 50 sec latency 11.377 ms oleg229-client.virtnet: 1 35581 1.81 MB/sec execute 51 sec latency 702.996 ms oleg229-client.virtnet: 1 35581 1.77 MB/sec execute 52 sec latency 1703.140 ms oleg229-client.virtnet: 1 35581 1.74 MB/sec execute 53 sec latency 2703.303 ms oleg229-client.virtnet: 1 35581 1.71 MB/sec execute 54 sec latency 3703.469 ms oleg229-client.virtnet: 1 35581 1.67 MB/sec execute 55 sec latency 4703.659 ms oleg229-client.virtnet: 1 35581 1.64 MB/sec execute 56 sec latency 5703.839 ms oleg229-client.virtnet: 1 35581 1.62 MB/sec execute 57 sec latency 6703.998 ms oleg229-client.virtnet: 1 35581 1.59 MB/sec execute 58 sec latency 7704.153 ms oleg229-client.virtnet: 1 35581 1.56 MB/sec execute 59 sec latency 8704.319 ms oleg229-client.virtnet: 1 35581 1.54 MB/sec execute 60 sec latency 9704.475 ms oleg229-client.virtnet: 1 35581 1.51 MB/sec execute 61 sec latency 10704.612 ms oleg229-client.virtnet: 1 35581 1.49 MB/sec execute 62 sec latency 11704.857 ms oleg229-client.virtnet: 1 35581 1.46 MB/sec execute 63 sec latency 12705.013 ms oleg229-client.virtnoleg229-server: oleg229-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Started lustre-MDT0000 00:02:57 (1735102977) targets are mounted 00:02:57 (1735102977) facet_failover done oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2440 1285248 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37572 3568972 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12224 3593896 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49796 7162868 1% /mnt/lustre test_70b fail mds2 6 times Failing mds2 on oleg229-server Stopping /mnt/lustre-mds2 (opts:) on oleg229-server 00:03:09 (1735102989) shut down facet: mds2 facet_host: oleg229-server facet_failover_host: oleg229-server Failover mds2 to oleg229-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg229-server: oleg229-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Started lustre-MDT0001 00:03:23 (1735103003) targets are mounted 00:03:23 (1735103003) facet_failover done oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2472 1285216 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37564 3568620 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12076 3591892 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49640 7160512 1% /mnt/lustre test_70b fail mds1 7 times Failing mds1 on oleg229-server Stopping /mnt/lustre-mds1 (opts:) on oleg229-server et: 1 35581 1.44 MB/sec execute 64 sec latency 13705.197 ms oleg229-client.virtnet: 1 35581 1.42 MB/sec execute 65 sec latency 14705.348 ms oleg229-client.virtnet: 1 35581 1.40 MB/sec execute 66 sec latency 15705.521 ms oleg229-client.virtnet: 1 35581 1.37 MB/sec execute 67 sec latency 16705.686 ms oleg229-client.virtnet: 1 35581 1.35 MB/sec execute 68 sec latency 17705.887 ms oleg229-client.virtnet: 1 35582 1.33 MB/sec execute 69 sec latency 18693.025 ms oleg229-client.virtnet: 1 35911 1.34 MB/sec execute 70 sec latency 26.090 ms oleg229-client.virtnet: 1 36325 1.33 MB/sec execute 71 sec latency 14.177 ms oleg229-client.virtnet: 1 36912 1.37 MB/sec execute 72 sec latency 12.557 ms oleg229-client.virtnet: 1 37398 1.36 MB/sec execute 73 sec latency 13.076 ms oleg229-client.virtnet: 1 37965 1.36 MB/sec execute 74 sec latency 10.205 ms oleg229-client.virtnet: 1 39012 1.48 MB/sec execute 75 sec latency 11.139 ms oleg229-client.virtnet: 1 39485 1.48 MB/sec execute 76 sec latency 12.564 ms oleg229-client.virtnet: 1 40025 1.47 MB/sec execute 77 sec latency 20.553 ms oleg229-client.virtnet: 1 40681 1.51 MB/sec execute 78 sec latency 9.604 ms oleg229-client.virtnet: 1 41217 1.50 MB/sec execute 79 sec latency 10.839 ms oleg229-client.virtnet: 1 41934 1.55 MB/sec execute 80 sec latency 12.145 ms oleg229-client.virtnet: 1 42739 1.61 MB/sec execute 81 sec latency 13.889 ms oleg229-client.virtnet: 1 43191 1.61 MB/sec execute 82 sec latency 11.978 ms oleg229-client.virtnet: 1 43697 1.61 MB/sec execute 83 sec latency 12.911 ms oleg229-client.virtnet: 1 44191 1.63 MB/sec execute 84 sec latency 13.161 ms oleg229-client.virtnet: 1 44576 1.62 MB/sec execute 85 sec latency 16.946 ms oleg229-client.virtnet: 1 45025 1.62 MB/sec execute 86 sec latency 14.342 ms oleg229-client.virtnet: 1 45868 1.68 MB/sec execute 87 sec latency 15.997 ms oleg229-client.virtnet: 1 46462 1.71 MB/sec execute 88 sec latency 13.299 ms oleg229-client.virtnet: 1 46869 1.70 MB/sec execute 89 sec latency 15.750 ms oleg229-client.virtnet: 1 47233 1.70 MB/sec execute 90 sec latency 16.350 ms oleg229-client.virtnet: 1 47738 1.72 MB/sec execute 91 sec latency 11.486 ms oleg229-client.virtnet: 1 48002 1.70 MB/sec execute 92 sec latency 197.503 ms oleg229-client.virtnet: 1 48002 1.68 MB/sec execute 93 sec latency 1197.896 ms oleg229-client.virtnet: 1 48002 1.66 MB/sec execute 94 sec latency 2198.068 ms oleg229-client.virtnet: 1 48095 1.65 MB/sec execute 95 sec latency 3064.879 ms oleg229-client.virtnet: 1 48554 1.65 MB/sec execute 96 sec latency 15.432 ms oleg229-client.virtnet: 1 49332 1.69 MB/sec execute 97 sec latency 15.262 ms oleg229-client.virtnet: 1 49951 1.73 MB/sec execute 98 sec latency 12.405 ms oleg229-client.virtnet: 1 50343 1.72 MB/sec execute 99 sec latency 15.749 ms oleg229-client.virtnet: 1 50678 1.71 MB/sec execute 100 sec latency 195.655 ms oleg229-client.virtnet: 1 51185 1.73 MB/sec execute 101 sec latency 425.876 ms oleg229-client.virtnet: 1 51647 1.72 MB/sec execute 102 sec latency 13.068 ms oleg229-client.virtnet: 1 51856 1.71 MB/sec execute 103 sec latency 612.386 ms oleg229-client.virtnet: 1 51856 1.69 MB/sec execute 104 sec latency 1612.795 ms oleg229-client.virtnet: 1 51856 1.67 MB/sec execute 105 sec latency 2612.948 ms oleg229-client.virtnet: 1 51856 1.66 MB/sec execute 106 sec latency 3613.113 ms oleg229-client.virtnet: 1 51856 1.64 MB/sec execute 107 sec latency 4613.486 ms oleg229-client.virtnet: 1 51856 1.63 MB/sec execute 108 sec late00:03:40 (1735103020) shut down facet: mds1 facet_host: oleg229-server facet_failover_host: oleg229-server Failover mds1 to oleg229-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg229-server: oleg229-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Started lustre-MDT0000 00:03:54 (1735103034) targets are mounted 00:03:54 (1735103034) facet_failover done oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2440 1285248 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37540 3568864 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12076 3592032 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49616 7160896 1% /mnt/lustre test_70b fail mds2 8 times Failing mds2 on oleg229-server Stopping /mnt/lustre-mds2 (opts:) on oleg229-server 00:04:06 (1735103046) shut down facet: mds2 facet_host: oleg229-server facet_failover_host: oleg229-server Failover mds2 to oleg229-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg229-server: oleg229-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Started lustre-MDT0001 00:04:21 (1735103061) targets are mounted 00:04:21 (1735103061) facet_failover done ncy 5613.676 ms oleg229-client.virtnet: 1 51856 1.61 MB/sec execute 109 sec latency 6613.798 ms oleg229-client.virtnet: 1 51856 1.60 MB/sec execute 110 sec latency 7613.955 ms oleg229-client.virtnet: 1 51856 1.58 MB/sec execute 111 sec latency 8614.111 ms oleg229-client.virtnet: 1 51856 1.57 MB/sec execute 112 sec latency 9614.335 ms oleg229-client.virtnet: 1 51856 1.56 MB/sec execute 113 sec latency 10614.541 ms oleg229-client.virtnet: 1 51856 1.54 MB/sec execute 114 sec latency 11614.708 ms oleg229-client.virtnet: 1 51856 1.53 MB/sec execute 115 sec latency 12614.861 ms oleg229-client.virtnet: 1 51856 1.52 MB/sec execute 116 sec latency 13615.106 ms oleg229-client.virtnet: 1 51856 1.50 MB/sec execute 117 sec latency 14615.281 ms oleg229-client.virtnet: 1 51856 1.49 MB/sec execute 118 sec latency 15615.429 ms oleg229-client.virtnet: 1 51856 1.48 MB/sec execute 119 sec latency 16615.567 ms oleg229-client.virtnet: 1 51856 1.47 MB/sec execute 120 sec latency 17615.784 ms oleg229-client.virtnet: 1 51856 1.45 MB/sec execute 121 sec latency 18615.996 ms oleg229-client.virtnet: 1 51856 1.44 MB/sec execute 122 sec latency 19616.209 ms oleg229-client.virtnet: 1 51856 1.43 MB/sec execute 123 sec latency 20616.429 ms oleg229-client.virtnet: 1 51856 1.42 MB/sec execute 124 sec latency 21616.709 ms oleg229-client.virtnet: 1 51856 1.41 MB/sec execute 125 sec latency 22616.856 ms oleg229-client.virtnet: 1 51856 1.40 MB/sec execute 126 sec latency 23617.018 ms oleg229-client.virtnet: 1 52019 1.39 MB/sec execute 127 sec latency 24387.999 ms oleg229-client.virtnet: 1 52663 1.43 MB/sec execute 128 sec latency 19.609 ms oleg229-client.virtnet: 1 53318 1.46 MB/sec execute 129 sec latency 13.548 ms oleg229-client.virtnet: 1 53677 1.46 MB/sec execute 130 sec latency 12.571 ms oleg229-client.virtnet: 1 54132 1.45 MB/sec execute 131 sec latency 13.220 ms oleg229-client.virtnet: 1 54724 1.47 MB/sec execute 132 sec latency 10.933 ms oleg229-client.virtnet: 1 55250 1.46 MB/sec execute 133 sec latency 11.413 ms oleg229-client.virtnet: 1 55870 1.47 MB/sec execute 134 sec latency 10.436 ms oleg229-client.virtnet: 1 56899 1.53 MB/sec execute 135 sec latency 10.647 ms oleg229-client.virtnet: 1 57383 1.53 MB/sec execute 136 sec latency 10.985 ms oleg229-client.virtnet: 1 57853 1.53 MB/sec execute 137 sec latency 11.726 ms oleg229-client.virtnet: 1 58400 1.55 MB/sec execute 138 sec latency 11.279 ms oleg229-client.virtnet: 1 58943 1.54 MB/sec execute 139 sec latency 11.635 ms oleg229-client.virtnet: 1 59966 1.58 MB/sec execute 140 sec latency 10.400 ms oleg229-client.virtnet: 1 60670 1.61 MB/sec execute 141 sec latency 9.794 ms oleg229-client.virtnet: 1 61139 1.61 MB/sec execute 142 sec latency 14.645 ms oleg229-client.virtnet: 1 61609 1.61 MB/sec execute 143 sec latency 11.669 ms oleg229-client.virtnet: 1 62130 1.62 MB/sec execute 144 sec latency 12.170 ms oleg229-client.virtnet: 1 62731 1.62 MB/sec execute 145 sec latency 9.877 ms oleg229-client.virtnet: 1 63517 1.65 MB/sec execute 146 sec latency 15.622 ms oleg229-client.virtnet: 1 64134 1.67 MB/sec execute 147 sec latency 11.307 ms oleg229-client.virtnet: 1 64551 1.67 MB/sec execute 148 sec latency 12.524 ms oleg229-client.virtnet: 1 64937 1.67 MB/sec execute 149 sec latency 15.350 ms oleg229-client.virtnet: 1 65005 1.66 MB/sec execute 150 sec latency 867.238 ms oleg229-client.virtnet: 1 65005 1.65 MB/sec execute 151 sec latency 1867.409 ms oleg229-client.virtnet: 1 65005 1.64 MB/sec execute 152 sec latency 2867.628 ms oleg229-client.virtnoleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2472 1285216 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37428 3569004 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12072 3593732 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49500 7162736 1% /mnt/lustre test_70b fail mds1 9 times Failing mds1 on oleg229-server Stopping /mnt/lustre-mds1 (opts:) on oleg229-server 00:04:32 (1735103072) shut down facet: mds1 facet_host: oleg229-server facet_failover_host: oleg229-server Failover mds1 to oleg229-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg229-server: oleg229-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Started lustre-MDT0000 00:04:46 (1735103086) targets are mounted 00:04:46 (1735103086) facet_failover done oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2440 1285248 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37540 3568780 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12096 3591956 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49636 7160736 1% /mnt/lustre test_70b fail mds2 10 times Failing mds2 on oleg229-server Stopping /mnt/lustre-mds2 (opts:) on oleg229-server 00:04:59 (1735103099) shut down facet: mds2 facet_host: oleg229-server facet_failover_host: oleg229-server et: 1 65280 1.64 MB/sec execute 153 sec latency 3469.890 ms oleg229-client.virtnet: 1 65690 1.64 MB/sec execute 154 sec latency 12.606 ms oleg229-client.virtnet: 1 66261 1.64 MB/sec execute 155 sec latency 12.728 ms oleg229-client.virtnet: 1 67384 1.69 MB/sec execute 156 sec latency 14.102 ms oleg229-client.virtnet: 1 67929 1.69 MB/sec execute 157 sec latency 8.658 ms oleg229-client.virtnet: 1 68294 1.69 MB/sec execute 158 sec latency 418.900 ms oleg229-client.virtnet: 1 68884 1.70 MB/sec execute 159 sec latency 10.660 ms oleg229-client.virtnet: 1 69302 1.69 MB/sec execute 160 sec latency 123.380 ms oleg229-client.virtnet: 1 69302 1.68 MB/sec execute 161 sec latency 1123.556 ms oleg229-client.virtnet: 1 69302 1.67 MB/sec execute 162 sec latency 2123.714 ms oleg229-client.virtnet: 1 69302 1.66 MB/sec execute 163 sec latency 3123.894 ms oleg229-client.virtnet: 1 69302 1.65 MB/sec execute 164 sec latency 4124.067 ms oleg229-client.virtnet: 1 69302 1.64 MB/sec execute 165 sec latency 5124.222 ms oleg229-client.virtnet: 1 69302 1.63 MB/sec execute 166 sec latency 6124.364 ms oleg229-client.virtnet: 1 69302 1.62 MB/sec execute 167 sec latency 7124.647 ms oleg229-client.virtnet: 1 69302 1.61 MB/sec execute 168 sec latency 8124.859 ms oleg229-client.virtnet: 1 69302 1.60 MB/sec execute 169 sec latency 9125.026 ms oleg229-client.virtnet: 1 69302 1.59 MB/sec execute 170 sec latency 10125.212 ms oleg229-client.virtnet: 1 69302 1.59 MB/sec execute 171 sec latency 11125.716 ms oleg229-client.virtnet: 1 69302 1.58 MB/sec execute 172 sec latency 12126.129 ms oleg229-client.virtnet: 1 69302 1.57 MB/sec execute 173 sec latency 13126.374 ms oleg229-client.virtnet: 1 69302 1.56 MB/sec execute 174 sec latency 14126.647 ms oleg229-client.virtnet: 1 69302 1.55 MB/sec execute 175 sec latency 15126.826 ms oleg229-client.virtnet: 1 69302 1.54 MB/sec execute 176 sec latency 16126.966 ms oleg229-client.virtnet: 1 69302 1.53 MB/sec execute 177 sec latency 17127.122 ms oleg229-client.virtnet: 1 69302 1.52 MB/sec execute 178 sec latency 18127.389 ms oleg229-client.virtnet: 1 69410 1.51 MB/sec execute 179 sec latency 18795.902 ms oleg229-client.virtnet: 1 69675 1.51 MB/sec execute 180 sec latency 26.993 ms oleg229-client.virtnet: 1 70001 1.51 MB/sec execute 181 sec latency 22.135 ms oleg229-client.virtnet: 1 70753 1.54 MB/sec execute 182 sec latency 17.675 ms oleg229-client.virtnet: 1 71180 1.55 MB/sec execute 183 sec latency 14.906 ms oleg229-client.virtnet: 1 71478 1.55 MB/sec execute 184 sec latency 14.343 ms oleg229-client.virtnet: 1 71843 1.54 MB/sec execute 185 sec latency 15.795 ms oleg229-client.virtnet: 1 72228 1.54 MB/sec execute 186 sec latency 17.302 ms oleg229-client.virtnet: 1 72733 1.55 MB/sec execute 187 sec latency 14.321 ms oleg229-client.virtnet: 1 73158 1.55 MB/sec execute 188 sec latency 14.686 ms oleg229-client.virtnet: 1 73892 1.57 MB/sec execute 189 sec latency 12.009 ms oleg229-client.virtnet: 1 74675 1.59 MB/sec execute 190 sec latency 12.817 ms oleg229-client.virtnet: 1 75018 1.59 MB/sec execute 191 sec latency 14.191 ms oleg229-client.virtnet: 1 75446 1.59 MB/sec execute 192 sec latency 13.778 ms oleg229-client.virtnet: 1 75978 1.60 MB/sec execute 193 sec latency 11.793 ms oleg229-client.virtnet: 1 76454 1.59 MB/sec execute 194 sec latency 12.363 ms oleg229-client.virtnet: 1 76971 1.60 MB/sec execute 195 sec latency 11.454 ms oleg229-client.virtnet: 1 77846 1.62 MB/sec execute 196 sec latency 12.700 ms oleg229-client.virtnet: 1 78435 1.64 MB/sec execuFailover mds2 to oleg229-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg229-server: oleg229-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Started lustre-MDT0001 00:05:13 (1735103113) targets are mounted 00:05:13 (1735103113) facet_failover done oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2464 1285224 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37548 3568884 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12088 3593044 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49636 7161928 1% /mnt/lustre test_70b fail mds1 11 times Failing mds1 on oleg229-server Stopping /mnt/lustre-mds1 (opts:) on oleg229-server 00:05:25 (1735103125) shut down facet: mds1 facet_host: oleg229-server facet_failover_host: oleg229-server Failover mds1 to oleg229-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg229-server: oleg229-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Started lustre-MDT0000 00:05:39 (1735103139) targets are mounted 00:05:39 (1735103139) facet_failover done oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2440 1285248 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37548 3569060 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12076 3592720 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49624 7161780 1% /mnt/lustre test_70b fail mds2 12 times Failing mds2 on oleg229-server Stopping /mnt/lustre-mds2 (opts:) on oleg229-server 00:05:51 (1735103151) shut down facet: mds2 facet_host: oleg229-server facet_failover_host: oleg229-server te 197 sec latency 10.083 ms oleg229-client.virtnet: 1 78824 1.63 MB/sec execute 198 sec latency 15.888 ms oleg229-client.virtnet: 1 79248 1.63 MB/sec execute 199 sec latency 13.996 ms oleg229-client.virtnet: 1 79778 1.64 MB/sec execute 200 sec latency 11.717 ms oleg229-client.virtnet: 1 80267 1.64 MB/sec execute 201 sec latency 10.920 ms oleg229-client.virtnet: 1 80541 1.63 MB/sec execute 202 sec latency 400.884 ms oleg229-client.virtnet: 1 80541 1.63 MB/sec execute 203 sec latency 1401.039 ms oleg229-client.virtnet: 1 80541 1.62 MB/sec execute 204 sec latency 2401.213 ms oleg229-client.virtnet: 1 80541 1.61 MB/sec execute 205 sec latency 3401.400 ms oleg229-client.virtnet: 1 81329 1.64 MB/sec execute 206 sec latency 3560.646 ms oleg229-client.virtnet: 1 81862 1.65 MB/sec execute 207 sec latency 13.546 ms oleg229-client.virtnet: 1 82129 1.65 MB/sec execute 208 sec latency 19.508 ms oleg229-client.virtnet: 1 82564 1.64 MB/sec execute 209 sec latency 17.357 ms oleg229-client.virtnet: 1 83047 1.65 MB/sec execute 210 sec latency 16.542 ms oleg229-client.virtnet: 1 83294 1.65 MB/sec execute 211 sec latency 354.798 ms oleg229-client.virtnet: 1 83777 1.64 MB/sec execute 212 sec latency 12.795 ms oleg229-client.virtnet: 1 84110 1.64 MB/sec execute 213 sec latency 196.632 ms oleg229-client.virtnet: 1 84110 1.63 MB/sec execute 214 sec latency 1196.988 ms oleg229-client.virtnet: 1 84110 1.63 MB/sec execute 215 sec latency 2197.141 ms oleg229-client.virtnet: 1 84110 1.62 MB/sec execute 216 sec latency 3197.321 ms oleg229-client.virtnet: 1 84110 1.61 MB/sec execute 217 sec latency 4197.448 ms oleg229-client.virtnet: 1 84110 1.60 MB/sec execute 218 sec latency 5197.612 ms oleg229-client.virtnet: 1 84110 1.60 MB/sec execute 219 sec latency 6197.784 ms oleg229-client.virtnet: 1 84110 1.59 MB/sec execute 220 sec latency 7198.418 ms oleg229-client.virtnet: 1 84110 1.58 MB/sec execute 221 sec latency 8198.593 ms oleg229-client.virtnet: 1 84110 1.58 MB/sec execute 222 sec latency 9198.718 ms oleg229-client.virtnet: 1 84110 1.57 MB/sec execute 223 sec latency 10198.872 ms oleg229-client.virtnet: 1 84110 1.56 MB/sec execute 224 sec latency 11199.111 ms oleg229-client.virtnet: 1 84110 1.55 MB/sec execute 225 sec latency 12199.713 ms oleg229-client.virtnet: 1 84110 1.55 MB/sec execute 226 sec latency 13199.936 ms oleg229-client.virtnet: 1 84110 1.54 MB/sec execute 227 sec latency 14200.127 ms oleg229-client.virtnet: 1 84110 1.53 MB/sec execute 228 sec latency 15200.320 ms oleg229-client.virtnet: 1 84110 1.53 MB/sec execute 229 sec latency 16200.499 ms oleg229-client.virtnet: 1 84110 1.52 MB/sec execute 230 sec latency 17200.667 ms oleg229-client.virtnet: 1 84110 1.51 MB/sec execute 231 sec latency 18200.917 ms oleg229-client.virtnet: 1 84351 1.52 MB/sec execute 232 sec latency 18803.887 ms oleg229-client.virtnet: 1 84997 1.53 MB/sec execute 233 sec latency 20.999 ms oleg229-client.virtnet: 1 85457 1.54 MB/sec execute 234 sec latency 14.544 ms oleg229-client.virtnet: 1 85843 1.54 MB/sec execute 235 sec latency 17.471 ms oleg229-client.virtnet: 1 86257 1.54 MB/sec execute 236 sec latency 16.312 ms oleg229-client.virtnet: 1 86730 1.55 MB/sec execute 237 sec latency 98.411 ms oleg229-client.virtnet: 1 87159 1.54 MB/sec execute 238 sec latency 11.415 ms oleg229-client.virtnet: 1 87618 1.54 MB/sec execute 239 sec latency 13.701 ms oleg229-client.virtnet: 1 88407 1.56 MB/sec execute 240 sec latency 15.293 ms oleg229-client.virtnet: 1 88982 1.58 MB/sec execute 241 sec latency 11.390 ms oleFailover mds2 to oleg229-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg229-server: oleg229-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Started lustre-MDT0001 00:06:06 (1735103166) targets are mounted 00:06:06 (1735103166) facet_failover done oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec g229-client.virtnet: 1 89354 1.57 MB/sec execute 242 sec latency 12.086 ms oleg229-client.virtnet: 1 89764 1.57 MB/sec execute 243 sec latency 13.444 ms oleg229-client.virtnet: 1 90308 1.58 MB/sec execute 244 sec latency 11.721 ms oleg229-client.virtnet: 1 90742 1.58 MB/sec execute 245 sec latency 13.026 ms oleg229-client.virtnet: 1 91287 1.58 MB/sec execute 246 sec latency 11.665 ms oleg229-client.virtnet: 1 92227 1.61 MB/sec execute 247 sec latency 12.385 ms oleg229-client.virtnet: 1 92655 1.61 MB/sec execute 248 sec latency 10.505 ms oleg229-client.virtnet: 1 93138 1.61 MB/sec execute 249 sec latency 12.369 ms oleg229-client.virtnet: 1 93717 1.62 MB/sec execute 250 sec latency 10.215 ms oleg229-client.virtnet: 1 94130 1.61 MB/sec execute 251 sec latency 11.935 ms oleg229-client.virtnet: 1 94712 1.61 MB/sec execute 252 sec latency 10.913 ms oleg229-client.virtnet: 1 95437 1.63 MB/sec execute 253 sec latency 17.443 ms oleg229-client.virtnet: 1 95972 1.64 MB/sec execute 254 sec latency 12.434 ms oleg229-client.virtnet: 1 96138 1.64 MB/sec execute 255 sec latency 453.814 ms oleg229-client.virtnet: 1 96138 1.63 MB/sec execute 256 sec latency 1454.018 ms oleg229-client.virtnet: 1 96138 1.63 MB/sec execute 257 sec latency 2454.231 ms oleg229-client.virtnet: 1 96213 1.62 MB/sec execute 258 sec latency 3204.193 ms oleg229-client.virtnet: 1 96568 1.62 MB/sec execute 259 sec latency 14.287 ms oleg229-client.virtnet: 1 96927 1.62 MB/sec execute 260 sec latency 17.255 ms oleg229-client.virtnet: 1 97350 1.62 MB/sec execute 261 sec latency 12.748 ms oleg229-client.virtnet: 1 97659 1.62 MB/sec execute 262 sec latency 15.966 ms oleg229-client.virtnet: 1 98119 1.62 MB/sec execute 263 sec latency 11.695 ms oleg229-client.virtnet: 1 98711 1.63 MB/sec execute 264 sec latency 19.316 ms oleg229-client.virtnet: 1 99379 1.65 MB/sec execute 265 sec latency 14.992 ms oleg229-client.virtnet: 1 99704 1.65 MB/sec execute 266 sec latency 17.749 ms oleg229-client.virtnet: 1 100038 1.64 MB/sec execute 267 sec latency 13.849 ms oleg229-client.virtnet: 1 100356 1.64 MB/sec execute 268 sec latency 17.430 ms oleg229-client.virtnet: 1 100833 1.65 MB/sec execute 269 sec latency 14.253 ms oleg229-client.virtnet: 1 101197 1.64 MB/sec execute 270 sec latency 12.174 ms oleg229-client.virtnet: 1 101554 1.64 MB/sec execute 271 sec latency 17.332 ms oleg229-client.virtnet: 1 101942 1.64 MB/sec execute 272 sec latency 18.249 ms oleg229-client.virtnet: 1 102649 1.66 MB/sec execute 273 sec latency 17.266 ms oleg229-client.virtnet: 1 103053 1.66 MB/sec execute 274 sec latency 16.781 ms oleg229-client.virtnet: 1 103311 1.66 MB/sec execute 275 sec latency 17.351 ms oleg229-client.virtnet: 1 103591 1.66 MB/sec execute 276 sec latency 26.959 ms oleg229-client.virtnet: 1 103859 1.65 MB/sec execute 277 sec latency 21.614 ms oleg229-client.virtnet: 1 104201 1.65 MB/sec execute 278 sec latency 16.217 ms oleg229-client.virtnet: 1 104571 1.66 MB/sec execute 279 sec latency 17.555 ms oleg229-client.virtnet: 1 104889 1.65 MB/sec execute 280 sec latency 17.368 ms oleg229-client.virtnet: 1 105257 1.65 MB/sec execute 281 sec latency 19.125 ms oleg229-client.virtnet: 1 105788 1.66 MB/sec execute 282 sec latency 15.564 ms oleg229-client.virtnet: 1 106432 1.68 MB/sec execute 283 sec latency 19.164 ms oleg229-client.virtnet: 1 106779 1.68 MB/sec execute 284 sec latency 16.150 ms oleg229-client.virtnet: 1 107148 1.67 MB/sec execute 285 sec latency 18.488 ms oleg229-client.virtnet: 1 107533 1.67 MB/sec execute 286 sec latency 13.830 ms oleg229-client.virtnet: 1 107973 1.68 MB/sec execute 287 sec latency 13.335 ms oleg229-client.virtnet: 1 108311 1.67 MB/sec execute 288 sec latency 13.601 ms oleg229-client.virtnet: 1 108757 1.67 MB/sec execute 289 sec latency 16.786 ms oleg229-client.virtnet: 1 109325 1.68 MB/sec execute 290 sec latency 16.333 ms oleg229-client.virtnet: 1 109996 1.70 MB/sec execute 291 sec latency 17.101 ms oleg229-client.virtnet: 1 110312 1.69 MB/sec execute 292 sec latency 16.594 ms oleg229-client.virtnet: 1 110655 1.69 MB/sec execute 293 sec latency 16.283 ms oleg229-client.virtnet: 1 111000 1.69 MB/sec execute 294 sec latency 16.986 ms oleg229-client.virtnet: 1 111493 1.70 MB/sec execute 295 sec latency 12.364 ms oleg229-client.virtnet: 1 111861 1.69 MB/sec execute 296 sec latency 16.316 ms oleg229-client.virtnet: 1 112346 1.69 MB/sec execute 297 sec latency 12.462 ms oleg229-client.virtnet: 1 112894 1.70 MB/sec execute 298 sec latency 19.091 ms oleg229-client.virtnet: 1 113561 1.71 MB/sec execute 299 sec latency 16.405 ms oleg229-client.virtnet: 1 cleanup 300 sec oleg229-client.virtnet: 0 cleanup 300 sec oleg229-client.virtnet: oleg229-client.virtnet: Operation Count AvgLat MaxLat oleg229-client.virtnet: ---------------------------------------- oleg229-client.virtnet: NTCreateX 16421 10.917 24387.992 oleg229-client.virtnet: Close 12071 2.749 18795.880 oleg229-client.virtnet: Rename 695 9.356 19.592 oleg229-client.virtnet: Unlink 3295 2.205 8.503 oleg229-client.virtnet: Qpathinfo 14890 1.879 3469.880 oleg229-client.virtnet: Qfileinfo 2607 0.165 2.385 oleg229-client.virtnet: Qfsinfo 2724 0.234 5.723 oleg229-client.virtnet: Sfileinfo 1325 19.818 18803.878 oleg229-client.virtnet: Find 5755 0.435 12.162 oleg229-client.virtnet: WriteX 8115 0.973 13.916 oleg229-client.virtnet: ReadX 25875 0.023 4.607 oleg229-client.virtnet: LockX 54 1.502 4.118 oleg229-client.virtnet: UnlockX 54 1.384 4.248 oleg229-client.virtnet: Flush 1137 7.488 425.864 oleg229-client.virtnet: oleg229-client.virtnet: Throughput 1.71439 MB/sec 1 clients 1 procs max_latency=24387.999 ms oleg229-client.virtnet: stopping dbench on /mnt/lustre/d70b.replay-single/oleg229-client.virtnet at Wed Dec 25 00:06:52 EST 2024 with return code 0 oleg229-client.virtnet: clean dbench files on /mnt/lustre/d70b.replay-single/oleg229-client.virtnet oleg229-client.virtnet: /mnt/lustre/d70b.replay-single/oleg229-client.virtnet /mnt/lustre/d70b.replay-single/oleg229-client.virtnet oleg229-client.virtnet: removed directory: 'clients/client0/~dmtmp/ACCESS' oleg229-client.virtnet: removed directory: 'clients/client0/~dmtmp/EXCEL' oleg229-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORDPRO' oleg229-client.virtnet: removed directory: 'clients/client0/~dmtmp/COREL' oleg229-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORD' oleg229-client.virtnet: removed directory: 'clients/client0/~dmtmp/PARADOX' oleg229-client.virtnet: removed directory: 'clients/client0/~dmtmp/PM' oleg229-client.virtnet: removed directory: 'clients/client0/~dmtmp/SEED' oleg229-client.virtnet: removed directory: 'clients/client0/~dmtmp/PWRPNT' oleg229-client.virtnet: removed directory: 'clients/client0/~dmtmp' oleg229-client.virtnet: removed directory: 'clients/client0' oleg229-client.virtnet: removed directory: 'clients' oleg229-client.virtnet: removed 'client.txt' oleg229-client.virtnet: /mnt/lustre/d70b.replay-single/oleg229-client.virtnet oleg229-client.virtnet: dbench successfully finished