== replay-single test 70b: dbench 2mdts recovery; 1 clients ========================================================== 00:38:49 (1733809129) Starting client oleg337-client.virtnet: -o user_xattr,flock 192.168.203.137@tcp:/lustre /mnt/lustre Started clients oleg337-client.virtnet: 192.168.203.137@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) striped dir -i0 -c2 -H crush /mnt/lustre/d70b.replay-single + MISSING_DBENCH_OK= + PATH=/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/opt/iozone/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin::/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests:/sbin:/usr/sbin:/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests/: + DBENCH_LIB= + TESTSUITE=replay-single + TESTNAME=test_70b + MOUNT=/mnt/lustre ++ hostname + DIR=/mnt/lustre/d70b.replay-single/oleg337-client.virtnet + LCTL=/home/green/git/lustre-release/lustre/utils/lctl + rundbench 1 -t 300 dbench: no process found Started rundbench load pid=13987 ... UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2520 1285168 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2116 1285572 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 26160 3556140 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 5800 3594212 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 31960 7150352 1% /mnt/lustre test_70b fail mds1 1 times Failing mds1 on oleg337-server Stopping /mnt/lustre-mds1 (opts:) on oleg337-server 00:38:56 (1733809136) shut down facet: mds1 facet_host: oleg337-server facet_failover_host: oleg337-server Failover mds1 to oleg337-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg337-server: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-MDT0000 00:39:10 (1733809150) targets are mounted 00:39:10 (1733809150) facet_failover done oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2476 1285212 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2076 1285612 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 26576 3568704 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 5404 3591688 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 31980 7160392 1% /mnt/lustre test_70b fail mds2 2 times Failing mds2 on oleg337-server Stopping /mnt/lustre-mds2 (opts:) on oleg337-server 00:39:22 (1733809162) shut down facet: mds2 facet_host: oleg337-server facet_failover_host: oleg337-server oleg337-client.virtnet: looking for dbench program oleg337-client.virtnet: /usr/bin/dbench oleg337-client.virtnet: creating output directory /mnt/lustre/d70b.replay-single/oleg337-client.virtnet oleg337-client.virtnet: mkdir: created directory '/mnt/lustre/d70b.replay-single/oleg337-client.virtnet' oleg337-client.virtnet: found dbench client file /usr/share/dbench/client.txt oleg337-client.virtnet: '/usr/share/dbench/client.txt' -> 'client.txt' oleg337-client.virtnet: running 'dbench 1 -t 300' on /mnt/lustre/d70b.replay-single/oleg337-client.virtnet at Tue Dec 10 00:38:50 EST 2024 oleg337-client.virtnet: waiting for dbench pid 14012 oleg337-client.virtnet: dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 oleg337-client.virtnet: oleg337-client.virtnet: Running for 300 seconds with load 'client.txt' and minimum warmup 60 secs oleg337-client.virtnet: failed to create barrier semaphore oleg337-client.virtnet: 0 of 1 processes prepared for launch 0 sec oleg337-client.virtnet: 1 of 1 processes prepared for launch 0 sec oleg337-client.virtnet: releasing clients oleg337-client.virtnet: 1 522 18.05 MB/sec warmup 1 sec latency 15.015 ms oleg337-client.virtnet: 1 1118 11.48 MB/sec warmup 2 sec latency 13.715 ms oleg337-client.virtnet: 1 1825 8.78 MB/sec warmup 3 sec latency 8.897 ms oleg337-client.virtnet: 1 2489 7.18 MB/sec warmup 4 sec latency 9.005 ms oleg337-client.virtnet: 1 3688 7.75 MB/sec warmup 5 sec latency 12.488 ms oleg337-client.virtnet: 1 4022 6.68 MB/sec warmup 6 sec latency 351.241 ms oleg337-client.virtnet: 1 4022 5.72 MB/sec warmup 7 sec latency 1351.438 ms oleg337-client.virtnet: 1 4022 5.01 MB/sec warmup 8 sec latency 2351.567 ms oleg337-client.virtnet: 1 4022 4.45 MB/sec warmup 9 sec latency 3351.689 ms oleg337-client.virtnet: 1 4022 4.01 MB/sec warmup 10 sec latency 4351.815 ms oleg337-client.virtnet: 1 4022 3.64 MB/sec warmup 11 sec latency 5351.968 ms oleg337-client.virtnet: 1 4022 3.34 MB/sec warmup 12 sec latency 6352.105 ms oleg337-client.virtnet: 1 4022 3.08 MB/sec warmup 13 sec latency 7352.245 ms oleg337-client.virtnet: 1 4022 2.86 MB/sec warmup 14 sec latency 8352.395 ms oleg337-client.virtnet: 1 4022 2.67 MB/sec warmup 15 sec latency 9352.550 ms oleg337-client.virtnet: 1 4022 2.50 MB/sec warmup 16 sec latency 10352.699 ms oleg337-client.virtnet: 1 4022 2.36 MB/sec warmup 17 sec latency 11352.842 ms oleg337-client.virtnet: 1 4022 2.23 MB/sec warmup 18 sec latency 12352.972 ms oleg337-client.virtnet: 1 4022 2.11 MB/sec warmup 19 sec latency 13353.064 ms oleg337-client.virtnet: 1 4022 2.00 MB/sec warmup 20 sec latency 14353.201 ms oleg337-client.virtnet: 1 4022 1.91 MB/sec warmup 21 sec latency 15353.298 ms oleg337-client.virtnet: 1 4022 1.82 MB/sec warmup 22 sec latency 16353.438 ms oleg337-client.virtnet: 1 4022 1.74 MB/sec warmup 23 sec latency 17353.605 ms oleg337-client.virtnet: 1 4022 1.67 MB/sec warmup 24 sec latency 18353.713 ms oleg337-client.virtnet: 1 4458 1.63 MB/sec warmup 25 sec latency 18450.878 ms oleg337-client.virtnet: 1 4976 1.74 MB/sec warmup 26 sec latency 12.305 ms oleg337-client.virtnet: 1 5317 1.68 MB/sec warmup 27 sec latency 15.892 ms oleg337-client.virtnet: 1 5812 1.64 MB/sec warmup 28 sec latency 13.776 ms oleg337-client.virtnet: 1 6317 1.76 MB/sec warmup 29 sec latency 19.281 ms oleg337-client.virtnet: 1 7109 1.92 MB/sec warmup 30 sec latency 16.928 ms oleg337-client.virtnet: 1 7476 1.91 MB/sec warmup 31 sec latency 15.830 ms oleg337-client.virtnet: 1 7891 1.86 MB/sec warmup 32 sec latency 18.009 ms oleg337-client.virtnet: 1 8353 1.86 MB/sec warmup 33 sec latency 12.Failover mds2 to oleg337-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg337-server: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-MDT0001 00:39:36 (1733809176) targets are mounted 00:39:36 (1733809176) facet_failover done oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2464 1285224 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37644 3566152 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 11956 3593840 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49600 7159992 1% /mnt/lustre test_70b fail mds1 3 times Failing mds1 on oleg337-server Stopping /mnt/lustre-mds1 (opts:) on oleg337-server 00:39:48 (1733809188) shut down facet: mds1 facet_host: oleg337-server facet_failover_host: oleg337-server Failover mds1 to oleg337-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg337-server: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-MDT0000 00:40:02 (1733809202) targets are mounted 00:40:02 (1733809202) facet_failover done oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 241 ms oleg337-client.virtnet: 1 8889 1.90 MB/sec warmup 34 sec latency 13.942 ms oleg337-client.virtnet: 1 9480 1.89 MB/sec warmup 35 sec latency 12.636 ms oleg337-client.virtnet: 1 10404 2.04 MB/sec warmup 36 sec latency 13.525 ms oleg337-client.virtnet: 1 11000 2.12 MB/sec warmup 37 sec latency 10.863 ms oleg337-client.virtnet: 1 11476 2.07 MB/sec warmup 38 sec latency 12.470 ms oleg337-client.virtnet: 1 12040 2.14 MB/sec warmup 39 sec latency 13.281 ms oleg337-client.virtnet: 1 12517 2.09 MB/sec warmup 40 sec latency 10.474 ms oleg337-client.virtnet: 1 13082 2.08 MB/sec warmup 41 sec latency 10.579 ms oleg337-client.virtnet: 1 14013 2.21 MB/sec warmup 42 sec latency 14.010 ms oleg337-client.virtnet: 1 14583 2.27 MB/sec warmup 43 sec latency 11.259 ms oleg337-client.virtnet: 1 15071 2.23 MB/sec warmup 44 sec latency 17.215 ms oleg337-client.virtnet: 1 15653 2.28 MB/sec warmup 45 sec latency 10.977 ms oleg337-client.virtnet: 1 15926 2.23 MB/sec warmup 46 sec latency 390.320 ms oleg337-client.virtnet: 1 15926 2.19 MB/sec warmup 47 sec latency 1390.489 ms oleg337-client.virtnet: 1 15926 2.14 MB/sec warmup 48 sec latency 2390.720 ms oleg337-client.virtnet: 1 15926 2.10 MB/sec warmup 49 sec latency 3390.849 ms oleg337-client.virtnet: 1 15926 2.05 MB/sec warmup 50 sec latency 4391.007 ms oleg337-client.virtnet: 1 16597 2.05 MB/sec warmup 51 sec latency 4449.927 ms oleg337-client.virtnet: 1 17623 2.17 MB/sec warmup 52 sec latency 11.599 ms oleg337-client.virtnet: 1 18182 2.20 MB/sec warmup 53 sec latency 10.679 ms oleg337-client.virtnet: 1 18680 2.17 MB/sec warmup 54 sec latency 18.528 ms oleg337-client.virtnet: 1 19232 2.21 MB/sec warmup 55 sec latency 151.814 ms oleg337-client.virtnet: 1 19675 2.18 MB/sec warmup 56 sec latency 305.122 ms oleg337-client.virtnet: 1 20203 2.17 MB/sec warmup 57 sec latency 10.273 ms oleg337-client.virtnet: 1 20377 2.15 MB/sec warmup 58 sec latency 668.386 ms oleg337-client.virtnet: 1 20377 2.11 MB/sec warmup 59 sec latency 1668.576 ms oleg337-client.virtnet: 1 20377 0.00 MB/sec execute 1 sec latency 3668.846 ms oleg337-client.virtnet: 1 20377 0.00 MB/sec execute 2 sec latency 4669.008 ms oleg337-client.virtnet: 1 20377 0.00 MB/sec execute 3 sec latency 5669.135 ms oleg337-client.virtnet: 1 20377 0.00 MB/sec execute 4 sec latency 6669.260 ms oleg337-client.virtnet: 1 20377 0.00 MB/sec execute 5 sec latency 7669.387 ms oleg337-client.virtnet: 1 20377 0.00 MB/sec execute 6 sec latency 8669.529 ms oleg337-client.virtnet: 1 20377 0.00 MB/sec execute 7 sec latency 9669.660 ms oleg337-client.virtnet: 1 20377 0.00 MB/sec execute 8 sec latency 10669.794 ms oleg337-client.virtnet: 1 20377 0.00 MB/sec execute 9 sec latency 11669.924 ms oleg337-client.virtnet: 1 20377 0.00 MB/sec execute 10 sec latency 12670.052 ms oleg337-client.virtnet: 1 20377 0.00 MB/sec execute 11 sec latency 13670.193 ms oleg337-client.virtnet: 1 20377 0.00 MB/sec execute 12 sec latency 14670.368 ms oleg337-client.virtnet: 1 20377 0.00 MB/sec execute 13 sec latency 15670.531 ms oleg337-client.virtnet: 1 20377 0.00 MB/sec execute 14 sec latency 16670.664 ms oleg337-client.virtnet: 1 20377 0.00 MB/sec execute 15 sec latency 17670.840 ms oleg337-client.virtnet: 1 21052 0.40 MB/sec execute 16 sec latency 18146.042 ms oleg337-client.virtnet: 1 21701 0.65 MB/sec execute 17 sec latency 12.398 ms oleg337-client.virtnet: 1 22227 0.66 MB/sec execute 18 sec latency 14.163 ms oleg337-client.virtnet: 1 22877 0.85 MB/secUUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2436 1285252 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37568 3568324 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12060 3594188 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49628 7162512 1% /mnt/lustre test_70b fail mds2 4 times Failing mds2 on oleg337-server Stopping /mnt/lustre-mds2 (opts:) on oleg337-server 00:40:13 (1733809213) shut down facet: mds2 facet_host: oleg337-server facet_failover_host: oleg337-server Failover mds2 to oleg337-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg337-server: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-MDT0001 00:40:27 (1733809227) targets are mounted 00:40:27 (1733809227) facet_failover done oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2472 1285216 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37436 3568836 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12124 3593012 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49560 7161848 1% /mnt/lustre test_70b fail mds1 5 times Failing mds1 on oleg337-server Stopping /mnt/lustre-mds1 (opts:) on oleg337-server 00:40:39 (1733809239) shut down facet: mds1 facet_host: oleg337-server facet_failover_host: oleg337-server Failover mds1 to oleg337-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg337-server: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-MDT0000 00:40:52 (1733809252) targets are mounted 00:40:52 (1733809252) facet_failover done execute 19 sec latency 10.594 ms oleg337-client.virtnet: 1 23536 0.84 MB/sec execute 20 sec latency 10.051 ms oleg337-client.virtnet: 1 24662 1.21 MB/sec execute 21 sec latency 9.452 ms oleg337-client.virtnet: 1 25317 1.37 MB/sec execute 22 sec latency 9.035 ms oleg337-client.virtnet: 1 25920 1.40 MB/sec execute 23 sec latency 11.227 ms oleg337-client.virtnet: 1 26600 1.48 MB/sec execute 24 sec latency 9.164 ms oleg337-client.virtnet: 1 27304 1.49 MB/sec execute 25 sec latency 8.746 ms oleg337-client.virtnet: 1 28409 1.83 MB/sec execute 26 sec latency 9.846 ms oleg337-client.virtnet: 1 28942 1.83 MB/sec execute 27 sec latency 10.179 ms oleg337-client.virtnet: 1 29655 1.84 MB/sec execute 28 sec latency 8.725 ms oleg337-client.virtnet: 1 30410 1.89 MB/sec execute 29 sec latency 7.806 ms oleg337-client.virtnet: 1 31348 2.06 MB/sec execute 30 sec latency 9.385 ms oleg337-client.virtnet: 1 32286 2.20 MB/sec execute 31 sec latency 8.372 ms oleg337-client.virtnet: 1 32867 2.16 MB/sec execute 32 sec latency 10.496 ms oleg337-client.virtnet: 1 33632 2.23 MB/sec execute 33 sec latency 9.995 ms oleg337-client.virtnet: 1 34382 2.22 MB/sec execute 34 sec latency 8.050 ms oleg337-client.virtnet: 1 35537 2.45 MB/sec execute 35 sec latency 10.159 ms oleg337-client.virtnet: 1 35856 2.43 MB/sec execute 36 sec latency 318.441 ms oleg337-client.virtnet: 1 35856 2.37 MB/sec execute 37 sec latency 1318.604 ms oleg337-client.virtnet: 1 35856 2.30 MB/sec execute 38 sec latency 2318.750 ms oleg337-client.virtnet: 1 35856 2.24 MB/sec execute 39 sec latency 3318.897 ms oleg337-client.virtnet: 1 35856 2.19 MB/sec execute 40 sec latency 4319.068 ms oleg337-client.virtnet: 1 36010 2.14 MB/sec execute 41 sec latency 4942.987 ms oleg337-client.virtnet: 1 36661 2.13 MB/sec execute 42 sec latency 9.454 ms oleg337-client.virtnet: 1 37351 2.16 MB/sec execute 43 sec latency 8.565 ms oleg337-client.virtnet: 1 38061 2.17 MB/sec execute 44 sec latency 11.335 ms oleg337-client.virtnet: 1 39124 2.34 MB/sec execute 45 sec latency 12.164 ms oleg337-client.virtnet: 1 39591 2.32 MB/sec execute 46 sec latency 10.546 ms oleg337-client.virtnet: 1 40195 2.31 MB/sec execute 47 sec latency 10.711 ms oleg337-client.virtnet: 1 40675 2.33 MB/sec execute 48 sec latency 85.269 ms oleg337-client.virtnet: 1 40675 2.28 MB/sec execute 49 sec latency 1085.406 ms oleg337-client.virtnet: 1 40675 2.24 MB/sec execute 50 sec latency 2085.549 ms oleg337-client.virtnet: 1 40675 2.19 MB/sec execute 51 sec latency 3085.716 ms oleg337-client.virtnet: 1 40675 2.15 MB/sec execute 52 sec latency 4085.850 ms oleg337-client.virtnet: 1 40675 2.11 MB/sec execute 53 sec latency 5086.017 ms oleg337-client.virtnet: 1 40675 2.07 MB/sec execute 54 sec latency 6086.215 ms oleg337-client.virtnet: 1 40675 2.03 MB/sec execute 55 sec latency 7086.461 ms oleg337-client.virtnet: 1 40675 2.00 MB/sec execute 56 sec latency 8086.641 ms oleg337-client.virtnet: 1 40675 1.96 MB/sec execute 57 sec latency 9086.846 ms oleg337-client.virtnet: 1 40675 1.93 MB/sec execute 58 sec latency 10086.989 ms oleg337-client.virtnet: 1 40675 1.90 MB/sec execute 59 sec latency 11087.188 ms oleg337-client.virtnet: 1 40675 1.86 MB/sec execute 60 sec latency 12087.323 ms oleg337-client.virtnet: 1 40675 1.83 MB/sec execute 61 sec latency 13087.492 ms oleg337-client.virtnet: 1 40675 1.80 MB/sec execute 62 sec latency 14087.716 ms oleg337-client.virtnet: 1 40675 1.78 MB/sec execute 63 sec latency 15087.872 ms oleg337-client.virtneoleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2440 1285248 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37568 3568612 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12200 3591816 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49768 7160428 1% /mnt/lustre test_70b fail mds2 6 times Failing mds2 on oleg337-server Stopping /mnt/lustre-mds2 (opts:) on oleg337-server 00:41:04 (1733809264) shut down facet: mds2 facet_host: oleg337-server facet_failover_host: oleg337-server Failover mds2 to oleg337-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg337-server: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-MDT0001 00:41:18 (1733809278) targets are mounted 00:41:18 (1733809278) facet_failover done oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2472 1285216 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37428 3569004 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12380 3593752 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49808 7162756 1% /mnt/lustre test_70b fail mds1 7 times Failing mds1 on oleg337-server Stopping /mnt/lustre-mds1 (opts:) on oleg337-server 00:41:29 (1733809289) shut down facet: mds1 facet_host: oleg337-server facet_failover_host: oleg337-server t: 1 40675 1.75 MB/sec execute 64 sec latency 16088.002 ms oleg337-client.virtnet: 1 40675 1.72 MB/sec execute 65 sec latency 17088.139 ms oleg337-client.virtnet: 1 40675 1.70 MB/sec execute 66 sec latency 18088.272 ms oleg337-client.virtnet: 1 40995 1.67 MB/sec execute 67 sec latency 18449.253 ms oleg337-client.virtnet: 1 41831 1.73 MB/sec execute 68 sec latency 9.219 ms oleg337-client.virtnet: 1 42858 1.82 MB/sec execute 69 sec latency 9.542 ms oleg337-client.virtnet: 1 43425 1.80 MB/sec execute 70 sec latency 11.497 ms oleg337-client.virtnet: 1 44156 1.84 MB/sec execute 71 sec latency 9.371 ms oleg337-client.virtnet: 1 44768 1.83 MB/sec execute 72 sec latency 10.465 ms oleg337-client.virtnet: 1 45770 1.90 MB/sec execute 73 sec latency 9.586 ms oleg337-client.virtnet: 1 46532 1.96 MB/sec execute 74 sec latency 8.697 ms oleg337-client.virtnet: 1 47142 1.94 MB/sec execute 75 sec latency 10.158 ms oleg337-client.virtnet: 1 47866 1.98 MB/sec execute 76 sec latency 8.937 ms oleg337-client.virtnet: 1 48566 1.97 MB/sec execute 77 sec latency 8.291 ms oleg337-client.virtnet: 1 49707 2.08 MB/sec execute 78 sec latency 9.803 ms oleg337-client.virtnet: 1 50243 2.08 MB/sec execute 79 sec latency 8.313 ms oleg337-client.virtnet: 1 50932 2.08 MB/sec execute 80 sec latency 10.995 ms oleg337-client.virtnet: 1 51709 2.09 MB/sec execute 81 sec latency 8.909 ms oleg337-client.virtnet: 1 52604 2.15 MB/sec execute 82 sec latency 8.638 ms oleg337-client.virtnet: 1 53486 2.20 MB/sec execute 83 sec latency 11.374 ms oleg337-client.virtnet: 1 54047 2.19 MB/sec execute 84 sec latency 11.810 ms oleg337-client.virtnet: 1 54796 2.22 MB/sec execute 85 sec latency 8.714 ms oleg337-client.virtnet: 1 55591 2.21 MB/sec execute 86 sec latency 8.542 ms oleg337-client.virtnet: 1 56644 2.28 MB/sec execute 87 sec latency 11.442 ms oleg337-client.virtnet: 1 56892 2.28 MB/sec execute 88 sec latency 695.434 ms oleg337-client.virtnet: 1 56892 2.26 MB/sec execute 89 sec latency 1695.578 ms oleg337-client.virtnet: 1 56892 2.23 MB/sec execute 90 sec latency 2695.897 ms oleg337-client.virtnet: 1 56892 2.21 MB/sec execute 91 sec latency 3696.040 ms oleg337-client.virtnet: 1 57241 2.20 MB/sec execute 92 sec latency 3997.463 ms oleg337-client.virtnet: 1 57726 2.18 MB/sec execute 93 sec latency 12.196 ms oleg337-client.virtnet: 1 58391 2.21 MB/sec execute 94 sec latency 10.113 ms oleg337-client.virtnet: 1 59029 2.19 MB/sec execute 95 sec latency 9.196 ms oleg337-client.virtnet: 1 59973 2.25 MB/sec execute 96 sec latency 51.781 ms oleg337-client.virtnet: 1 60488 2.27 MB/sec execute 97 sec latency 459.368 ms oleg337-client.virtnet: 1 61075 2.26 MB/sec execute 98 sec latency 10.346 ms oleg337-client.virtnet: 1 61347 2.25 MB/sec execute 99 sec latency 543.650 ms oleg337-client.virtnet: 1 61347 2.23 MB/sec execute 100 sec latency 1543.781 ms oleg337-client.virtnet: 1 61347 2.21 MB/sec execute 101 sec latency 2543.944 ms oleg337-client.virtnet: 1 61347 2.18 MB/sec execute 102 sec latency 3544.097 ms oleg337-client.virtnet: 1 61347 2.16 MB/sec execute 103 sec latency 4544.254 ms oleg337-client.virtnet: 1 61347 2.14 MB/sec execute 104 sec latency 5544.400 ms oleg337-client.virtnet: 1 61347 2.12 MB/sec execute 105 sec latency 6544.563 ms oleg337-client.virtnet: 1 61347 2.10 MB/sec execute 106 sec latency 7544.705 ms oleg337-client.virtnet: 1 61347 2.08 MB/sec execute 107 sec latency 8544.883 ms oleg337-client.virtnet: 1 61347 2.06 MB/sec execute 108 sec latency 9545.03Failover mds1 to oleg337-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg337-server: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-MDT0000 00:41:43 (1733809303) targets are mounted 00:41:43 (1733809303) facet_failover done oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2432 1285256 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37480 3568868 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12192 3593036 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49672 7161904 1% /mnt/lustre test_70b fail mds2 8 times Failing mds2 on oleg337-server Stopping /mnt/lustre-mds2 (opts:) on oleg337-server 00:41:55 (1733809315) shut down facet: mds2 facet_host: oleg337-server facet_failover_host: oleg337-server Failover mds2 to oleg337-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg337-server: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-MDT0001 00:42:09 (1733809329) targets are mounted 00:42:09 (1733809329) facet_failover done oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2472 1285216 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37568 3568616 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12196 3591736 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49764 7160352 1% /mnt/lustre test_70b fail mds1 9 times Failing mds1 on oleg337-server Stopping /mnt/lustre-mds1 (opts:) on oleg337-server 00:42:20 (1733809340) shut down facet: mds1 facet_host: oleg337-server facet_failover_host: oleg337-server 5 ms oleg337-client.virtnet: 1 61347 2.04 MB/sec execute 109 sec latency 10545.146 ms oleg337-client.virtnet: 1 61347 2.02 MB/sec execute 110 sec latency 11545.241 ms oleg337-client.virtnet: 1 61347 2.01 MB/sec execute 111 sec latency 12545.421 ms oleg337-client.virtnet: 1 61347 1.99 MB/sec execute 112 sec latency 13545.608 ms oleg337-client.virtnet: 1 61347 1.97 MB/sec execute 113 sec latency 14545.825 ms oleg337-client.virtnet: 1 61347 1.95 MB/sec execute 114 sec latency 15545.954 ms oleg337-client.virtnet: 1 61347 1.94 MB/sec execute 115 sec latency 16546.100 ms oleg337-client.virtnet: 1 61347 1.92 MB/sec execute 116 sec latency 17546.205 ms oleg337-client.virtnet: 1 61483 1.91 MB/sec execute 117 sec latency 18364.343 ms oleg337-client.virtnet: 1 62134 1.92 MB/sec execute 118 sec latency 11.783 ms oleg337-client.virtnet: 1 62898 1.93 MB/sec execute 119 sec latency 8.346 ms oleg337-client.virtnet: 1 63998 1.99 MB/sec execute 120 sec latency 8.859 ms oleg337-client.virtnet: 1 64462 1.99 MB/sec execute 121 sec latency 11.191 ms oleg337-client.virtnet: 1 65024 1.99 MB/sec execute 122 sec latency 11.089 ms oleg337-client.virtnet: 1 65667 2.00 MB/sec execute 123 sec latency 8.965 ms oleg337-client.virtnet: 1 66312 2.00 MB/sec execute 124 sec latency 11.454 ms oleg337-client.virtnet: 1 67445 2.06 MB/sec execute 125 sec latency 8.922 ms oleg337-client.virtnet: 1 67946 2.06 MB/sec execute 126 sec latency 8.950 ms oleg337-client.virtnet: 1 68544 2.06 MB/sec execute 127 sec latency 12.776 ms oleg337-client.virtnet: 1 69141 2.07 MB/sec execute 128 sec latency 11.325 ms oleg337-client.virtnet: 1 69809 2.07 MB/sec execute 129 sec latency 8.180 ms oleg337-client.virtnet: 1 70847 2.12 MB/sec execute 130 sec latency 8.880 ms oleg337-client.virtnet: 1 71435 2.13 MB/sec execute 131 sec latency 9.598 ms oleg337-client.virtnet: 1 72000 2.13 MB/sec execute 132 sec latency 12.212 ms oleg337-client.virtnet: 1 72622 2.14 MB/sec execute 133 sec latency 11.365 ms oleg337-client.virtnet: 1 73160 2.12 MB/sec execute 134 sec latency 10.714 ms oleg337-client.virtnet: 1 73910 2.16 MB/sec execute 135 sec latency 12.598 ms oleg337-client.virtnet: 1 74708 2.18 MB/sec execute 136 sec latency 12.144 ms oleg337-client.virtnet: 1 75195 2.18 MB/sec execute 137 sec latency 11.217 ms oleg337-client.virtnet: 1 75543 2.17 MB/sec execute 138 sec latency 170.408 ms oleg337-client.virtnet: 1 75543 2.15 MB/sec execute 139 sec latency 1171.298 ms oleg337-client.virtnet: 1 75543 2.13 MB/sec execute 140 sec latency 2171.419 ms oleg337-client.virtnet: 1 75543 2.12 MB/sec execute 141 sec latency 3171.534 ms oleg337-client.virtnet: 1 75543 2.10 MB/sec execute 142 sec latency 4171.689 ms oleg337-client.virtnet: 1 75960 2.12 MB/sec execute 143 sec latency 4690.649 ms oleg337-client.virtnet: 1 76580 2.11 MB/sec execute 144 sec latency 14.967 ms oleg337-client.virtnet: 1 77542 2.14 MB/sec execute 145 sec latency 8.967 ms oleg337-client.virtnet: 1 78379 2.17 MB/sec execute 146 sec latency 9.624 ms oleg337-client.virtnet: 1 78889 2.16 MB/sec execute 147 sec latency 10.558 ms oleg337-client.virtnet: 1 79197 2.16 MB/sec execute 148 sec latency 436.740 ms oleg337-client.virtnet: 1 79815 2.16 MB/sec execute 149 sec latency 10.287 ms oleg337-client.virtnet: 1 80214 2.15 MB/sec execute 150 sec latency 289.902 ms oleg337-client.virtnet: 1 80214 2.14 MB/sec execute 151 sec latency 1290.047 ms oleg337-client.virtnet: 1 80214 2.12 MB/sec execute 152 sec latency 2290.191 ms oleg337-client.virtnet: 1 80214 2.11 MB/seFailover mds1 to oleg337-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg337-server: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-MDT0000 00:42:34 (1733809354) targets are mounted 00:42:34 (1733809354) facet_failover done oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2432 1285256 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37544 3568860 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12072 3591156 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49616 7160016 1% /mnt/lustre test_70b fail mds2 10 times Failing mds2 on oleg337-server Stopping /mnt/lustre-mds2 (opts:) on oleg337-server 00:42:46 (1733809366) shut down facet: mds2 facet_host: oleg337-server facet_failover_host: oleg337-server Failover mds2 to oleg337-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg337-server: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-MDT0001 00:43:00 (1733809380) targets are mounted 00:43:00 (1733809380) facet_failover done oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2468 1285220 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37428 3569028 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12136 3593800 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49564 7162828 1% /mnt/lustre c execute 153 sec latency 3290.337 ms oleg337-client.virtnet: 1 80214 2.10 MB/sec execute 154 sec latency 4290.485 ms oleg337-client.virtnet: 1 80214 2.08 MB/sec execute 155 sec latency 5290.660 ms oleg337-client.virtnet: 1 80214 2.07 MB/sec execute 156 sec latency 6290.810 ms oleg337-client.virtnet: 1 80214 2.06 MB/sec execute 157 sec latency 7290.947 ms oleg337-client.virtnet: 1 80214 2.04 MB/sec execute 158 sec latency 8291.106 ms oleg337-client.virtnet: 1 80214 2.03 MB/sec execute 159 sec latency 9291.242 ms oleg337-client.virtnet: 1 80214 2.02 MB/sec execute 160 sec latency 10291.390 ms oleg337-client.virtnet: 1 80214 2.00 MB/sec execute 161 sec latency 11291.518 ms oleg337-client.virtnet: 1 80214 1.99 MB/sec execute 162 sec latency 12291.736 ms oleg337-client.virtnet: 1 80214 1.98 MB/sec execute 163 sec latency 13291.862 ms oleg337-client.virtnet: 1 80214 1.97 MB/sec execute 164 sec latency 14292.012 ms oleg337-client.virtnet: 1 80214 1.96 MB/sec execute 165 sec latency 15292.131 ms oleg337-client.virtnet: 1 80214 1.94 MB/sec execute 166 sec latency 16292.474 ms oleg337-client.virtnet: 1 80214 1.93 MB/sec execute 167 sec latency 17292.769 ms oleg337-client.virtnet: 1 80295 1.92 MB/sec execute 168 sec latency 18127.033 ms oleg337-client.virtnet: 1 81292 1.95 MB/sec execute 169 sec latency 12.490 ms oleg337-client.virtnet: 1 82074 1.98 MB/sec execute 170 sec latency 7.564 ms oleg337-client.virtnet: 1 82777 1.98 MB/sec execute 171 sec latency 9.320 ms oleg337-client.virtnet: 1 83479 1.99 MB/sec execute 172 sec latency 9.088 ms oleg337-client.virtnet: 1 84125 1.99 MB/sec execute 173 sec latency 11.965 ms oleg337-client.virtnet: 1 85265 2.03 MB/sec execute 174 sec latency 9.078 ms oleg337-client.virtnet: 1 85792 2.03 MB/sec execute 175 sec latency 16.840 ms oleg337-client.virtnet: 1 86422 2.03 MB/sec execute 176 sec latency 10.045 ms oleg337-client.virtnet: 1 87091 2.04 MB/sec execute 177 sec latency 9.392 ms oleg337-client.virtnet: 1 87810 2.04 MB/sec execute 178 sec latency 13.594 ms oleg337-client.virtnet: 1 88942 2.09 MB/sec execute 179 sec latency 8.802 ms oleg337-client.virtnet: 1 89539 2.08 MB/sec execute 180 sec latency 10.532 ms oleg337-client.virtnet: 1 90265 2.10 MB/sec execute 181 sec latency 9.017 ms oleg337-client.virtnet: 1 90893 2.09 MB/sec execute 182 sec latency 10.562 ms oleg337-client.virtnet: 1 91747 2.11 MB/sec execute 183 sec latency 12.514 ms oleg337-client.virtnet: 1 92631 2.14 MB/sec execute 184 sec latency 10.055 ms oleg337-client.virtnet: 1 93231 2.13 MB/sec execute 185 sec latency 11.433 ms oleg337-client.virtnet: 1 93891 2.14 MB/sec execute 186 sec latency 10.199 ms oleg337-client.virtnet: 1 94562 2.14 MB/sec execute 187 sec latency 11.144 ms oleg337-client.virtnet: 1 95476 2.16 MB/sec execute 188 sec latency 10.353 ms oleg337-client.virtnet: 1 96155 2.18 MB/sec execute 189 sec latency 18.530 ms oleg337-client.virtnet: 1 96326 2.17 MB/sec execute 190 sec latency 577.995 ms oleg337-client.virtnet: 1 96326 2.16 MB/sec execute 191 sec latency 1578.187 ms oleg337-client.virtnet: 1 96326 2.15 MB/sec execute 192 sec latency 2578.393 ms oleg337-client.virtnet: 1 96326 2.14 MB/sec execute 193 sec latency 3578.554 ms oleg337-client.virtnet: 1 96761 2.13 MB/sec execute 194 sec latency 3685.628 ms oleg337-client.virtnet: 1 97375 2.14 MB/sec execute 195 sec latency 10.843 ms oleg337-client.virtnet: 1 98043 2.13 MB/sec execute 196 sec latency 8.128 ms oleg337-client.virtnet: 1 99154 2.17 MB/sec execute 197 sec latency 8.862 ms oleg337-clientest_70b fail mds1 11 times Failing mds1 on oleg337-server Stopping /mnt/lustre-mds1 (opts:) on oleg337-server 00:43:11 (1733809391) shut down facet: mds1 facet_host: oleg337-server facet_failover_host: oleg337-server Failover mds1 to oleg337-server mount facets: mds1 Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg337-server: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-MDT0000 00:43:25 (1733809405) targets are mounted 00:43:25 (1733809405) facet_failover done oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 2440 1285248 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2008 1285680 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 37544 3568832 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 12092 3592028 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 49636 7160860 1% /mnt/lustre test_70b fail mds2 12 times Failing mds2 on oleg337-server Stopping /mnt/lustre-mds2 (opts:) on oleg337-server 00:43:37 (1733809417) shut down facet: mds2 facet_host: oleg337-server facet_failover_host: oleg337-server Failover mds2 to oleg337-server mount facets: mds2 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg337-server: oleg337-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Started lustre-MDT0001 00:43:51 (1733809431) targets are mounted 00:43:51 (1733809431) facet_failover done t.virtnet: 1 99823 2.18 MB/sec execute 198 sec latency 8.877 ms oleg337-client.virtnet: 1 100488 2.18 MB/sec execute 199 sec latency 10.287 ms oleg337-client.virtnet: 1 101188 2.19 MB/sec execute 200 sec latency 9.654 ms oleg337-client.virtnet: 1 101422 2.18 MB/sec execute 201 sec latency 688.173 ms oleg337-client.virtnet: 1 101422 2.17 MB/sec execute 202 sec latency 1688.365 ms oleg337-client.virtnet: 1 101422 2.15 MB/sec execute 203 sec latency 2688.565 ms oleg337-client.virtnet: 1 101422 2.14 MB/sec execute 204 sec latency 3688.814 ms oleg337-client.virtnet: 1 101422 2.13 MB/sec execute 205 sec latency 4689.037 ms oleg337-client.virtnet: 1 101422 2.12 MB/sec execute 206 sec latency 5689.213 ms oleg337-client.virtnet: 1 101422 2.11 MB/sec execute 207 sec latency 6689.377 ms oleg337-client.virtnet: 1 101422 2.10 MB/sec execute 208 sec latency 7689.517 ms oleg337-client.virtnet: 1 101422 2.09 MB/sec execute 209 sec latency 8689.710 ms oleg337-client.virtnet: 1 101422 2.08 MB/sec execute 210 sec latency 9689.907 ms oleg337-client.virtnet: 1 101422 2.07 MB/sec execute 211 sec latency 10690.036 ms oleg337-client.virtnet: 1 101422 2.06 MB/sec execute 212 sec latency 11690.186 ms oleg337-client.virtnet: 1 101422 2.05 MB/sec execute 213 sec latency 12690.447 ms oleg337-client.virtnet: 1 101422 2.04 MB/sec execute 214 sec latency 13691.258 ms oleg337-client.virtnet: 1 101422 2.03 MB/sec execute 215 sec latency 14691.596 ms oleg337-client.virtnet: 1 101422 2.03 MB/sec execute 216 sec latency 15691.756 ms oleg337-client.virtnet: 1 101422 2.02 MB/sec execute 217 sec latency 16691.943 ms oleg337-client.virtnet: 1 101422 2.01 MB/sec execute 218 sec latency 17692.146 ms oleg337-client.virtnet: 1 101422 2.00 MB/sec execute 219 sec latency 18692.253 ms oleg337-client.virtnet: 1 102288 2.02 MB/sec execute 220 sec latency 18740.595 ms oleg337-client.virtnet: 1 103207 2.04 MB/sec execute 221 sec latency 9.438 ms oleg337-client.virtnet: 1 103702 2.03 MB/sec execute 222 sec latency 10.576 ms oleg337-client.virtnet: 1 104316 2.04 MB/sec execute 223 sec latency 10.825 ms oleg337-client.virtnet: 1 104825 2.04 MB/sec execute 224 sec latency 12.781 ms oleg337-client.virtnet: 1 105432 2.04 MB/sec execute 225 sec latency 9.327 ms oleg337-client.virtnet: 1 106534 2.07 MB/sec execute 226 sec latency 11.228 ms oleg337-client.virtnet: 1 106968 2.07 MB/sec execute 227 sec latency 10.828 ms oleg337-client.virtnet: 1 107550 2.07 MB/sec execute 228 sec latency 10.502 ms oleg337-client.virtnet: 1 108154 2.08 MB/sec execute 229 sec latency 10.504 ms oleg337-client.virtnet: 1 108797 2.07 MB/sec execute 230 sec latency 9.532 ms oleg337-client.virtnet: 1 109745 2.10 MB/sec execute 231 sec latency 11.605 ms oleg337-client.virtnet: 1 110356 2.11 MB/sec execute 232 sec latency 10.066 ms oleg337-client.virtnet: 1 110913 2.10 MB/sec execute 233 sec latency 9.137 ms oleg337-client.virtnet: 1 111498 2.11 MB/sec execute 234 sec latency 11.780 ms oleg337-client.virtnet: 1 112165 2.11 MB/sec execute 235 sec latency 9.169 ms oleg337-client.virtnet: 1 112989 2.13 MB/sec execute 236 sec latency 12.143 ms oleg337-client.virtnet: 1 113808 2.14 MB/sec execute 237 sec latency 10.931 ms oleg337-client.virtnet: 1 114314 2.14 MB/sec execute 238 sec latency 11.596 ms oleg337-client.virtnet: 1 114960 2.15 MB/sec execute 239 sec latency 10.530 ms oleg337-client.virtnet: 1 115525 2.14 MB/sec execute 240 sec latency 12.377 ms oleg337-client.virtnet: 1 115525 2.13 MB/sec execute 241 sec latency 1012.497 ms oleg337-client.virtnet: 1 115525 2.12 MB/sec execuoleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec te 242 sec latency 2012.643 ms oleg337-client.virtnet: 1 115525 2.11 MB/sec execute 243 sec latency 3012.794 ms oleg337-client.virtnet: 1 115525 2.11 MB/sec execute 244 sec latency 4012.954 ms oleg337-client.virtnet: 1 115731 2.10 MB/sec execute 245 sec latency 4659.139 ms oleg337-client.virtnet: 1 116888 2.12 MB/sec execute 246 sec latency 9.900 ms oleg337-client.virtnet: 1 117505 2.13 MB/sec execute 247 sec latency 11.988 ms oleg337-client.virtnet: 1 118085 2.13 MB/sec execute 248 sec latency 12.035 ms oleg337-client.virtnet: 1 118715 2.14 MB/sec execute 249 sec latency 10.988 ms oleg337-client.virtnet: 1 119331 2.13 MB/sec execute 250 sec latency 10.720 ms oleg337-client.virtnet: 1 120321 2.15 MB/sec execute 251 sec latency 10.104 ms oleg337-client.virtnet: 1 120984 2.17 MB/sec execute 252 sec latency 9.886 ms oleg337-client.virtnet: 1 121507 2.16 MB/sec execute 253 sec latency 12.529 ms oleg337-client.virtnet: 1 122221 2.17 MB/sec execute 254 sec latency 9.419 ms oleg337-client.virtnet: 1 122967 2.17 MB/sec execute 255 sec latency 10.153 ms oleg337-client.virtnet: 1 123979 2.19 MB/sec execute 256 sec latency 9.273 ms oleg337-client.virtnet: 1 124617 2.20 MB/sec execute 257 sec latency 9.863 ms oleg337-client.virtnet: 1 125317 2.20 MB/sec execute 258 sec latency 10.295 ms oleg337-client.virtnet: 1 126041 2.20 MB/sec execute 259 sec latency 8.433 ms oleg337-client.virtnet: 1 126794 2.21 MB/sec execute 260 sec latency 9.887 ms oleg337-client.virtnet: 1 127896 2.24 MB/sec execute 261 sec latency 8.642 ms oleg337-client.virtnet: 1 128417 2.23 MB/sec execute 262 sec latency 14.209 ms oleg337-client.virtnet: 1 129121 2.24 MB/sec execute 263 sec latency 10.308 ms oleg337-client.virtnet: 1 129715 2.24 MB/sec execute 264 sec latency 11.307 ms oleg337-client.virtnet: 1 130648 2.25 MB/sec execute 265 sec latency 8.206 ms oleg337-client.virtnet: 1 131639 2.27 MB/sec execute 266 sec latency 8.127 ms oleg337-client.virtnet: 1 132292 2.27 MB/sec execute 267 sec latency 9.555 ms oleg337-client.virtnet: 1 132935 2.27 MB/sec execute 268 sec latency 11.665 ms oleg337-client.virtnet: 1 133655 2.27 MB/sec execute 269 sec latency 9.281 ms oleg337-client.virtnet: 1 134857 2.30 MB/sec execute 270 sec latency 7.807 ms oleg337-client.virtnet: 1 135335 2.30 MB/sec execute 271 sec latency 12.150 ms oleg337-client.virtnet: 1 135955 2.30 MB/sec execute 272 sec latency 10.765 ms oleg337-client.virtnet: 1 136539 2.30 MB/sec execute 273 sec latency 11.067 ms oleg337-client.virtnet: 1 137235 2.30 MB/sec execute 274 sec latency 9.431 ms oleg337-client.virtnet: 1 138396 2.33 MB/sec execute 275 sec latency 10.397 ms oleg337-client.virtnet: 1 139041 2.33 MB/sec execute 276 sec latency 7.985 ms oleg337-client.virtnet: 1 139597 2.33 MB/sec execute 277 sec latency 12.506 ms oleg337-client.virtnet: 1 140196 2.33 MB/sec execute 278 sec latency 10.633 ms oleg337-client.virtnet: 1 140823 2.33 MB/sec execute 279 sec latency 9.530 ms oleg337-client.virtnet: 1 141889 2.36 MB/sec execute 280 sec latency 11.225 ms oleg337-client.virtnet: 1 142513 2.35 MB/sec execute 281 sec latency 6.792 ms oleg337-client.virtnet: 1 143099 2.35 MB/sec execute 282 sec latency 13.324 ms oleg337-client.virtnet: 1 143698 2.36 MB/sec execute 283 sec latency 9.805 ms oleg337-client.virtnet: 1 144360 2.35 MB/sec execute 284 sec latency 8.186 ms oleg337-client.virtnet: 1 145527 2.38 MB/sec execute 285 sec latency 9.652 ms oleg337-client.virtnet: 1 146121 2.38 MB/sec execute 286 sec latency 8.354 ms oleg337-client.virtnet: 1 146892 2.39 MB/sec execute 287 sec latency 12.849 ms oleg337-client.virtnet: 1 147469 2.38 MB/sec execute 288 sec latency 9.667 ms oleg337-client.virtnet: 1 148359 2.40 MB/sec execute 289 sec latency 10.087 ms oleg337-client.virtnet: 1 149329 2.41 MB/sec execute 290 sec latency 9.804 ms oleg337-client.virtnet: 1 149914 2.41 MB/sec execute 291 sec latency 9.732 ms oleg337-client.virtnet: 1 150578 2.41 MB/sec execute 292 sec latency 9.909 ms oleg337-client.virtnet: 1 151110 2.41 MB/sec execute 293 sec latency 13.054 ms oleg337-client.virtnet: 1 152200 2.42 MB/sec execute 294 sec latency 8.334 ms oleg337-client.virtnet: 1 152895 2.44 MB/sec execute 295 sec latency 9.137 ms oleg337-client.virtnet: 1 153439 2.43 MB/sec execute 296 sec latency 11.258 ms oleg337-client.virtnet: 1 154130 2.44 MB/sec execute 297 sec latency 11.332 ms oleg337-client.virtnet: 1 154810 2.43 MB/sec execute 298 sec latency 9.186 ms oleg337-client.virtnet: 1 155911 2.45 MB/sec execute 299 sec latency 11.045 ms oleg337-client.virtnet: 1 cleanup 300 sec oleg337-client.virtnet: 0 cleanup 300 sec oleg337-client.virtnet: oleg337-client.virtnet: Operation Count AvgLat MaxLat oleg337-client.virtnet: ---------------------------------------- oleg337-client.virtnet: NTCreateX 23543 8.325 18740.582 oleg337-client.virtnet: Close 17251 1.857 18127.020 oleg337-client.virtnet: Rename 998 6.787 19.045 oleg337-client.virtnet: Unlink 4769 1.618 14.846 oleg337-client.virtnet: Qpathinfo 21312 1.205 9.958 oleg337-client.virtnet: Qfileinfo 3718 0.131 1.621 oleg337-client.virtnet: Qfsinfo 3933 0.189 3.128 oleg337-client.virtnet: Sfileinfo 1900 3.948 15.544 oleg337-client.virtnet: Find 8253 0.913 4659.127 oleg337-client.virtnet: WriteX 11621 0.725 5.931 oleg337-client.virtnet: ReadX 37097 0.018 3.088 oleg337-client.virtnet: LockX 78 0.874 1.973 oleg337-client.virtnet: UnlockX 78 0.830 1.585 oleg337-client.virtnet: Flush 1643 4.831 459.355 oleg337-client.virtnet: oleg337-client.virtnet: Throughput 2.45163 MB/sec 1 clients 1 procs max_latency=18740.595 ms oleg337-client.virtnet: stopping dbench on /mnt/lustre/d70b.replay-single/oleg337-client.virtnet at Tue Dec 10 00:44:50 EST 2024 with return code 0 oleg337-client.virtnet: clean dbench files on /mnt/lustre/d70b.replay-single/oleg337-client.virtnet oleg337-client.virtnet: /mnt/lustre/d70b.replay-single/oleg337-client.virtnet /mnt/lustre/d70b.replay-single/oleg337-client.virtnet oleg337-client.virtnet: removed directory: 'clients/client0/~dmtmp/SEED' oleg337-client.virtnet: removed directory: 'clients/client0/~dmtmp/PARADOX' oleg337-client.virtnet: removed directory: 'clients/client0/~dmtmp/ACCESS' oleg337-client.virtnet: removed directory: 'clients/client0/~dmtmp/COREL' oleg337-client.virtnet: removed directory: 'clients/client0/~dmtmp/EXCEL' oleg337-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORD' oleg337-client.virtnet: removed directory: 'clients/client0/~dmtmp/PM' oleg337-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORDPRO' oleg337-client.virtnet: removed directory: 'clients/client0/~dmtmp/PWRPNT' oleg337-client.virtnet: removed directory: 'clients/client0/~dmtmp' oleg337-client.virtnet: removed directory: 'clients/client0' oleg337-client.virtnet: removed directory: 'clients' oleg337-client.virtnet: removed 'client.txt' oleg337-client.virtnet: /mnt/lustre/d70b.replay-single/oleg337-client.virtnet oleg337-client.virtnet: dbench successfully finished