== replay-single test 70b: dbench 1mdts recovery; 1 clients ========================================================== 11:12:48 (1736266368) Starting client oleg414-client.virtnet: -o user_xattr,flock oleg414-server@tcp:/lustre /mnt/lustre Started clients oleg414-client.virtnet: 192.168.204.114@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt) + MISSING_DBENCH_OK= + PATH=/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/home/green/git/lustre-release/lustre/tests/mpi:/home/green/git/lustre-release/lustre/tests/racer:/home/green/git/lustre-release/lustre/../lustre-iokit/sgpdd-survey:/home/green/git/lustre-release/lustre/tests:/home/green/git/lustre-release/lustre/utils/gss:/home/green/git/lustre-release/lustre/utils:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin::/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests:/sbin:/usr/sbin:/home/green/git/lustre-release/lustre/utils:/home/green/git/lustre-release/lustre/tests/: + DBENCH_LIB= + TESTSUITE=replay-single + TESTNAME=test_70b + MOUNT=/mnt/lustre ++ hostname + DIR=/mnt/lustre/d70b.replay-single/oleg414-client.virtnet + LCTL=/home/green/git/lustre-release/lustre/utils/lctl + rundbench 1 -t 300 dbench: no process found Started rundbench load pid=11867 ... UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3712 2204672 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 31744 3679232 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 3072 3716096 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 34816 7395328 1% /mnt/lustre test_70b fail mds1 1 times Failing mds1 on oleg414-server Stopping /mnt/lustre-mds1 (opts:) on oleg414-server reboot facets: mds1 Failover mds1 to oleg414-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg414-server: oleg414-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3712 2204672 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 31744 3682304 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 13312 3734528 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 45056 7416832 1% /mnt/lustre test_70b fail mds1 2 times Failing mds1 on oleg414-server Stopping /mnt/lustre-mds1 (opts:) on oleg414-server reboot facets: mds1 oleg414-client.virtnet: looking for dbench program oleg414-client.virtnet: /usr/bin/dbench oleg414-client.virtnet: creating output directory /mnt/lustre/d70b.replay-single/oleg414-client.virtnet oleg414-client.virtnet: mkdir: created directory '/mnt/lustre/d70b.replay-single' oleg414-client.virtnet: mkdir: created directory '/mnt/lustre/d70b.replay-single/oleg414-client.virtnet' oleg414-client.virtnet: found dbench client file /usr/share/dbench/client.txt oleg414-client.virtnet: '/usr/share/dbench/client.txt' -> 'client.txt' oleg414-client.virtnet: running 'dbench 1 -t 300' on /mnt/lustre/d70b.replay-single/oleg414-client.virtnet at Tue Jan 7 11:12:49 EST 2025 oleg414-client.virtnet: waiting for dbench pid 11891 oleg414-client.virtnet: dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 oleg414-client.virtnet: oleg414-client.virtnet: Running for 300 seconds with load 'client.txt' and minimum warmup 60 secs oleg414-client.virtnet: failed to create barrier semaphore oleg414-client.virtnet: 0 of 1 processes prepared for launch 0 sec oleg414-client.virtnet: 1 of 1 processes prepared for launch 0 sec oleg414-client.virtnet: releasing clients oleg414-client.virtnet: 1 405 14.34 MB/sec warmup 1 sec latency 23.963 ms oleg414-client.virtnet: 1 726 10.65 MB/sec warmup 2 sec latency 48.079 ms oleg414-client.virtnet: 1 983 7.30 MB/sec warmup 3 sec latency 45.864 ms oleg414-client.virtnet: 1 1422 6.50 MB/sec warmup 4 sec latency 20.396 ms oleg414-client.virtnet: 1 1839 5.27 MB/sec warmup 5 sec latency 31.770 ms oleg414-client.virtnet: 1 2022 4.42 MB/sec warmup 6 sec latency 643.611 ms oleg414-client.virtnet: 1 2022 3.79 MB/sec warmup 7 sec latency 1643.882 ms oleg414-client.virtnet: 1 2022 3.32 MB/sec warmup 8 sec latency 2644.214 ms oleg414-client.virtnet: 1 2022 2.95 MB/sec warmup 9 sec latency 3644.436 ms oleg414-client.virtnet: 1 2022 2.65 MB/sec warmup 10 sec latency 4644.649 ms oleg414-client.virtnet: 1 2022 2.41 MB/sec warmup 11 sec latency 5644.935 ms oleg414-client.virtnet: 1 2022 2.21 MB/sec warmup 12 sec latency 6645.205 ms oleg414-client.virtnet: 1 2022 2.04 MB/sec warmup 13 sec latency 7645.450 ms oleg414-client.virtnet: 1 2022 1.90 MB/sec warmup 14 sec latency 8645.708 ms oleg414-client.virtnet: 1 2022 1.77 MB/sec warmup 15 sec latency 9645.983 ms oleg414-client.virtnet: 1 2022 1.66 MB/sec warmup 16 sec latency 10646.244 ms oleg414-client.virtnet: 1 2022 1.56 MB/sec warmup 17 sec latency 11646.489 ms oleg414-client.virtnet: 1 2022 1.47 MB/sec warmup 18 sec latency 12646.718 ms oleg414-client.virtnet: 1 2022 1.40 MB/sec warmup 19 sec latency 13646.922 ms oleg414-client.virtnet: 1 2022 1.33 MB/sec warmup 20 sec latency 14647.179 ms oleg414-client.virtnet: 1 2022 1.26 MB/sec warmup 21 sec latency 15647.407 ms oleg414-client.virtnet: 1 2022 1.21 MB/sec warmup 22 sec latency 16647.617 ms oleg414-client.virtnet: 1 2022 1.15 MB/sec warmup 23 sec latency 17647.911 ms oleg414-client.virtnet: 1 2158 1.11 MB/sec warmup 24 sec latency 18419.413 ms oleg414-client.virtnet: 1 2570 1.16 MB/sec warmup 25 sec latency 44.374 ms oleg414-client.virtnet: 1 3271 1.36 MB/sec warmup 26 sec latency 37.167 ms oleg414-client.virtnet: 1 3838 1.47 MB/sec warmup 27 sec latency 17.469 ms oleg414-client.virtnet: 1 4270 1.44 MB/sec warmup 28 sec latency 27.468 ms oleg414-client.virtnet: 1 4734 1.45 MB/sec warmup 29 sec latency 31.389 ms oleg414-client.virtnet: 1 5422 1.52 MB/sec warmup 30 sec latency 20.894 ms oleg414-client.virtnet: 1 5955 1.52 MB/sec warmup 31 sec latency 40.536 ms oleg414-client.virtnet: 1 6163 1.50 MB/sec warmup 32 sec latency 571.481 ms oFailover mds1 to oleg414-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg414-server: oleg414-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3712 2204672 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 40960 3702784 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 14336 3752960 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 55296 7455744 1% /mnt/lustre test_70b fail mds1 3 times Failing mds1 on oleg414-server Stopping /mnt/lustre-mds1 (opts:) on oleg414-server reboot facets: mds1 leg414-client.virtnet: 1 6163 1.46 MB/sec warmup 33 sec latency 1571.734 ms oleg414-client.virtnet: 1 6163 1.42 MB/sec warmup 34 sec latency 2572.018 ms oleg414-client.virtnet: 1 6163 1.37 MB/sec warmup 35 sec latency 3572.288 ms oleg414-client.virtnet: 1 6163 1.34 MB/sec warmup 36 sec latency 4572.587 ms oleg414-client.virtnet: 1 6163 1.30 MB/sec warmup 37 sec latency 5572.884 ms oleg414-client.virtnet: 1 6163 1.27 MB/sec warmup 38 sec latency 6573.176 ms oleg414-client.virtnet: 1 6163 1.23 MB/sec warmup 39 sec latency 7573.418 ms oleg414-client.virtnet: 1 6163 1.20 MB/sec warmup 40 sec latency 8573.730 ms oleg414-client.virtnet: 1 6163 1.17 MB/sec warmup 41 sec latency 9574.044 ms oleg414-client.virtnet: 1 6163 1.15 MB/sec warmup 42 sec latency 10574.320 ms oleg414-client.virtnet: 1 6163 1.12 MB/sec warmup 43 sec latency 11574.579 ms oleg414-client.virtnet: 1 6163 1.09 MB/sec warmup 44 sec latency 12574.945 ms oleg414-client.virtnet: 1 6163 1.07 MB/sec warmup 45 sec latency 13575.243 ms oleg414-client.virtnet: 1 6163 1.05 MB/sec warmup 46 sec latency 14575.489 ms oleg414-client.virtnet: 1 6163 1.02 MB/sec warmup 47 sec latency 15575.747 ms oleg414-client.virtnet: 1 6163 1.00 MB/sec warmup 48 sec latency 16576.026 ms oleg414-client.virtnet: 1 6163 0.98 MB/sec warmup 49 sec latency 17576.168 ms oleg414-client.virtnet: 1 6163 0.96 MB/sec warmup 50 sec latency 18576.439 ms oleg414-client.virtnet: 1 6163 0.94 MB/sec warmup 51 sec latency 19576.703 ms oleg414-client.virtnet: 1 6163 0.93 MB/sec warmup 52 sec latency 20576.917 ms oleg414-client.virtnet: 1 6163 0.91 MB/sec warmup 53 sec latency 21577.178 ms oleg414-client.virtnet: 1 6163 0.89 MB/sec warmup 54 sec latency 22577.502 ms oleg414-client.virtnet: 1 6163 0.87 MB/sec warmup 55 sec latency 23577.733 ms oleg414-client.virtnet: 1 6163 0.86 MB/sec warmup 56 sec latency 24578.018 ms oleg414-client.virtnet: 1 6163 0.84 MB/sec warmup 57 sec latency 25578.434 ms oleg414-client.virtnet: 1 6163 0.83 MB/sec warmup 58 sec latency 26578.638 ms oleg414-client.virtnet: 1 6163 0.82 MB/sec warmup 59 sec latency 27578.842 ms oleg414-client.virtnet: 1 6324 2.96 MB/sec execute 1 sec latency 29436.115 ms oleg414-client.virtnet: 1 6964 3.40 MB/sec execute 2 sec latency 38.308 ms oleg414-client.virtnet: 1 7364 3.58 MB/sec execute 3 sec latency 20.586 ms oleg414-client.virtnet: 1 7632 2.77 MB/sec execute 4 sec latency 16.886 ms oleg414-client.virtnet: 1 8051 2.38 MB/sec execute 5 sec latency 28.928 ms oleg414-client.virtnet: 1 8493 2.68 MB/sec execute 6 sec latency 44.418 ms oleg414-client.virtnet: 1 8792 2.33 MB/sec execute 7 sec latency 60.160 ms oleg414-client.virtnet: 1 9287 2.11 MB/sec execute 8 sec latency 32.504 ms oleg414-client.virtnet: 1 9656 2.11 MB/sec execute 9 sec latency 40.797 ms oleg414-client.virtnet: 1 9879 2.21 MB/sec execute 10 sec latency 608.735 ms oleg414-client.virtnet: 1 9879 2.01 MB/sec execute 11 sec latency 1608.998 ms oleg414-client.virtnet: 1 9879 1.84 MB/sec execute 12 sec latency 2609.256 ms oleg414-client.virtnet: 1 9879 1.70 MB/sec execute 13 sec latency 3609.547 ms oleg414-client.virtnet: 1 9879 1.58 MB/sec execute 14 sec latency 4609.820 ms oleg414-client.virtnet: 1 9879 1.47 MB/sec execute 15 sec latency 5610.247 ms oleg414-client.virtnet: 1 9879 1.38 MB/sec execute 16 sec latency 6610.484 ms oleg414-client.virtnet: 1 9879 1.30 MB/sec execute 17 sec latency 7610.701 ms oleg414-client.virtnet: 1 Failover mds1 to oleg414-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg414-server: oleg414-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3712 2204672 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 41984 3727360 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 14336 3745792 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 56320 7473152 1% /mnt/lustre test_70b fail mds1 4 times Failing mds1 on oleg414-server Stopping /mnt/lustre-mds1 (opts:) on oleg414-server reboot facets: mds1 Failover mds1 to oleg414-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg414-server: oleg414-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 Started lustre-MDT0000 9879 1.23 MB/sec execute 18 sec latency 8611.020 ms oleg414-client.virtnet: 1 9879 1.16 MB/sec execute 19 sec latency 9611.364 ms oleg414-client.virtnet: 1 9879 1.10 MB/sec execute 20 sec latency 10611.599 ms oleg414-client.virtnet: 1 9879 1.05 MB/sec execute 21 sec latency 11611.894 ms oleg414-client.virtnet: 1 9879 1.00 MB/sec execute 22 sec latency 12612.131 ms oleg414-client.virtnet: 1 9879 0.96 MB/sec execute 23 sec latency 13612.430 ms oleg414-client.virtnet: 1 9879 0.92 MB/sec execute 24 sec latency 14612.699 ms oleg414-client.virtnet: 1 9879 0.88 MB/sec execute 25 sec latency 15612.996 ms oleg414-client.virtnet: 1 9879 0.85 MB/sec execute 26 sec latency 16613.316 ms oleg414-client.virtnet: 1 9879 0.82 MB/sec execute 27 sec latency 17613.578 ms oleg414-client.virtnet: 1 9879 0.79 MB/sec execute 28 sec latency 18613.778 ms oleg414-client.virtnet: 1 9879 0.76 MB/sec execute 29 sec latency 19613.997 ms oleg414-client.virtnet: 1 9879 0.74 MB/sec execute 30 sec latency 20614.266 ms oleg414-client.virtnet: 1 9879 0.71 MB/sec execute 31 sec latency 21614.488 ms oleg414-client.virtnet: 1 9879 0.69 MB/sec execute 32 sec latency 22614.763 ms oleg414-client.virtnet: 1 9879 0.67 MB/sec execute 33 sec latency 23615.070 ms oleg414-client.virtnet: 1 9879 0.65 MB/sec execute 34 sec latency 24615.284 ms oleg414-client.virtnet: 1 9879 0.63 MB/sec execute 35 sec latency 25615.525 ms oleg414-client.virtnet: 1 9879 0.61 MB/sec execute 36 sec latency 26615.792 ms oleg414-client.virtnet: 1 9879 0.60 MB/sec execute 37 sec latency 27616.131 ms oleg414-client.virtnet: 1 10077 0.62 MB/sec execute 38 sec latency 28405.616 ms oleg414-client.virtnet: 1 10866 0.77 MB/sec execute 39 sec latency 22.857 ms oleg414-client.virtnet: 1 11372 0.76 MB/sec execute 40 sec latency 32.517 ms oleg414-client.virtnet: 1 11653 0.76 MB/sec execute 41 sec latency 33.870 ms oleg414-client.virtnet: 1 12215 0.84 MB/sec execute 42 sec latency 26.365 ms oleg414-client.virtnet: 1 12696 0.83 MB/sec execute 43 sec latency 29.043 ms oleg414-client.virtnet: 1 13567 0.97 MB/sec execute 44 sec latency 18.738 ms oleg414-client.virtnet: 1 14399 1.09 MB/sec execute 45 sec latency 133.467 ms oleg414-client.virtnet: 1 14399 1.06 MB/sec execute 46 sec latency 1133.644 ms oleg414-client.virtnet: 1 14399 1.04 MB/sec execute 47 sec latency 2133.868 ms oleg414-client.virtnet: 1 14399 1.02 MB/sec execute 48 sec latency 3134.184 ms oleg414-client.virtnet: 1 14399 1.00 MB/sec execute 49 sec latency 4134.427 ms oleg414-client.virtnet: 1 14399 0.98 MB/sec execute 50 sec latency 5134.627 ms oleg414-client.virtnet: 1 14399 0.96 MB/sec execute 51 sec latency 6134.876 ms oleg414-client.virtnet: 1 14399 0.94 MB/sec execute 52 sec latency 7135.201 ms oleg414-client.virtnet: 1 14399 0.92 MB/sec execute 53 sec latency 8135.512 ms oleg414-client.virtnet: 1 14399 0.91 MB/sec execute 54 sec latency 9135.792 ms oleg414-client.virtnet: 1 14399 0.89 MB/sec execute 55 sec latency 10136.126 ms oleg414-client.virtnet: 1 14399 0.87 MB/sec execute 56 sec latency 11136.374 ms oleg414-client.virtnet: 1 14399 0.86 MB/sec execute 57 sec latency 12136.658 ms oleg414-client.virtnet: 1 14399 0.84 MB/sec execute 58 sec latency 13136.942 ms oleg414-client.virtnet: 1 14399 0.83 MB/sec execute 59 sec latency 14137.225 ms oleg414-client.virtnet: 1 14399 0.82 MB/sec execute 60 sec latency 15137.442 ms oleg414-client.virtnet: 1 14399 0.80 MB/sec execute 61 sec latency 16137.664 ms oleg414-client.virtneoleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3712 2204672 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 40960 3714048 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 16384 3743744 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 57344 7457792 1% /mnt/lustre test_70b fail mds1 5 times Failing mds1 on oleg414-server Stopping /mnt/lustre-mds1 (opts:) on oleg414-server reboot facets: mds1 Failover mds1 to oleg414-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg414-server: oleg414-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 Started lustre-MDT0000 t: 1 14399 0.79 MB/sec execute 62 sec latency 17137.991 ms oleg414-client.virtnet: 1 14399 0.78 MB/sec execute 63 sec latency 18138.197 ms oleg414-client.virtnet: 1 14399 0.77 MB/sec execute 64 sec latency 19138.430 ms oleg414-client.virtnet: 1 14399 0.75 MB/sec execute 65 sec latency 20138.699 ms oleg414-client.virtnet: 1 14399 0.74 MB/sec execute 66 sec latency 21138.911 ms oleg414-client.virtnet: 1 14399 0.73 MB/sec execute 67 sec latency 22139.213 ms oleg414-client.virtnet: 1 14399 0.72 MB/sec execute 68 sec latency 23139.441 ms oleg414-client.virtnet: 1 14399 0.71 MB/sec execute 69 sec latency 24139.773 ms oleg414-client.virtnet: 1 14399 0.70 MB/sec execute 70 sec latency 25140.110 ms oleg414-client.virtnet: 1 14399 0.69 MB/sec execute 71 sec latency 26140.447 ms oleg414-client.virtnet: 1 14399 0.68 MB/sec execute 72 sec latency 27140.719 ms oleg414-client.virtnet: 1 14399 0.67 MB/sec execute 73 sec latency 28141.007 ms oleg414-client.virtnet: 1 14399 0.66 MB/sec execute 74 sec latency 29141.250 ms oleg414-client.virtnet: 1 14484 0.65 MB/sec execute 75 sec latency 29945.046 ms oleg414-client.virtnet: 1 14933 0.65 MB/sec execute 76 sec latency 26.721 ms oleg414-client.virtnet: 1 15426 0.67 MB/sec execute 77 sec latency 25.006 ms oleg414-client.virtnet: 1 15941 0.70 MB/sec execute 78 sec latency 33.918 ms oleg414-client.virtnet: 1 16317 0.70 MB/sec execute 79 sec latency 37.079 ms oleg414-client.virtnet: 1 16799 0.72 MB/sec execute 80 sec latency 33.185 ms oleg414-client.virtnet: 1 17570 0.79 MB/sec execute 81 sec latency 35.207 ms oleg414-client.virtnet: 1 18115 0.83 MB/sec execute 82 sec latency 12.614 ms oleg414-client.virtnet: 1 18131 0.82 MB/sec execute 83 sec latency 958.386 ms oleg414-client.virtnet: 1 18131 0.81 MB/sec execute 84 sec latency 1958.559 ms oleg414-client.virtnet: 1 18131 0.80 MB/sec execute 85 sec latency 2958.741 ms oleg414-client.virtnet: 1 18131 0.80 MB/sec execute 86 sec latency 3958.958 ms oleg414-client.virtnet: 1 18131 0.79 MB/sec execute 87 sec latency 4959.141 ms oleg414-client.virtnet: 1 18131 0.78 MB/sec execute 88 sec latency 5959.365 ms oleg414-client.virtnet: 1 18131 0.77 MB/sec execute 89 sec latency 6959.641 ms oleg414-client.virtnet: 1 18131 0.76 MB/sec execute 90 sec latency 7959.855 ms oleg414-client.virtnet: 1 18131 0.75 MB/sec execute 91 sec latency 8960.106 ms oleg414-client.virtnet: 1 18131 0.74 MB/sec execute 92 sec latency 9960.300 ms oleg414-client.virtnet: 1 18131 0.74 MB/sec execute 93 sec latency 10960.468 ms oleg414-client.virtnet: 1 18131 0.73 MB/sec execute 94 sec latency 11960.644 ms oleg414-client.virtnet: 1 18131 0.72 MB/sec execute 95 sec latency 12960.828 ms oleg414-client.virtnet: 1 18131 0.71 MB/sec execute 96 sec latency 13961.107 ms oleg414-client.virtnet: 1 18131 0.71 MB/sec execute 97 sec latency 14961.348 ms oleg414-client.virtnet: 1 18131 0.70 MB/sec execute 98 sec latency 15961.595 ms oleg414-client.virtnet: 1 18131 0.69 MB/sec execute 99 sec latency 16961.749 ms oleg414-client.virtnet: 1 18131 0.68 MB/sec execute 100 sec latency 17961.994 ms oleg414-client.virtnet: 1 18131 0.68 MB/sec execute 101 sec latency 18962.277 ms oleg414-client.virtnet: 1 18131 0.67 MB/sec execute 102 sec latency 19962.559 ms oleg414-client.virtnet: 1 18131 0.66 MB/sec execute 103 sec latency 20962.842 ms oleg414-client.virtnet: 1 18131 0.66 MB/sec execute 104 sec latency 21963.073 ms oleg414-client.virtnet: 1 18131 0.65 MB/sec execute 105 sec latency 22963.255 ms oleg414-clienoleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3712 2204672 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 45056 3715072 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 13312 3741696 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 58368 7456768 1% /mnt/lustre test_70b fail mds1 6 times Failing mds1 on oleg414-server Stopping /mnt/lustre-mds1 (opts:) on oleg414-server reboot facets: mds1 Failover mds1 to oleg414-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg414-server: oleg414-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid t.virtnet: 1 18131 0.65 MB/sec execute 106 sec latency 23963.501 ms oleg414-client.virtnet: 1 18131 0.64 MB/sec execute 107 sec latency 24963.809 ms oleg414-client.virtnet: 1 18131 0.63 MB/sec execute 108 sec latency 25964.108 ms oleg414-client.virtnet: 1 18131 0.63 MB/sec execute 109 sec latency 26964.304 ms oleg414-client.virtnet: 1 18131 0.62 MB/sec execute 110 sec latency 27964.532 ms oleg414-client.virtnet: 1 18131 0.62 MB/sec execute 111 sec latency 28964.804 ms oleg414-client.virtnet: 1 18131 0.61 MB/sec execute 112 sec latency 29965.083 ms oleg414-client.virtnet: 1 18447 0.61 MB/sec execute 113 sec latency 30055.883 ms oleg414-client.virtnet: 1 18771 0.61 MB/sec execute 114 sec latency 38.475 ms oleg414-client.virtnet: 1 19316 0.64 MB/sec execute 115 sec latency 25.222 ms oleg414-client.virtnet: 1 19807 0.64 MB/sec execute 116 sec latency 29.075 ms oleg414-client.virtnet: 1 20542 0.68 MB/sec execute 117 sec latency 22.112 ms oleg414-client.virtnet: 1 21157 0.70 MB/sec execute 118 sec latency 35.861 ms oleg414-client.virtnet: 1 21621 0.73 MB/sec execute 119 sec latency 14.039 ms oleg414-client.virtnet: 1 21973 0.73 MB/sec execute 120 sec latency 393.503 ms oleg414-client.virtnet: 1 21973 0.72 MB/sec execute 121 sec latency 1393.706 ms oleg414-client.virtnet: 1 21973 0.72 MB/sec execute 122 sec latency 2394.001 ms oleg414-client.virtnet: 1 21973 0.71 MB/sec execute 123 sec latency 3394.291 ms oleg414-client.virtnet: 1 21973 0.71 MB/sec execute 124 sec latency 4394.637 ms oleg414-client.virtnet: 1 21973 0.70 MB/sec execute 125 sec latency 5394.879 ms oleg414-client.virtnet: 1 21973 0.70 MB/sec execute 126 sec latency 6395.079 ms oleg414-client.virtnet: 1 21973 0.69 MB/sec execute 127 sec latency 7395.390 ms oleg414-client.virtnet: 1 21973 0.69 MB/sec execute 128 sec latency 8395.687 ms oleg414-client.virtnet: 1 21973 0.68 MB/sec execute 129 sec latency 9395.918 ms oleg414-client.virtnet: 1 21973 0.67 MB/sec execute 130 sec latency 10396.184 ms oleg414-client.virtnet: 1 21973 0.67 MB/sec execute 131 sec latency 11396.401 ms oleg414-client.virtnet: 1 21973 0.66 MB/sec execute 132 sec latency 12396.600 ms oleg414-client.virtnet: 1 21973 0.66 MB/sec execute 133 sec latency 13396.835 ms oleg414-client.virtnet: 1 21973 0.65 MB/sec execute 134 sec latency 14397.107 ms oleg414-client.virtnet: 1 21973 0.65 MB/sec execute 135 sec latency 15397.360 ms oleg414-client.virtnet: 1 21973 0.64 MB/sec execute 136 sec latency 16397.537 ms oleg414-client.virtnet: 1 21973 0.64 MB/sec execute 137 sec latency 17397.821 ms oleg414-client.virtnet: 1 21973 0.64 MB/sec execute 138 sec latency 18398.121 ms oleg414-client.virtnet: 1 21973 0.63 MB/sec execute 139 sec latency 19398.444 ms oleg414-client.virtnet: 1 21973 0.63 MB/sec execute 140 sec latency 20398.729 ms oleg414-client.virtnet: 1 21973 0.62 MB/sec execute 141 sec latency 21398.924 ms oleg414-client.virtnet: 1 21973 0.62 MB/sec execute 142 sec latency 22399.250 ms oleg414-client.virtnet: 1 21973 0.61 MB/sec execute 143 sec latency 23399.457 ms oleg414-client.virtnet: 1 21973 0.61 MB/sec execute 144 sec latency 24399.683 ms oleg414-client.virtnet: 1 21973 0.60 MB/sec execute 145 sec latency 25399.965 ms oleg414-client.virtnet: 1 21973 0.60 MB/sec execute 146 sec latency 26400.185 ms oleg414-client.virtnet: 1 21973 0.60 MB/sec execute 147 sec latency 27400.377 ms oleg414-client.virtnet: 1 21973 0.59 MB/sec execute 148 sec latency 28400.624 ms oleg414-client.virtnet: 1 22013 0.59 MB/sec execute 149 sec latency 29264.715 ms olmdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3712 2204672 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 40960 3707904 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 13312 3744768 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 54272 7452672 1% /mnt/lustre test_70b fail mds1 7 times Failing mds1 on oleg414-server Stopping /mnt/lustre-mds1 (opts:) on oleg414-server reboot facets: mds1 Failover mds1 to oleg414-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg414-server: oleg414-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3712 2204672 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 40960 3710976 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 14336 3731456 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 55296 7442432 1% /mnt/lustre test_70b fail mds1 8 times Failing mds1 on oleg414-server eg414-client.virtnet: 1 22532 0.60 MB/sec execute 150 sec latency 20.779 ms oleg414-client.virtnet: 1 23170 0.62 MB/sec execute 151 sec latency 36.395 ms oleg414-client.virtnet: 1 23760 0.62 MB/sec execute 152 sec latency 27.165 ms oleg414-client.virtnet: 1 24683 0.67 MB/sec execute 153 sec latency 26.491 ms oleg414-client.virtnet: 1 25302 0.69 MB/sec execute 154 sec latency 9.958 ms oleg414-client.virtnet: 1 25836 0.69 MB/sec execute 155 sec latency 29.817 ms oleg414-client.virtnet: 1 26347 0.72 MB/sec execute 156 sec latency 301.946 ms oleg414-client.virtnet: 1 26347 0.71 MB/sec execute 157 sec latency 1302.203 ms oleg414-client.virtnet: 1 26347 0.71 MB/sec execute 158 sec latency 2302.418 ms oleg414-client.virtnet: 1 26347 0.70 MB/sec execute 159 sec latency 3302.612 ms oleg414-client.virtnet: 1 26347 0.70 MB/sec execute 160 sec latency 4302.801 ms oleg414-client.virtnet: 1 26347 0.69 MB/sec execute 161 sec latency 5302.934 ms oleg414-client.virtnet: 1 26347 0.69 MB/sec execute 162 sec latency 6303.142 ms oleg414-client.virtnet: 1 26347 0.69 MB/sec execute 163 sec latency 7303.318 ms oleg414-client.virtnet: 1 26347 0.68 MB/sec execute 164 sec latency 8303.527 ms oleg414-client.virtnet: 1 26347 0.68 MB/sec execute 165 sec latency 9303.811 ms oleg414-client.virtnet: 1 26347 0.67 MB/sec execute 166 sec latency 10303.959 ms oleg414-client.virtnet: 1 26347 0.67 MB/sec execute 167 sec latency 11304.081 ms oleg414-client.virtnet: 1 26347 0.67 MB/sec execute 168 sec latency 12304.255 ms oleg414-client.virtnet: 1 26347 0.66 MB/sec execute 169 sec latency 13304.541 ms oleg414-client.virtnet: 1 26347 0.66 MB/sec execute 170 sec latency 14304.819 ms oleg414-client.virtnet: 1 26347 0.65 MB/sec execute 171 sec latency 15305.045 ms oleg414-client.virtnet: 1 26347 0.65 MB/sec execute 172 sec latency 16305.243 ms oleg414-client.virtnet: 1 26347 0.65 MB/sec execute 173 sec latency 17305.496 ms oleg414-client.virtnet: 1 26347 0.64 MB/sec execute 174 sec latency 18305.742 ms oleg414-client.virtnet: 1 26347 0.64 MB/sec execute 175 sec latency 19306.055 ms oleg414-client.virtnet: 1 26347 0.64 MB/sec execute 176 sec latency 20306.261 ms oleg414-client.virtnet: 1 26347 0.63 MB/sec execute 177 sec latency 21306.404 ms oleg414-client.virtnet: 1 26347 0.63 MB/sec execute 178 sec latency 22306.712 ms oleg414-client.virtnet: 1 26347 0.62 MB/sec execute 179 sec latency 23307.033 ms oleg414-client.virtnet: 1 26347 0.62 MB/sec execute 180 sec latency 24307.254 ms oleg414-client.virtnet: 1 26347 0.62 MB/sec execute 181 sec latency 25307.436 ms oleg414-client.virtnet: 1 26347 0.61 MB/sec execute 182 sec latency 26307.720 ms oleg414-client.virtnet: 1 26347 0.61 MB/sec execute 183 sec latency 27308.010 ms oleg414-client.virtnet: 1 26347 0.61 MB/sec execute 184 sec latency 28308.299 ms oleg414-client.virtnet: 1 26347 0.60 MB/sec execute 185 sec latency 29308.638 ms oleg414-client.virtnet: 1 26347 0.60 MB/sec execute 186 sec latency 30308.869 ms oleg414-client.virtnet: 1 26588 0.60 MB/sec execute 187 sec latency 30493.284 ms oleg414-client.virtnet: 1 26941 0.60 MB/sec execute 188 sec latency 50.187 ms oleg414-client.virtnet: 1 27428 0.61 MB/sec execute 189 sec latency 38.254 ms oleg414-client.virtnet: 1 27897 0.63 MB/sec execute 190 sec latency 50.782 ms oleg414-client.virtnet: 1 28491 0.65 MB/sec execute 191 sec latency 38.258 ms oleg414-client.virtnet: 1 28764 0.66 MB/sec execute 192 sec latency 20.341 ms oleg414-client.virtnet: 1 29041 0.65 MB/sec execute 193 sec latency 17.865 ms oleg414-clStopping /mnt/lustre-mds1 (opts:) on oleg414-server reboot facets: mds1 Failover mds1 to oleg414-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg414-server: oleg414-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 3712 2204672 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 40960 3699712 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 13312 3737600 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 54272 7437312 1% /mnt/lustre test_70b fail mds1 9 times Failing mds1 on oleg414-server Stopping /mnt/lustre-mds1 (opts:) on oleg414-server reboot facets: mds1 ient.virtnet: 1 29348 0.65 MB/sec execute 194 sec latency 53.194 ms oleg414-client.virtnet: 1 29511 0.66 MB/sec execute 195 sec latency 594.319 ms oleg414-client.virtnet: 1 29511 0.65 MB/sec execute 196 sec latency 1594.595 ms oleg414-client.virtnet: 1 29511 0.65 MB/sec execute 197 sec latency 2594.808 ms oleg414-client.virtnet: 1 29511 0.65 MB/sec execute 198 sec latency 3595.013 ms oleg414-client.virtnet: 1 29511 0.64 MB/sec execute 199 sec latency 4595.289 ms oleg414-client.virtnet: 1 29511 0.64 MB/sec execute 200 sec latency 5595.492 ms oleg414-client.virtnet: 1 29511 0.64 MB/sec execute 201 sec latency 6595.726 ms oleg414-client.virtnet: 1 29511 0.63 MB/sec execute 202 sec latency 7595.875 ms oleg414-client.virtnet: 1 29511 0.63 MB/sec execute 203 sec latency 8596.045 ms oleg414-client.virtnet: 1 29511 0.63 MB/sec execute 204 sec latency 9596.357 ms oleg414-client.virtnet: 1 29511 0.62 MB/sec execute 205 sec latency 10596.602 ms oleg414-client.virtnet: 1 29511 0.62 MB/sec execute 206 sec latency 11596.859 ms oleg414-client.virtnet: 1 29511 0.62 MB/sec execute 207 sec latency 12597.009 ms oleg414-client.virtnet: 1 29511 0.61 MB/sec execute 208 sec latency 13597.178 ms oleg414-client.virtnet: 1 29511 0.61 MB/sec execute 209 sec latency 14597.388 ms oleg414-client.virtnet: 1 29511 0.61 MB/sec execute 210 sec latency 15597.627 ms oleg414-client.virtnet: 1 29511 0.61 MB/sec execute 211 sec latency 16597.813 ms oleg414-client.virtnet: 1 29511 0.60 MB/sec execute 212 sec latency 17597.981 ms oleg414-client.virtnet: 1 29511 0.60 MB/sec execute 213 sec latency 18598.147 ms oleg414-client.virtnet: 1 29511 0.60 MB/sec execute 214 sec latency 19598.453 ms oleg414-client.virtnet: 1 29511 0.59 MB/sec execute 215 sec latency 20598.749 ms oleg414-client.virtnet: 1 29511 0.59 MB/sec execute 216 sec latency 21598.984 ms oleg414-client.virtnet: 1 29511 0.59 MB/sec execute 217 sec latency 22599.295 ms oleg414-client.virtnet: 1 29511 0.59 MB/sec execute 218 sec latency 23599.511 ms oleg414-client.virtnet: 1 29511 0.58 MB/sec execute 219 sec latency 24599.735 ms oleg414-client.virtnet: 1 29511 0.58 MB/sec execute 220 sec latency 25600.033 ms oleg414-client.virtnet: 1 29511 0.58 MB/sec execute 221 sec latency 26600.307 ms oleg414-client.virtnet: 1 29511 0.58 MB/sec execute 222 sec latency 27600.582 ms oleg414-client.virtnet: 1 29586 0.57 MB/sec execute 223 sec latency 28482.472 ms oleg414-client.virtnet: 1 30112 0.59 MB/sec execute 224 sec latency 43.992 ms oleg414-client.virtnet: 1 30746 0.59 MB/sec execute 225 sec latency 26.871 ms oleg414-client.virtnet: 1 31210 0.61 MB/sec execute 226 sec latency 49.812 ms oleg414-client.virtnet: 1 31772 0.62 MB/sec execute 227 sec latency 38.962 ms oleg414-client.virtnet: 1 32213 0.63 MB/sec execute 228 sec latency 16.833 ms oleg414-client.virtnet: 1 32465 0.63 MB/sec execute 229 sec latency 17.586 ms oleg414-client.virtnet: 1 32929 0.63 MB/sec execute 230 sec latency 24.295 ms oleg414-client.virtnet: 1 33375 0.65 MB/sec execute 231 sec latency 459.875 ms oleg414-client.virtnet: 1 33375 0.65 MB/sec execute 232 sec latency 1460.088 ms oleg414-client.virtnet: 1 33375 0.64 MB/sec execute 233 sec latency 2460.427 ms oleg414-client.virtnet: 1 33375 0.64 MB/sec execute 234 sec latency 3460.686 ms oleg414-client.virtnet: 1 33375 0.64 MB/sec execute 235 sec latency 4460.974 ms oleg414-client.virtnet: 1 33375 0.64 MB/sec execute 236 sec latency 5461.312 ms oleg414-client.virtnet: 1 33375 0.63 MB/sec execute 237 sec latency 6461.595 ms oleg414-clienFailover mds1 to oleg414-server mount facets: mds1 Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg414-server: oleg414-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec t.virtnet: 1 33375 0.63 MB/sec execute 238 sec latency 7461.893 ms oleg414-client.virtnet: 1 33375 0.63 MB/sec execute 239 sec latency 8462.126 ms oleg414-client.virtnet: 1 33375 0.62 MB/sec execute 240 sec latency 9462.347 ms oleg414-client.virtnet: 1 33375 0.62 MB/sec execute 241 sec latency 10462.611 ms oleg414-client.virtnet: 1 33375 0.62 MB/sec execute 242 sec latency 11462.834 ms oleg414-client.virtnet: 1 33375 0.62 MB/sec execute 243 sec latency 12463.069 ms oleg414-client.virtnet: 1 33375 0.61 MB/sec execute 244 sec latency 13463.423 ms oleg414-client.virtnet: 1 33375 0.61 MB/sec execute 245 sec latency 14463.599 ms oleg414-client.virtnet: 1 33375 0.61 MB/sec execute 246 sec latency 15463.780 ms oleg414-client.virtnet: 1 33375 0.61 MB/sec execute 247 sec latency 16463.991 ms oleg414-client.virtnet: 1 33375 0.60 MB/sec execute 248 sec latency 17464.309 ms oleg414-client.virtnet: 1 33375 0.60 MB/sec execute 249 sec latency 18464.579 ms oleg414-client.virtnet: 1 33375 0.60 MB/sec execute 250 sec latency 19464.796 ms oleg414-client.virtnet: 1 33375 0.60 MB/sec execute 251 sec latency 20465.104 ms oleg414-client.virtnet: 1 33375 0.60 MB/sec execute 252 sec latency 21465.415 ms oleg414-client.virtnet: 1 33375 0.59 MB/sec execute 253 sec latency 22465.684 ms oleg414-client.virtnet: 1 33375 0.59 MB/sec execute 254 sec latency 23465.927 ms oleg414-client.virtnet: 1 33375 0.59 MB/sec execute 255 sec latency 24466.196 ms oleg414-client.virtnet: 1 33375 0.59 MB/sec execute 256 sec latency 25466.465 ms oleg414-client.virtnet: 1 33375 0.58 MB/sec execute 257 sec latency 26466.661 ms oleg414-client.virtnet: 1 33375 0.58 MB/sec execute 258 sec latency 27466.831 ms oleg414-client.virtnet: 1 33375 0.58 MB/sec execute 259 sec latency 28467.147 ms oleg414-client.virtnet: 1 33375 0.58 MB/sec execute 260 sec latency 29467.374 ms oleg414-client.virtnet: 1 33775 0.58 MB/sec execute 261 sec latency 29625.591 ms oleg414-client.virtnet: 1 34235 0.58 MB/sec execute 262 sec latency 28.772 ms oleg414-client.virtnet: 1 34550 0.58 MB/sec execute 263 sec latency 47.115 ms oleg414-client.virtnet: 1 35171 0.60 MB/sec execute 264 sec latency 47.151 ms oleg414-client.virtnet: 1 35736 0.62 MB/sec execute 265 sec latency 18.213 ms oleg414-client.virtnet: 1 35988 0.62 MB/sec execute 266 sec latency 17.854 ms oleg414-client.virtnet: 1 36501 0.62 MB/sec execute 267 sec latency 31.088 ms oleg414-client.virtnet: 1 37082 0.63 MB/sec execute 268 sec latency 47.216 ms oleg414-client.virtnet: 1 37382 0.63 MB/sec execute 269 sec latency 39.847 ms oleg414-client.virtnet: 1 37925 0.63 MB/sec execute 270 sec latency 43.804 ms oleg414-client.virtnet: 1 38433 0.65 MB/sec execute 271 sec latency 49.489 ms oleg414-client.virtnet: 1 39010 0.67 MB/sec execute 272 sec latency 46.705 ms oleg414-client.virtnet: 1 39396 0.67 MB/sec execute 273 sec latency 12.908 ms oleg414-client.virtnet: 1 39836 0.67 MB/sec execute 274 sec latency 46.793 ms oleg414-client.virtnet: 1 40226 0.67 MB/sec execute 275 sec latency 34.421 ms oleg414-client.virtnet: 1 40760 0.68 MB/sec execute 276 sec latency 29.443 ms oleg414-client.virtnet: 1 41459 0.69 MB/sec execute 277 sec latency 24.125 ms oleg414-client.virtnet: 1 42358 0.71 MB/sec execute 278 sec latency 30.508 ms oleg414-client.virtnet: 1 42906 0.72 MB/sec execute 279 sec latency 13.044 ms oleg414-client.virtnet: 1 43337 0.72 MB/sec execute 280 sec latency 25.597 ms oleg414-client.virtnet: 1 43691 0.73 MB/sec execute 281 sec latency 33.589 ms oleg414-client.virtnet: 1 44146 0.74 MB/sec execute 282 sec latency 43.897 ms oleg414-client.virtnet: 1 44766 0.74 MB/sec execute 283 sec latency 24.366 ms oleg414-client.virtnet: 1 45496 0.75 MB/sec execute 284 sec latency 31.740 ms oleg414-client.virtnet: 1 46230 0.77 MB/sec execute 285 sec latency 25.994 ms oleg414-client.virtnet: 1 46520 0.77 MB/sec execute 286 sec latency 18.086 ms oleg414-client.virtnet: 1 46826 0.77 MB/sec execute 287 sec latency 50.784 ms oleg414-client.virtnet: 1 47081 0.77 MB/sec execute 288 sec latency 46.005 ms oleg414-client.virtnet: 1 47635 0.78 MB/sec execute 289 sec latency 39.326 ms oleg414-client.virtnet: 1 48090 0.78 MB/sec execute 290 sec latency 39.365 ms oleg414-client.virtnet: 1 48560 0.79 MB/sec execute 291 sec latency 46.499 ms oleg414-client.virtnet: 1 49143 0.80 MB/sec execute 292 sec latency 30.828 ms oleg414-client.virtnet: 1 49839 0.82 MB/sec execute 293 sec latency 34.459 ms oleg414-client.virtnet: 1 50114 0.82 MB/sec execute 294 sec latency 17.503 ms oleg414-client.virtnet: 1 50401 0.82 MB/sec execute 295 sec latency 64.058 ms oleg414-client.virtnet: 1 50655 0.82 MB/sec execute 296 sec latency 44.065 ms oleg414-client.virtnet: 1 51256 0.83 MB/sec execute 297 sec latency 27.448 ms oleg414-client.virtnet: 1 51675 0.83 MB/sec execute 298 sec latency 22.413 ms oleg414-client.virtnet: 1 52078 0.83 MB/sec execute 299 sec latency 42.102 ms oleg414-client.virtnet: 1 cleanup 300 sec oleg414-client.virtnet: 0 cleanup 300 sec oleg414-client.virtnet: oleg414-client.virtnet: Operation Count AvgLat MaxLat oleg414-client.virtnet: ---------------------------------------- oleg414-client.virtnet: NTCreateX 8008 20.873 30493.259 oleg414-client.virtnet: Close 5880 11.313 29625.557 oleg414-client.virtnet: Rename 339 8.336 17.915 oleg414-client.virtnet: Unlink 1616 2.247 4.501 oleg414-client.virtnet: Qpathinfo 7247 5.503 28405.599 oleg414-client.virtnet: Qfileinfo 1272 0.135 2.220 oleg414-client.virtnet: Qfsinfo 1337 0.121 6.041 oleg414-client.virtnet: Sfileinfo 650 48.669 29264.702 oleg414-client.virtnet: Find 2804 0.467 20.000 oleg414-client.virtnet: WriteX 3957 0.893 6.016 oleg414-client.virtnet: ReadX 12703 0.014 2.262 oleg414-client.virtnet: LockX 28 1.320 1.871 oleg414-client.virtnet: UnlockX 28 1.284 1.963 oleg414-client.virtnet: Flush 560 20.044 64.036 oleg414-client.virtnet: oleg414-client.virtnet: Throughput 0.827836 MB/sec 1 clients 1 procs max_latency=30493.284 ms oleg414-client.virtnet: stopping dbench on /mnt/lustre/d70b.replay-single/oleg414-client.virtnet at Tue Jan 7 11:18:50 EST 2025 with return code 0 oleg414-client.virtnet: clean dbench files on /mnt/lustre/d70b.replay-single/oleg414-client.virtnet oleg414-client.virtnet: /mnt/lustre/d70b.replay-single/oleg414-client.virtnet /mnt/lustre/d70b.replay-single/oleg414-client.virtnet oleg414-client.virtnet: removed directory: 'clients/client0/~dmtmp/PARADOX' oleg414-client.virtnet: removed directory: 'clients/client0/~dmtmp/EXCEL' oleg414-client.virtnet: removed directory: 'clients/client0/~dmtmp/PM' oleg414-client.virtnet: removed directory: 'clients/client0/~dmtmp/PWRPNT' oleg414-client.virtnet: removed directory: 'clients/client0/~dmtmp/COREL' oleg414-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORD' oleg414-client.virtnet: removed directory: 'clients/client0/~dmtmp/WORDPRO' oleg414-client.virtnet: removed directory: 'clients/client0/~dmtmp/ACCESS' oleg414-client.virtnet: removed directory: 'clients/client0/~dmtmp/SEED' oleg414-client.virtnet: removed directory: 'clients/client0/~dmtmp' oleg414-client.virtnet: removed directory: 'clients/client0' oleg414-client.virtnet: removed directory: 'clients' oleg414-client.virtnet: removed 'client.txt' oleg414-client.virtnet: /mnt/lustre/d70b.replay-single/oleg414-client.virtnet oleg414-client.virtnet: dbench successfully finished