== replay-dual test 26: dbench and tar with mds failover ========================================================== 15:56:29 (1744142189) Starting client oleg305-client.virtnet: -o user_xattr,flock 192.168.203.105@tcp:/lustre /mnt/lustre Started clients oleg305-client.virtnet: 192.168.203.105@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) Started tar loop with pid 15530 Started dbench loop with 15531 looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Tue Apr 8 15:56:31 EDT 2025 waiting for dbench pid 15562 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs failed to create barrier semaphore 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 255 9.66 MB/sec warmup 1 sec latency 25.163 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210688 3328 2205312 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 3072 3631104 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 4096 3704832 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 7168 7335936 1% /mnt/lustre 1 503 8.73 MB/sec warmup 2 sec latency 29.314 ms 1 723 7.10 MB/sec warmup 3 sec latency 65.596 ms 1 892 5.42 MB/sec warmup 4 sec latency 51.844 ms test_26 fail mds1 1 times Failing mds1 on oleg305-server Stopping /mnt/lustre-mds1 (opts:) on oleg305-server 1 1142 4.60 MB/sec warmup 5 sec latency 57.072 ms 15:56:37 (1744142197) shut down facet: mds1 facet_host: oleg305-server facet_failover_host: oleg305-server 1 1142 3.83 MB/sec warmup 6 sec latency 1057.219 ms 1 1142 3.28 MB/sec warmup 7 sec latency 2057.343 ms 1 1142 2.87 MB/sec warmup 8 sec latency 3057.526 ms 1 1142 2.55 MB/sec warmup 9 sec latency 4057.767 ms 1 1142 2.30 MB/sec warmup 10 sec latency 5057.959 ms 1 1142 2.09 MB/sec warmup 11 sec latency 6058.192 ms 1 1142 1.92 MB/sec warmup 12 sec latency 7058.475 ms 1 1142 1.77 MB/sec warmup 13 sec latency 8058.653 ms 1 1142 1.64 MB/sec warmup 14 sec latency 9058.812 ms 1 1142 1.53 MB/sec warmup 15 sec latency 10059.013 ms Failover mds1 to oleg305-server mount facets: mds1 1 1142 1.44 MB/sec warmup 16 sec latency 11059.191 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 1142 1.35 MB/sec warmup 17 sec latency 12059.329 ms oleg305-server: oleg305-server.virtnet: executing set_default_debug -1 all pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 1 1142 1.28 MB/sec warmup 18 sec latency 13059.445 ms Started lustre-MDT0000 15:56:49 (1744142209) targets are mounted 15:56:49 (1744142209) facet_failover done 1 1142 1.21 MB/sec warmup 19 sec latency 14059.604 ms 1 1142 0.00 MB/sec execute 1 sec latency 16060.038 ms 1 1142 0.00 MB/sec execute 2 sec latency 17060.179 ms oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 1296 0.16 MB/sec execute 3 sec latency 17506.876 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 1618 0.79 MB/sec execute 4 sec latency 51.547 ms 1 1886 0.68 MB/sec execute 5 sec latency 38.981 ms 1 2187 0.63 MB/sec execute 6 sec latency 21.341 ms 1 2499 0.82 MB/sec execute 7 sec latency 39.016 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 4736 2203776 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 38912 3553280 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 18432 3639296 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 57344 7192576 1% /mnt/lustre 1 3028 1.32 MB/sec execute 8 sec latency 34.704 ms 1 3636 1.74 MB/sec execute 9 sec latency 26.870 ms test_26 fail mds1 2 times 1 3885 1.68 MB/sec execute 10 sec latency 15.901 ms Failing mds1 on oleg305-server Stopping /mnt/lustre-mds1 (opts:) on oleg305-server 1 4068 1.55 MB/sec execute 11 sec latency 215.974 ms 15:57:02 (1744142222) shut down facet: mds1 facet_host: oleg305-server facet_failover_host: oleg305-server 1 4068 1.42 MB/sec execute 12 sec latency 1216.107 ms 1 4068 1.31 MB/sec execute 13 sec latency 2216.357 ms 1 4068 1.22 MB/sec execute 14 sec latency 3216.554 ms 1 4068 1.14 MB/sec execute 15 sec latency 4216.777 ms 1 4068 1.07 MB/sec execute 16 sec latency 5217.045 ms 1 4068 1.00 MB/sec execute 17 sec latency 6217.288 ms 1 4068 0.95 MB/sec execute 18 sec latency 7217.572 ms 1 4068 0.90 MB/sec execute 19 sec latency 8217.805 ms 1 4068 0.85 MB/sec execute 20 sec latency 9218.096 ms 1 4068 0.81 MB/sec execute 21 sec latency 10218.348 ms Failover mds1 to oleg305-server mount facets: mds1 1 4068 0.78 MB/sec execute 22 sec latency 11218.572 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 4068 0.74 MB/sec execute 23 sec latency 12218.910 ms oleg305-server: oleg305-server.virtnet: executing set_default_debug -1 all 1 4068 0.71 MB/sec execute 24 sec latency 13219.029 ms pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 Started lustre-MDT0000 15:57:15 (1744142235) targets are mounted 15:57:15 (1744142235) facet_failover done 1 4068 0.68 MB/sec execute 25 sec latency 14219.260 ms 1 4068 0.66 MB/sec execute 26 sec latency 15219.537 ms 1 4068 0.63 MB/sec execute 27 sec latency 16219.722 ms 1 4068 0.61 MB/sec execute 28 sec latency 17219.881 ms oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 4248 0.60 MB/sec execute 29 sec latency 17448.034 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 4483 0.60 MB/sec execute 30 sec latency 31.485 ms 1 4795 0.62 MB/sec execute 31 sec latency 30.865 ms 1 5137 0.70 MB/sec execute 32 sec latency 46.935 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 6400 2202112 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 41984 3465216 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 21504 3486720 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 63488 6951936 1% /mnt/lustre 1 5362 0.68 MB/sec execute 33 sec latency 37.876 ms 1 5650 0.67 MB/sec execute 34 sec latency 37.861 ms 1 5946 0.69 MB/sec execute 35 sec latency 55.074 ms test_26 fail mds1 3 times 1 6317 0.78 MB/sec execute 36 sec latency 34.506 ms Failing mds1 on oleg305-server Stopping /mnt/lustre-mds1 (opts:) on oleg305-server 1 6573 0.80 MB/sec execute 37 sec latency 316.694 ms 15:57:28 (1744142248) shut down facet: mds1 facet_host: oleg305-server facet_failover_host: oleg305-server 1 6573 0.78 MB/sec execute 38 sec latency 1316.870 ms 1 6573 0.76 MB/sec execute 39 sec latency 2317.017 ms 1 6573 0.74 MB/sec execute 40 sec latency 3317.167 ms 1 6573 0.72 MB/sec execute 41 sec latency 4317.386 ms 1 6573 0.71 MB/sec execute 42 sec latency 5317.607 ms 1 6573 0.69 MB/sec execute 43 sec latency 6317.846 ms 1 6573 0.67 MB/sec execute 44 sec latency 7318.003 ms 1 6573 0.66 MB/sec execute 45 sec latency 8318.160 ms 1 6573 0.64 MB/sec execute 46 sec latency 9318.318 ms 1 6573 0.63 MB/sec execute 47 sec latency 10318.488 ms Failover mds1 to oleg305-server mount facets: mds1 1 6573 0.62 MB/sec execute 48 sec latency 11318.579 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 6573 0.61 MB/sec execute 49 sec latency 12318.751 ms oleg305-server: oleg305-server.virtnet: executing set_default_debug -1 all pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 Started lustre-MDT0000 15:57:41 (1744142261) targets are mounted 15:57:41 (1744142261) facet_failover done 1 6573 0.59 MB/sec execute 50 sec latency 13318.951 ms 1 6573 0.58 MB/sec execute 51 sec latency 14319.062 ms 1 6573 0.57 MB/sec execute 52 sec latency 15319.219 ms 1 6573 0.56 MB/sec execute 53 sec latency 16319.500 ms 1 6573 0.55 MB/sec execute 54 sec latency 17319.682 ms 1 6573 0.54 MB/sec execute 55 sec latency 18319.857 ms 1 6573 0.53 MB/sec execute 56 sec latency 19320.078 ms 1 6573 0.52 MB/sec execute 57 sec latency 20320.225 ms 1 6573 0.51 MB/sec execute 58 sec latency 21320.401 ms oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1 6933 0.54 MB/sec execute 59 sec latency 21681.776 ms mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 7231 0.58 MB/sec execute 60 sec latency 20.204 ms 1 7396 0.59 MB/sec execute 61 sec latency 27.395 ms 1 7545 0.58 MB/sec execute 62 sec latency 23.126 ms 1 7679 0.57 MB/sec execute 63 sec latency 34.613 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 7808 2200704 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 43008 3402752 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 23552 3493888 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 66560 6896640 1% /mnt/lustre 1 7878 0.57 MB/sec execute 64 sec latency 92.489 ms 1 8082 0.57 MB/sec execute 65 sec latency 29.211 ms 1 8288 0.58 MB/sec execute 66 sec latency 52.421 ms test_26 fail mds1 4 times Failing mds1 on oleg305-server 1 8562 0.62 MB/sec execute 67 sec latency 29.394 ms Stopping /mnt/lustre-mds1 (opts:) on oleg305-server 1 8661 0.61 MB/sec execute 68 sec latency 560.068 ms 15:57:59 (1744142279) shut down facet: mds1 facet_host: oleg305-server facet_failover_host: oleg305-server 1 8661 0.60 MB/sec execute 69 sec latency 1560.255 ms 1 8661 0.59 MB/sec execute 70 sec latency 2560.474 ms 1 8661 0.58 MB/sec execute 71 sec latency 3560.685 ms 1 8661 0.57 MB/sec execute 72 sec latency 4560.863 ms 1 8661 0.57 MB/sec execute 73 sec latency 5561.046 ms 1 8661 0.56 MB/sec execute 74 sec latency 6561.195 ms 1 8661 0.55 MB/sec execute 75 sec latency 7561.364 ms 1 8661 0.54 MB/sec execute 76 sec latency 8561.689 ms 1 8661 0.54 MB/sec execute 77 sec latency 9561.959 ms 1 8661 0.53 MB/sec execute 78 sec latency 10562.084 ms Failover mds1 to oleg305-server mount facets: mds1 1 8661 0.52 MB/sec execute 79 sec latency 11562.247 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 8661 0.52 MB/sec execute 80 sec latency 12562.379 ms oleg305-server: oleg305-server.virtnet: executing set_default_debug -1 all pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 Started lustre-MDT0000 15:58:12 (1744142292) targets are mounted 15:58:12 (1744142292) facet_failover done 1 8661 0.51 MB/sec execute 81 sec latency 13562.504 ms 1 8661 0.50 MB/sec execute 82 sec latency 14562.611 ms 1 8661 0.50 MB/sec execute 83 sec latency 15562.774 ms 1 8661 0.49 MB/sec execute 84 sec latency 16562.980 ms 1 8661 0.49 MB/sec execute 85 sec latency 17563.144 ms 1 8661 0.48 MB/sec execute 86 sec latency 18563.369 ms 1 8661 0.48 MB/sec execute 87 sec latency 19563.508 ms 1 8742 0.47 MB/sec execute 88 sec latency 20180.880 ms oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 9033 0.47 MB/sec execute 89 sec latency 28.383 ms 1 9451 0.48 MB/sec execute 90 sec latency 34.213 ms 1 9868 0.52 MB/sec execute 91 sec latency 30.067 ms 1 10437 0.55 MB/sec execute 92 sec latency 31.267 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 9472 2199040 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 45056 3404800 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 24576 3420160 1% /mnt/lustre[OST:1] filesystem_summary: 7542784 69632 6824960 2% /mnt/lustre 1 10797 0.58 MB/sec execute 93 sec latency 14.078 ms 1 11056 0.59 MB/sec execute 94 sec latency 14.652 ms 1 11325 0.58 MB/sec execute 95 sec latency 23.790 ms test_26 fail mds1 5 times Failing mds1 on oleg305-server Stopping /mnt/lustre-mds1 (opts:) on oleg305-server 1 11528 0.58 MB/sec execute 96 sec latency 60.459 ms 15:58:28 (1744142308) shut down facet: mds1 facet_host: oleg305-server facet_failover_host: oleg305-server 1 11528 0.58 MB/sec execute 97 sec latency 1025.618 ms 1 11528 0.57 MB/sec execute 98 sec latency 2025.843 ms 1 11528 0.57 MB/sec execute 99 sec latency 3026.025 ms 1 cleanup 100 sec 1 cleanup 101 sec 1 cleanup 102 sec 1 cleanup 103 sec 1 cleanup 104 sec 1 cleanup 105 sec 1 cleanup 106 sec Failover mds1 to oleg305-server mount facets: mds1 1 cleanup 107 sec Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 cleanup 108 sec oleg305-server: oleg305-server.virtnet: executing set_default_debug -1 all pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 1 cleanup 109 sec Started lustre-MDT0000 15:58:40 (1744142320) targets are mounted 15:58:40 (1744142320) facet_failover done 1 cleanup 110 sec 1 cleanup 111 sec 1 cleanup 112 sec 1 cleanup 113 sec oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 0 cleanup 113 sec Operation Count AvgLat MaxLat ---------------------------------------- NTCreateX 1794 31.346 21681.754 Close 1308 27.947 17506.839 Rename 74 15.255 29.377 Unlink 366 4.911 13.744 Qpathinfo 1652 15.077 20180.850 Qfileinfo 286 0.441 4.542 Qfsinfo 295 0.568 3.599 Sfileinfo 138 8.963 16.659 Find 630 1.240 17.222 WriteX 873 1.925 10.834 ReadX 2836 0.127 27.318 LockX 6 1.357 1.537 UnlockX 6 1.561 1.858 Flush 123 21.901 92.476 Throughput 0.565708 MB/sec 1 clients 1 procs max_latency=21681.776 ms stopping dbench on /mnt/lustre at Tue Apr 8 15:58:44 EDT 2025 with return code 0 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec looking for dbench program /usr/bin/dbench found dbench client file /usr/share/dbench/client.txt '/usr/share/dbench/client.txt' -> 'client.txt' running 'dbench 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100' on /mnt/lustre at Tue Apr 8 15:58:46 EDT 2025 waiting for dbench pid 20750 dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 Running for 100 seconds with load 'client.txt' and minimum warmup 20 secs 0 of 1 processes prepared for launch 0 sec 1 of 1 processes prepared for launch 0 sec releasing clients 1 289 10.70 MB/sec warmup 1 sec latency 27.201 ms 1 642 10.53 MB/sec warmup 2 sec latency 16.211 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 10752 2197760 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 49152 3478528 2% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 51200 3420160 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 100352 6898688 2% /mnt/lustre 1 873 7.22 MB/sec warmup 3 sec latency 60.930 ms 1 1161 5.75 MB/sec warmup 4 sec latency 44.802 ms 1 1551 5.22 MB/sec warmup 5 sec latency 34.041 ms test_26 fail mds1 6 times Failing mds1 on oleg305-server Stopping /mnt/lustre-mds1 (opts:) on oleg305-server 1 1905 4.40 MB/sec warmup 6 sec latency 33.353 ms 15:58:53 (1744142333) shut down facet: mds1 facet_host: oleg305-server facet_failover_host: oleg305-server 1 2002 3.79 MB/sec warmup 7 sec latency 772.752 ms 1 2002 3.31 MB/sec warmup 8 sec latency 1773.369 ms 1 2002 2.94 MB/sec warmup 9 sec latency 2773.542 ms 1 2002 2.65 MB/sec warmup 10 sec latency 3773.711 ms 1 2002 2.41 MB/sec warmup 11 sec latency 4773.880 ms 1 2002 2.21 MB/sec warmup 12 sec latency 5774.042 ms 1 2002 2.04 MB/sec warmup 13 sec latency 6774.194 ms 1 2002 1.89 MB/sec warmup 14 sec latency 7774.347 ms 1 2002 1.77 MB/sec warmup 15 sec latency 8774.493 ms 1 2002 1.66 MB/sec warmup 16 sec latency 9774.688 ms Failover mds1 to oleg305-server mount facets: mds1 1 2002 1.56 MB/sec warmup 17 sec latency 10774.811 ms Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 2002 1.47 MB/sec warmup 18 sec latency 11774.940 ms oleg305-server: oleg305-server.virtnet: executing set_default_debug -1 all 1 2002 1.39 MB/sec warmup 19 sec latency 12775.117 ms pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 Started lustre-MDT0000 15:59:05 (1744142345) targets are mounted 15:59:05 (1744142345) facet_failover done 1 2002 0.00 MB/sec execute 1 sec latency 14775.488 ms 1 2002 0.00 MB/sec execute 2 sec latency 15775.618 ms 1 2002 0.00 MB/sec execute 3 sec latency 16775.747 ms oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 2345 0.35 MB/sec execute 4 sec latency 16814.846 ms 1 2752 1.09 MB/sec execute 5 sec latency 57.494 ms 1 3278 1.47 MB/sec execute 6 sec latency 59.807 ms 1 3759 1.89 MB/sec execute 7 sec latency 24.948 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210560 10880 2197632 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 20480 3564544 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 53248 3523584 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 73728 7088128 2% /mnt/lustre 1 3995 1.69 MB/sec execute 8 sec latency 17.536 ms 1 4243 1.53 MB/sec execute 9 sec latency 77.551 ms 1 4448 1.43 MB/sec execute 10 sec latency 60.545 ms test_26 fail mds1 7 times Failing mds1 on oleg305-server 1 4727 1.42 MB/sec execute 11 sec latency 23.826 ms Stopping /mnt/lustre-mds1 (opts:) on oleg305-server 1 4842 1.31 MB/sec execute 12 sec latency 593.592 ms 15:59:18 (1744142358) shut down facet: mds1 facet_host: oleg305-server facet_failover_host: oleg305-server 1 4842 1.21 MB/sec execute 13 sec latency 1593.786 ms 1 4842 1.13 MB/sec execute 14 sec latency 2594.018 ms 1 4842 1.05 MB/sec execute 15 sec latency 3594.220 ms 1 4842 0.98 MB/sec execute 16 sec latency 4594.375 ms 1 4842 0.93 MB/sec execute 17 sec latency 5594.503 ms 1 4842 0.88 MB/sec execute 18 sec latency 6594.722 ms 1 4842 0.83 MB/sec execute 19 sec latency 7594.873 ms 1 4842 0.79 MB/sec execute 20 sec latency 8595.055 ms 1 4842 0.75 MB/sec execute 21 sec latency 9595.305 ms 1 4842 0.72 MB/sec execute 22 sec latency 10595.447 ms Failover mds1 to oleg305-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 4842 0.69 MB/sec execute 23 sec latency 11595.555 ms 1 4842 0.66 MB/sec execute 24 sec latency 12595.702 ms oleg305-server: oleg305-server.virtnet: executing set_default_debug -1 all pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 Started lustre-MDT0000 15:59:30 (1744142370) targets are mounted 15:59:30 (1744142370) facet_failover done 1 4842 0.63 MB/sec execute 25 sec latency 13595.863 ms 1 4842 0.61 MB/sec execute 26 sec latency 14596.023 ms 1 4842 0.58 MB/sec execute 27 sec latency 15596.253 ms 1 4842 0.56 MB/sec execute 28 sec latency 16596.399 ms 1 4842 0.54 MB/sec execute 29 sec latency 17596.587 ms 1 4842 0.53 MB/sec execute 30 sec latency 18596.770 ms 1 4842 0.51 MB/sec execute 31 sec latency 19596.923 ms 1 4842 0.49 MB/sec execute 32 sec latency 20597.041 ms 1 4985 0.56 MB/sec execute 33 sec latency 21331.224 ms oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 5249 0.55 MB/sec execute 34 sec latency 57.777 ms 1 5519 0.54 MB/sec execute 35 sec latency 49.355 ms 1 5881 0.57 MB/sec execute 36 sec latency 13.627 ms 1 6180 0.59 MB/sec execute 37 sec latency 54.394 ms UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 2210432 7680 2200704 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 3771392 21504 3707904 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3771392 56320 3686400 2% /mnt/lustre[OST:1] filesystem_summary: 7542784 77824 7394304 2% /mnt/lustre 1 6767 0.71 MB/sec execute 38 sec latency 52.074 ms 1 7244 0.80 MB/sec execute 39 sec latency 15.189 ms 1 7514 0.82 MB/sec execute 40 sec latency 13.805 ms test_26 fail mds1 8 times Failing mds1 on oleg305-server Stopping /mnt/lustre-mds1 (opts:) on oleg305-server 1 7754 0.80 MB/sec execute 41 sec latency 19.908 ms 1 7788 0.78 MB/sec execute 42 sec latency 717.688 ms 15:59:48 (1744142388) shut down facet: mds1 facet_host: oleg305-server facet_failover_host: oleg305-server 1 7788 0.77 MB/sec execute 43 sec latency 1717.899 ms 1 7788 0.75 MB/sec execute 44 sec latency 2718.073 ms 1 7788 0.73 MB/sec execute 45 sec latency 3718.242 ms 1 7788 0.72 MB/sec execute 46 sec latency 4718.407 ms 1 7788 0.70 MB/sec execute 47 sec latency 5718.679 ms 1 7788 0.69 MB/sec execute 48 sec latency 6718.819 ms 1 7788 0.67 MB/sec execute 49 sec latency 7718.975 ms 1 7788 0.66 MB/sec execute 50 sec latency 8719.128 ms 1 7788 0.65 MB/sec execute 51 sec latency 9719.272 ms 1 7788 0.63 MB/sec execute 52 sec latency 10719.416 ms Failover mds1 to oleg305-server mount facets: mds1 Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 1 7788 0.62 MB/sec execute 53 sec latency 11719.536 ms 1 7788 0.61 MB/sec execute 54 sec latency 12719.713 ms oleg305-server: oleg305-server.virtnet: executing set_default_debug -1 all pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 Started lustre-MDT0000 16:00:00 (1744142400) targets are mounted 16:00:00 (1744142400) facet_failover done 1 7788 0.60 MB/sec execute 55 sec latency 13719.898 ms 1 7788 0.59 MB/sec execute 56 sec latency 14720.045 ms 1 7788 0.58 MB/sec execute 57 sec latency 15720.151 ms 1 7796 0.57 MB/sec execute 58 sec latency 16647.617 ms oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 1 8025 0.57 MB/sec execute 59 sec latency 39.389 ms 1 8337 0.58 MB/sec execute 60 sec latency 24.279 ms 1 8674 0.62 MB/sec execute 61 sec latency 65.371 ms 1 8933 0.61 MB/sec execute 62 sec latency 55.694 ms 1 9262 0.61 MB/sec execute 63 sec latency 43.659 ms 1 9546 0.62 MB/sec execute 64 sec latency 59.834 ms 1 10021 0.70 MB/sec execute 65 sec latency 45.865 ms 1 10592 0.76 MB/sec execute 66 sec latency 58.643 ms dbench killed by signal 15 stopping dbench on /mnt/lustre at Tue Apr 8 16:00:12 EDT 2025 with return code 0 20750 pts/0 S+ 0:00 dbench -c client.txt 1 -D /mnt/lustre2/d26.replay-dual/run_dbench -t 100 killed dbench main pid 20750 clean dbench files on /mnt/lustre /mnt/lustre /mnt/lustre removed 'client.txt' /mnt/lustre dbench successfully finished